The decision follows weeks of negotiations between the council, designer Capita and Thomas Barnes.
The builder said it has expressed concerns about the structural integrity of a novel design by Capita to support the station canopy.
Eric Wright, Kier and Galliford Try all bid the project, which Barnes started last May as a 40-week contract.
The council said it had become concerned at the relationships between the various parties working on the contract and the impact it was having on project delivery.
Under the striking design, the bus station canopy is supported by several looping steel supports, which appear to hang from the roof
The composition of the canopy draws inspiration from the cotton weaving looms which is a nod to the towns industrial past.
First steel being installed last year.
Thomas Barnes, director of Thomas Barnes & Sons Group said:
“Naturally we are extremely disappointed to have been required to leave site by the Council, particularly as we have had productive discussions about completing the bus station in a realistic timeframe.
“We were certainly working to achieve completion as set out in our contract with Blackburn Council.
“We have also taken proactive measures with the Council to ensure the bus station’s completion in a manner that was mutually agreeable with all parties, however, we have on numerous occasions expressed our concerns over the structural integrity of the bus station’s design by Capita and what this means in terms of delivering a safe transport hub given the budgets and resources available.
He added: “It was only in April of this year, in response to our concerns, that Blackburn Borough Council asked Thomas Barnes’s permission for independent structural engineers to enter the site and inspect the structural integrity of the structure which had deflected excessively when part loaded.
“Though Thomas Barnes have requested a copy of the report/results of the independent investigation the Council have refused to provide a copy.
“It should be noted that Thomas Barnes & Sons have successfully completed three new bus stations – Eccles, Oldham and Middleton for Greater Manchester Passenger Transport Executive valued in excess of £7.6m.
Barnes added: “The whole matter is now in the hands of our legal advisors.”
Councillor Phil Riley, executive member for regeneration at Blackburn with Darwen Council, said it had taken “decisive action” to ensure the bus station progresses.
“We have terminated the contract with Thomas Barnes and are proceeding with procuring a contractor to complete the development.
“It is important that people understand we have been acting under legal privilege so we have been unable – rather than unwilling – to provide an update before now.”
The bus station delay has impacted other major schemes in the town centre regeneration including selling the top part of the market site, currently used as a temporary bus station, to a high quality retailer and could affect the completion date of the whole Pennine Reach scheme.