The project has been hit by delays after more than 400 tonnes of asbestos were uncovered across the building in July.
Site sources told the Enquirer on Thursday afternoon that Styles & Wood had suddenly left the job.
A Styles & Wood spokesperson said: “There have been significant issues giving rise to delays on India Buildings, namely the discovery of large volumes of asbestos and extensive additional ceiling works requiring specialist heritage planning permission.
“A dispute has arisen, and Styles&Wood can confirm it is no longer working on the project.
“We consider the termination of our contract to be wrongful and we’re currently taking advice on next steps.
“Styles&Wood is being supported by specialist legal and expert advisors in this respect.
“Styles&Wood was deeply committed to delivering this project. We are very disappointed that this decision has been taken.
“We have been working closely with all parties to overcome the issues that have been causing delays to the programme as we are very experienced in the renovation of heritage buildings and the challenges in restoring them.
“We very much hope that the renovation of this beautiful building is not put at risk by this action and that an equally experienced contractor is appointed to complete the project to the standard and timescale required.”
The India Buildings are being converted into a new home for HMRC tax workers.
Legal & General acquired the Grade II listed building for £125m from Shelborn Asset Management last year.
Shelborn Asset Management continues to manage the refurbishment works.