The contract was due to run until June 2015 with the option of a further five-year extension.
But the council confirmed that both sides have “mutually agreed” to end the deal early.
It is believed the final straw for the council was the £5.1m refurbishment of Draper House at Elephant & Castle where work was stopped last November when a resident was hospitalised after a gas leak.
Councillor Ian Wingfield, Deputy Leader and Cabinet Member for Housing, said: “We always put the needs of our residents first, and are fully committed to providing the highest standard of major works. We owe it to residents to constantly review the delivery of works and make changes where necessary.
“Southwark Council and Breyer Group PLC have mutually agreed to conclude their five year partnering agreement. This agreement means that Breyer will no longer be responsible for the completion of works at Draper House.
“Our priority is to resume work at Draper House as swiftly as possible. We know that is what residents want, and we will be speaking to them about our plans to get this important work underway again.”
Breyer is the latest contractor to fall out with Southwark after the council terminated contracts with Wates and Morrison last year.