Sisk is currently working on the £75m first phase of the mixed-use scheme which is due for completion next year.
Developer HDG Mansur has held talks with Bristol City Council over original Section 106 demands to include a further 37 affordable homes on the site costing £4.5m.
Private flats in the scheme start at £150,000.
The council could also remove obligations for the firm to provide contributions towards roads, schools and public art.
These form part of the Section 106 agreements but the coalition government has said these can be relaxed in order to facilitate the construction of new homes.
Councillor Brenda Hugill, Labour, told the BBC: “I think it’s a great tragedy and we are in desperate need for affordable homes, and I’m also disappointed that the developers feel they have to go down this road.”
Michael Baker, director of HDG Mansur, said: “The financial landscape has changed so significantly since we first agreed with Bristol City Council for the social housing.
“In particular the rising costs of raw materials has affected the overall buildings costs and the value of commercial and residential developments have fallen quite dramatically.”
A decision will be made at the Central Area Development Committee meeting next Wednesday.