The council and development partners McAlpine and Bristol-based Deeley Freed were due to finalise the deal to form a local assets based partnership company last month.
Torbay council will now go back to the drawing board and try to negotiate with a rival bidder from the developer competition held earlier this year.
A council statement said: “Sir Robert McAlpine and their partner Deeley Freed withdrew from the joint venture project for a combination of reasons including changing priorities within the business and the increasing weaknesses and uncertainties in the global and European economy.”
Torbay council planned to commit land and buildings, which were to be matched with cash or assets from the private sector partners.
It is believed an early sticking point was over plans to build on existing car parks, which some councillors were keen to protect.
Several key sites in Torquay and Paignton had been selected as ripe for development.
Five of the sites highlighted for development were the Victoria Centre and Station Lane, in Paignton, and the Terrace car park, Union Street and the Castle Circus area in Torquay.
The council hopes the sites could be turned into a range of leisure, office, retail and residential developments.
Further sites have been identified for a second wave of developments.