Kier was awarded a £24m contract by developer Bond City in May to deliver the scheme which has a total value of £53m.
The contract included the construction of a six-screen cinema, hotel, restaurants, and shops.
Kier started enabling works, including piling, with the main build due to be completed by Autumn 2017.
But the firm has now walked away from the job leaving Bond City looking for a replacement contractor.
Kier operations director Paul Abrahams told Kentlive: “We have successfully delivered our element of the enabling works package for the St James Development.
“However, we have been unable to reach consensus on the next phase of the project. We will continue to support our client Bond City Ltd as part of our handover.”
Bond City said it was in discussions with a replacement contractor.