The contract was awarded to Lendlease in late 2018 and clearance work is underway on the Perry Barr site.
But athletes and games officials will now be put up in existing accommodation at the universities of Birmingham and Warwick and the National Exhibition Centre.
The original construction contract was for £350m but cost estimates have been steadily rising.
The impact of the Covid-19 pandemic was the final straw for the bespoke village scheme.
But building of new homes on the site will still go-ahead without the time pressure of being ready for the games.
Organisers said the decision “follows an assessment by the project delivery team, who has been working with the construction supply chain, as well as independent experts, on how they could de-risk Games delivery.
“With a shorter than normal timeframe for delivery of the Birmingham 2022 Games, the new build accommodation site was under continued review from the outset, with expert consultation throughout, and had very little scope to withstand the impact Covid-19 has had on construction.”
They added: “The Perry Barr Regeneration Scheme, with new housing and transport infrastructure, will still be delivered.
“This council-led residential scheme has been accelerated due to the Birmingham 2022 Commonwealth Games and has been made possible with additional investment from central government.”
Ian Reid, chief executive officer for Birmingham 2022 said: “These are challenging times for all of us and delivering a major multi sports event during this period has meant we have needed to collaborate effectively, be pragmatic to change, and remain realistic about the challenges we face.
“We recognise that this new model is a move away from the historic norm and we are grateful for the support shown by our partners across the Commonwealth Games Associations.
“Birmingham and the West Midlands is extremely fortunate to have superb alternate facilities and we are making this decision now, with two years to go, to de-risk the project, ensure delivery for athletes and teams and secure the legacy of new housing and transport infrastructure in Perry Barr.
“This is the sensible and pragmatic thing to do for the Games, the athletes and for the people of Birmingham.”
Cllr Ian Ward, Leader of Birmingham City Council, said: “The key physical legacy from Birmingham 2022 is a regenerated Perry Barr – and by working closely with our Games Partners that is absolutely secure going forward.
“Work to deliver housing which is currently ongoing will continue in the months and years to come as part of the wider plan to deliver 5,000 much needed homes for people in this area of the city.
“The fact we are moving straight to legacy on the residential scheme gives the council and its partners a chance to review the longer-term plans for homes in Perry Barr, to ensure they best meet the needs of local people.
“We will engage with local communities and offer more precise details on project timescales in due course.
“Put simply, this decision on the future of residential scheme brings greater certainty for the Commonwealth Games and will help us define a vision for Perry Barr which has the wellbeing of the community at its heart, to deliver the best possible legacy from the Games.”
