The worries come despite industry-leading pay rates on the project which could see skilled civil engineering operatives earn £70,000 a year.
Much of the groundwork has now been completed on the site by a Kier BAM joint venture.
Main civils contractor Bouygues and Laing O’Rourke (BYLOR) will be looking to employ around 3,500 people when the site is in full swing.
One project source said: “Quite a lot of the workers on site up to now have been employed through agencies but the main civils workers will be directly employed.
“That’s a lot of staff to find and people are realising now what a big ask that is going to be.
“They are offering very attractive wages but the job will need a massive influx of workers.”
A new pay deal for the site was thrashed out late last year which will rates rise to over £18 an hour for skilled workers.
Unite described the deal as “a landmark agreement which will now set the benchmark for pay rates on flagship construction projects throughout the UK.”
Local training experts are more confident that roles will be filled.
Stuart Anderson of Construction Skills People said: “This skills shortage should not be all doom and gloom.
“It is important to tell employers in Somerset that training providers, such as ourselves, do have NVQ funding available which will help develop the workforce and plug skills gaps.”