The six-strong gang of Latvian workers was spotted living in portable cabins on the £5m Homes and Communities Agency owned site in Hood Street.
Ren Davies, Acting Regional Secretary for Ucatt North West, said: “Workers should never be asked to live on a construction site, it is likely to be both dangerous and with poor welfare facilities.
“Even in exceptional cases where accommodation is provided it must be entirely separate from the site where work is being undertaken.
The principal contractor on the site is Northern Group.
The company told the Manchester Evening News it would find new accommodation for the workers.
A spokesman added that the accommodation provided was “very similar to that used by workers on the Olympic Park.”