No-one was working on the lightweight tower which crashed more than 100ft to the ground below.
The tower fell last week from Astor House in Notte Street which is being transformed into new student accommodation by Leadbitter.
A source told the Plymouth Herald: “These towers are very, very lightweight and used for quick access to do jobs like change lightbulbs or paint ceilings.
“There were wind warnings last week so everything loose should have been made secure. You’d think there would have been a full strap and wrap routine as well as stabilisers and outriggers to make it secure.
“Maybe they were and something went wrong.
“But it blew off the roof and landed in a central courtyard. Thankfully it didn’t fall off the other way and land on the road.”
The source added: “I doubt the men’s weight would have been enough to hold it down. If anyone had been on it, they’d probably have gone with it.”
A spokesman for the Health and Safety Executive said: “We have been made aware of the incident and initial enquiries are continuing.
“However, we have not issued a prohibition notice preventing work from continuing.”
The tower was in no way connected with the fixed scaffolding around the £20m student accommodation project.
A spokesperson from the Leadbitter Group told the Herald they could not comment.