The Enquirer understands that the builder was standing in a smoking area on a residential scheme in Lancaster Gate when the pole was dropped on him.
Site sources said the scaffold hit him directly on the head last Friday morning.
Health and Safety Executive inspectors are investigating the tragedy but the authorities were unable to confirm the name of the main contractor on the site or who the dead man was working for.
A HSE spokesman said: “HSE has been informed of a fatal accident at a construction site in Lancaster Gate, London W2.
“Inspectors from the Health and Safety Executive are attending the scene as part of a joint investigation with the Metropolitan Police.”