The warning comes amid a surge in fly tipping and workers admitted to hospital after handling contaminated waste.
The agency said: “Over the past year we have taken action against those illegally collecting and dumping significant waste originating from builders.
“In some cases, workers have been admitted to A&E following the collection of contaminated waste, unscrupulous skip businesses have collected building site waste only to dump it, and career criminals have made millions dealing in waste that they go on to burn or bury.”
The Environment Agency is now calling on all construction and demolition sites that produce, transport, treat or dispose of waste to fulfil their duty of care.
Builders have a legal responsibility to ensure they send their site waste to legal waste collectors. Not doing so can result in them facing up to a £50,000 fine.
Malcolm Lythgo, Head of Waste Regulation at the Environment Agency, said: “We have seen many issues arising from builders not checking who they pay to take their site waste away.
“Failing to deal with construction and demolition waste properly could cause injuries or illness, increase fly-tipping and put your livelihood at risk.
“As England’s waste regulator, the Environment Agency will take action against people who don’t follow their duty of care and so harm people and the environment.
“I would strongly urge all builders to do the right thing and take full responsibility for their waste to limit disruption to the environment as much as possible. You should know who is taking your waste and where it is going.”