Readstone Construction is main contractor on the £2m Accrington Stanley community sports hub.
Fences were erected around the site two weeks ago but vandals have been a constant menace ever since.
Director Simon Whittam told the Lancashire Telegraph: “As soon as the fences went up we started to experience vandalism.
“We’re not due to start work on the site just yet, but we marked out the boundaries and put the fences up in preparation and to stop people accessing the site.
“But every morning all of the panelling has been knocked down and several of our panels have been damaged.
“It’s a big waste of time for us having to go down there every day to put them all back up again.”
The contractor is now considering placing security guards at the site and Accrington Stanley football club has appealed to local residents to report any further problems.