Spurs and its development partner AEG plan to rip-up the athletics track and make the stadium an 80,0000-seater football ground.
The club is on a shortlist with West Ham to takeover the stadium post 2012.
The Hammers want to create a 60,000-capacity arena for football, athletics, concerts and community use in a collaboration with Newham Council.
UK Athletics chairman Ed Warner told the BBC: “It is essential for Tottenham and AEG to go back to the original promise made in 2005.
“That was about UK Athletics being at the heart of the Olympic Park.
“Tottenham and AEG have made it clear that they will rip up the track and make it a football only venue and that is hugely concerning to me.
“The greatest city in the world deserves a facility that is capable of hosting world championships and major athletics events. That is really the legacy we need from the Games for our sport.
“We’ve had sporadic conversations with AEG over the course of the year and one brief conversation with Spurs.
“Anything they might propose for an athletics legacy has to be a compromise to the stadium continuing to operate at a world-class level. To my mind that is completely unacceptable.”
Warner urged Spurs to rethink their proposal.
He said: “There is an important opportunity here to invest in a whole range of sports to sustain them for the future.
“West Ham and Newham have shown that they can embrace that opportunity and work with athletics with a very credible proposal.
“I would like to see Spurs and AEG come up with something equally exciting.”