London 2012 officials have insisted there is no health risk after learning part of the planned Olympic Park is on the site of a former nuclear reactor.
Conservative politicians have called for a fresh survey at the site in East London's Lower Lea Valley.
But a London 2012 spokesman said the London Development Agency had already carried out a full environmental impact assessment.
"It showed no evidence of any nuclear contamination," said the spokesman.
Details of the reactor were revealed by a Conservative member of the London Assembly at an assembly meeting on Wednesday where 2012 team spokesman Mike Lee was questioned about the Olympics.
Bob Blackman, the Tory economic development spokesman, said: "During this process, officers at the bid team have been very blase about the problems of contamination in the Lower Lea Valley, but they must take this matter seriously.
"We have got a huge amount of building to do and this work cannot even start until we clean the site up."
NUCLEAR REACTOR FACTS
Location: Main college campus, Mile End, then moved to Marshgate Lane. Decommissioned 1982
Reactor: Was used for undergraduate experiments and postgraduate projects
Training: One of three centres which trained people for work at nuclear power stations. Others were Navy bases at Greenwich and near Ascot
It is understood bid officials did not know about the reactor until Mr Blackman raised the issue.
But they are satisfied the area was fully decontaminated 23 years ago when the reactor was decommissioned.
The reactor was used by Queen Mary College's department of nuclear engineering, which is now defunct, until 1982.
In a statement, the college said: "The reactor was exceptionally small; the core was the size of a bucket and produced virtually no energy.
"Decommissioning staff were able to stand inside the reactor void with no protective clothing."
London beat four cities to win the right to host the Games, and its bid won cross-party support.
Tories say they did not raise the nuclear reactor issue earlier in case it damaged the capital's bid.
And Mr Blackman believes a fresh survey of the site will help to resolve the matter.
But London 2012 insiders feel he is being "mischievous", and officials have already pledged to carry out any clean-up work - where it is needed - on former industrial sites.
Although many sports fans and Londoners welcome the Games being held in London, there has been opposition.
Businesses at Marshgate Lane in Newham, where the new Olympic Stadium will be built, have been unhappy with relocation plans.