Security arrangements for the Games have yet to be finalised
Organisers of the London Olympics have been warned they still face "formidable challenges" which could see costs rise.
Unsigned construction contracts, the legacy of the games and security could all affect the £9.3bn budget, the National Audit Office (NAO) said.
The Olympic Delivery Authority (ODA) was urged to maintain a "firm stance" on costs as plans developed.
The Department for Culture, Media and Sport said the budget had not increased "and will not be allowed to increase".
Similarly an ODA spokesman said any further public investment would be "within the existing £9.3bn budget".
The NAO's warning came as similar concerns were highlighted in a review of preparations for the games, commissioned by mayor Boris Johnson.
In the latest report the head of the NAO, Tim Burr, noted: "The preparations for the games are well under way."
But he said: "Important challenges remain which will become more formidable as the spotlight turns to London after the Beijing Games."
It highlighted the lack of a deal for construction of the Olympic village in east London, which is expected to cost more than £1bn.
Also noted was that plans for individual venues, including the main stadium, after 2012, were yet to be finalised.
The NAO said there was no "firm basis" for security arrangements and costed plans from the government were not anticipated until the end of the year.
All these factors, the report said, "may add cost, or compromise the preparations for a successful games".
The report warned: "There will be a risk of cost overruns and loss of time unless a firm line is adopted on subsequent calls for changes in the designs for the venues and infrastructure."
An ODA spokesman said the level of public investment in the Olympic village would be finalised later this year.
"It is possible, given the economic situation, that more public investment may be needed," he added.
"However, this would be within the existing £9.3bn budget for the games so it is incorrect to say that taxpayers would pay more."
A spokesman for the Department for Culture, Media and Sport said it was "inevitable" in "a project of this complexity" that estimated costs would both rise and fall.
"Some, for infrastructure, are currently lower than expected, but we are working to reduce any that may be rising at this time," he added. "Individual venue budgets remain unaltered."