Tessa Jowell has defended her handling of the 2012 budget against critics who say she will have to "raid" Lottery funds or London's council tax coffers.
Ms Jowell said she was proud of progress that had been made
Tory, Lib Dem and SNP MPs all warned projects across the UK would be hit if more Lottery money was diverted to pay for an estimated £900m overspend.
But Ms Jowell accused the Conservatives of "talking down" the Olympics, having supported it publicly.
She said: "I'm proud of the progress that has been made with the Olympics."
She told the Commons: "What the Conservatives do is to support the Olympics publicly, but then in a narrow and populist way, seek to undermine every reasonable way that this government is trying to make sure that there's a proper and sustainable budget."
Shadow culture secretary Hugo Swire said he had been contacted by "dozens of good causes who face a bleak future due to this government's reckless raiding of the lottery".
He accused Ms Jowell of losing control of the figures and said charities up and down the country would be faced with closure if their lottery funding was axed.
For the Lib Dems Paul Holmes asked for assurances there would be no "smash and grab raid" on the Lottery, over and above the £1.5bn it was already providing.
He said if the £900m overspend was also met through the Lottery, every constituency in the country would lose £500,000 each for local projects over the next five years.
Pete Wishart, for the SNP, added: "Isn't it absurd and totally unfair that homeless projects and children's care groups in Scotland should lose out, to pay for the redevelopment of London's East End?"
Sports Minister Richard Caborn refused to answer "hypothetical" questions, saying discussions were still going on in government, and with the office of the Mayor of London, about the funding and how to deal with the overspend.
But he said a Memorandum of Understanding - which allows for overspend to be covered by a combination of Lottery money and London's council tax - had been the subject of an all-party agreement.
Lib Dem Don Foster said Tony Blair said on 6 November last year that he did not think the government would look for more money from London council tax payers.
But Ms Jowell said while the prime minister was expressing his view, she was involved in negotiations about the fairest way to cover the costs.
"This is the biggest public construction project in Europe, it's like building two Terminal 5s but in half the time.
"I am in charge of a project where we are years ahead of where Sydney and Athens were at an equivalent stage.
"This is how we are going to proceed, by ensuring that the money is there that the costs are controlled and that we in time host the best games ever."