The government has said it may dip into National Lottery funds to make up for a shortfall in the final budget for the London Olympics in 2012.
The government says the cost of the Olympic park has risen
Big Lotto Fund chairman Sir Clive Booth told Channel 4 News that 86,000 grants, totalling up to £950m, would be threatened if more money was taken.
The Liberal Democrats said the Lottery should not be used as "a Treasury slush fund for Olympic overruns".
But the government said the benefits would "far outstrip" the costs.
Sir Clive said: "I think part of the problem is that a lot of Whitehall doesn't really understand the Lottery and so they just assume that the lottery is this enormous pot of money which keeps generating huge amounts of cash - and that you can just help yourself to some of it, and that there are no adverse consequences."
Culture Secretary Tessa Jowell has told MPs that the cost of London's Olympic park will increase by £900m, from £2.375bn to £3.3bn.
The Department for Culture, Media and Sport (DCMS) said a revised rundown of the cost of the games was expected "in this quarter".
This would include expenses such as materials, security, possible VAT liability and contingency plans.
A DCMS spokesman said: "The London 2012 Olympic and Paralympic games provide a once-in-a-lifetime opportunity to improve society and transform the lives of millions.
"Whether it be in terms of regenerating of one of the most deprived areas in the country, inspiring a generation of youngsters to get active and take up sport, or creating a legacy of world class facilities for the whole country to use, the 2012 games will bring immense benefits to this country.
"Exactly the sorts of benefits the Lottery was set up to deliver.
"We accept that using National Lottery funding for London 2012 represents a loss of income to the non-Olympic good causes.
"However, we remain convinced that the benefits the games will bring will far outstrip any effect created by such a diversion."
But Liberal Democrat culture spokesman Don Foster said: "The Lottery is already funding a large proportion of Olympic costs.
"Any further raids of Lottery cash could seriously damage funding for arts, heritage, sports and charities.
"We all agreed the Lottery should bear the burden of some of the costs of these games, but it must not be seen as a Treasury slush fund for Olympic overruns."