Arts, sports and community groups have expressed anger that more Lottery funds are to be diverted to the Olympics.
Lottery funds will go towards the 2012 Olympics
Culture Secretary Tessa Jowell has announced an increased budget for the London 2012 Games of £9.3bn.
The funding will now include £675m more from the National Lottery, bringing its contribution to £2.2bn.
The Arts Council England, Heritage Lottery Fund and Sport England were among groups who were disappointed at the increase.
Ms Jowell said she planned to transfer £425 million from the Big Lottery Fund and £250 million from other good causes after 2009.
Sport England, which will lose £55.9m in funding, said the decrease was a "a real blow to community sport in England".
Chairman Derek Mapp said although the agency gave its full support to the Games, the reduction "seriously endangers the creation of a sporting legacy from the 2012 Games."
For the Arts Council England, chief executive Peter Hewitt said he was "deeply disappointed" that funds were being diverted.
"The impact is likely to be felt across the whole of England and disproportionately by smaller arts organisations, local projects and individual artists - the main recipients of our Lottery funds and precisely the sorts of people who may be asked to contribute to the cultural Olympics."
Britain's heritage would also suffer, said Liz Forgan, chairwoman of the Heritage Lottery Fund.
Cabinet Office minister Ed Miliband sought to reassure voluntary groups that their funding would not be cut because of Olympic requirements.
He said the government had taken into account concerns from groups over resources taken from the Big Lottery Fund.
"Today's funding settlement protects both existing programmes and future resources for the voluntary sector.
"After today's announcement, the voluntary sector will get the same level of resources from the Big Lottery Fund that they would otherwise have received.
"Instead, the Olympic transfer will come from that portion which was expected to go to statutory agencies."
The National Council of Voluntary Organisations said it welcomed such commitment.
Director of public policy Liz Atkins said: "We think that's very good news.
"But of course, we do need firm details from the government outlining how they will ensure that charities and community groups don't suffer, and receive the funding they need to carry out their vital work in local communities."
Big Lottery Fund chairman Sir Clive Booth told the BBC: "Frankly, it could have been far worse.
"When I go back to the beginning of February and the numbers we were looking at in terms of the increase in costs and what that could have meant in the terms of impact on us, then this outcome is much much better than what we were looking at."