Charities and sports bodies in Wales could lose over £24m to pay for the 2012 London Olympics, it is claimed.
Lottery funding will contribute to the cost of the Olympics
The scale of possible cuts for charities is spelt out in a letter from Welsh Culture Minister Alun Pugh to his Westminster equivalent, Tessa Jowell.
A lottery body has warned some projects could go to the wall.
But the Department of Culture, Media and Sport (DCMS) said they understood concerns and were working to "minimise the impact on good causes in Wales."
The letter was obtained by Plaid Cymru under the Freedom of Information Act.
The UK Government has said it may use some National Lottery funds to make up for a shortfall in the final budget for the London Olympics in 2012.
Mr Pugh is concerned this may affect the money available for Welsh good causes.
His letter to Tessa Jowell, seen by the BBC Wales' Dragon's Eye programme, predicts that Wales faces losing between £24m and £36m.
He said: "The fall in income will have a significant impact on the ability of distributors in Wales to meet their requirements and to fund worthy projects across the good causes.
"Any increase in the National Lottery's commitment to the 2012 Olympic Games and the Paralympic Games will inevitably impact on Welsh distributors and Welsh good causes."
Mr Pugh said the Olympics represented good news for Wales overall, as athletes were set to use venues such as Cardiff's Millennium Stadium, Newport's velodrome and Swansea's 50m swimming pool.
Olympic athletes are expected to train in Wales prior to the Games
He said some 500 Welsh business also stood to benefit from contracts with the Olympics bid.
He added: "The Olympics overall is great news for Wales, and we'd love to see some Welsh athletes have some gold medals around their necks.
"We're just sounding an early word of caution about potential cost overruns on the Olympics and who actually picks up the tab for those cost overruns.
"If all cost overruns fell on the lottery, that could have quite a significant impact on sporting, artistic and cultural projects across Wales and that would be quite disappointing to see that happening."
The Big Lottery Fund, the organisation which distributes lottery money to good causes, has warned that funds being diverted to Olympics projects could mean cutting off aid to small community projects.
Ceri Doyle, Wales director of the Big Lottery Fund, said money should not necessarily be taken from charities.
She said: "At the moment we take 20p in every pound spent on lottery tickets that goes towards good causes.
"There's 12p already going to the treasury.
"Even the culture select committee has said 'can this 12p in every pound not be used to support some of these escalating costs that are coming forward?'
"I can't give a solution but I think there are other means and other options that we should be considering."
Graham Benfield, chief executive of the Wales Council for Voluntary Action, added: "The government has already taken two big slices of lottery money.
"Firstly, the lottery has already contributed £1.5bn to the Olympics infrastructure and out of every pound, when you buy a lottery ticket, 12p is already going to the treasury in tax.
"That 12p in every ticket could go to the Olympics.
"As far as voluntary and community organisations are concerned...£30m-£40m may be lost and that will have an appalling and devastating effect on many organisations.
"There will be a reduction in the programmes and some organisations will probably go to the wall."
Plaid Cymru AM Owen John Thomas said it was "staggering" that Labour "haven't had this checked out".
A spokesman for the Department of Culture Media and Sport said they understood the concerns about the possible impact on the lottery but they would "continue to work with Welsh ministers to minimise the impact on good causes in Wales".
The spokesman said: "We accept that using National Lottery funding for London 2012 represents a loss of income to non-Olympic good causes in the short-term.
"However, we remain convinced that the benefits the Games will bring will far outstrip any such effect."
He added that discussions were continuing about how the additional costs would be funded but it would be "wrong for us to speculate on how any shortfall should be met at this stage."
Dragon's Eye is screened on Thursday on BBC2W at 2030 GMT and on BBC 1 Wales at 2235 GMT.