But Family and Community Affairs Minister Jocelyn Newman rejected such claims, saying that Mr Herron was simply presenting the facts to rebut exaggerated claims against the government.
Aborigines are outraged at a government report, which suggests that the harmful effects of removing Aboriginal children from their parents have been exaggerated.
The children are known as the "stolen generation".
Conflict
However, a government submission to a senate inquiry says that no more than 10% of children were separated from their families, which it claims is hardly a generation.
"Certainly the Olympic Games will now be in jeopardy," he said.
"We did not want to target the Games, but we have nothing to lose now. We have racism at the highest level of government now, destroying the relationship between the whites and the blacks," he said.
More than 100 Aborginal land councils in New South Wales, which is hosting the Sydney Olympics, have voted to hold protest marches on 16 September, the day the Olympics start.
"Aboriginal people will rise up in this country and show the world how racist Australia is," said Lyall Munro, an Olympic protest campaign delegate to the Metropolitan Land Council.
Regrets
The Minister for Aboriginal Affairs, John Herron, said he regrets the hardship that separation must have caused, but has ruled out any expectations that they government would pay thousands of millions of dollars in compensation.
Some Aboriginal leaders say that the Australian government's attitude is like saying that the Nazi Holocaust did not happen.
But Family and Community Affairs Minister Jocelyn Newman rejected such claims, saying that Mr Herron was simply presenting the facts to rebut exaggerated claims against the government.
"People using the Holocaust for comparison, I think it's disgraceful," she said.
She said any protests by Aborigines during the Olympics would merely hurt Australia's international image.
"They can do it but do you think it's fair to Australia to do that to its image?" she said.
Australia's Prime Minister John Howard has persistently refused to apologise for what is now regarded as a misguided policy aimed at assimilating Aborigines into the white population.