The African Union has rejected calls from the UK and the US to put pressure on Zimbabwe to stop its demolition of illegal houses and market stalls.
The opposition say their supporters are being punished
An AU spokesman told the BBC that it had many more serious problems to consider than Zimbabwe.
The UN says that 275,000 people have been made homeless. At least three children have been crushed to death.
Urging the AU to take action, US Secretary of State Condoleezza Rice described recent events as "tragic".
The opposition say the demolitions are meant to punish urban residents, who have rejected President Robert Mugabe in recent elections.
He denies this, saying the crackdown is designed to "restore sanity" in urban areas, which he says have become overrun with criminals.
"If the government that they elected say they are restoring order by their actions, I don't think it would be proper for us to go interfering in their internal legislation," AU spokesman Desmond Orjiako told the BBC's Network Africa programme.
His comments were backed up by South Africa, Zimbabwe's giant neighbour, which some see as the key to solving Zimbabwe's problems.
Presidential spokesman Bheki Khumalo said he was "irritated" by calls from UK Foreign Secretary Jack Straw to do more to end the "horrors" in Zimbabwe.
"South Africa refuses to accept the notion that because suddenly we're going to a G8 summit, we must be reminded that we must look good and appease the G8 leaders," he said.
"We will do things because we believe they are correct and right."
The G8 summit of the world's most powerful nations is due to discuss efforts to relieve poverty in Africa on 8 July in Scotland.
The UK wants the G8 to do more to forgive Africa's debts and improve its term of trade.
The US insists that such efforts should only be made if African countries improve their standard of governance.
Correspondents say that many African leaders see Mr Mugabe as a hero for leading the fight against colonial rule.
South Africa's President Thabo Mbeki has questioned why the West is so concerned by Zimbabwe but makes relatively little noise about other African emergencies, such as the Democratic Republic of Congo, where some three million people died in a civil war, and where armed bands kill, rape and loot with impunity in some areas.
The BBC's Elizabeth Blunt also says that many African countries have carried out similar slum clearances and so will be unwilling to criticise Zimbabwe.
Some 46,000 people have been arrested for trading without a licence, hoarding and illegal possession of foreign currency, Zimbabwe's police chief Augustine Chihuri said, according to state radio.
Mr Chihuri said that burglary and car-theft had declined by 20% since the operation began four weeks ago.
The children who have died were crushed to death when their homes were knocked down during Operation Murambatsvina [Drive out rubbish].
Some children have left school after their homes were demolished
One of those killed in the capital, Harare, was the 18-month-old son of a police officer, reports the state-run Herald newspaper.
The police have moved across Zimbabwe's urban areas, armed with bulldozers and sledge-hammers, destroying shacks and informal markets.
Often, residents have been made to demolish the structures themselves.
The United Nations is due to send a special envoy to Zimbabwe to investigate the demolitions.
Many people are living on the streets, while others have returned to their rural homes, encouraged by the government.