He visited the ruins of the UN mission, where several senior UN staff were killed. One Danish UN worker was pulled alive from the rubble of the building on Sunday afternoon - one of at least five people who were rescued in Port-au-Prince during the day.
Mr Ban also went to an improvised settlement for survivors opposite the destroyed presidential palace, where people were shouting: "Where is the food? Where is the help?"
Later Mr Ban said he understood people's frustration, but that he did not want to see violence among desperate survivors.
"I appeal to the Haitian people to be more patient," Mr Ban said.
He said providing daily food to two million people, as the UN has pledged, would be a "huge challenge".
"We need to make sure our help is getting to people who need it as fast as possible," he added.
The UN has launched an appeal for $562m (£346m) intended to help three million people for six months, most of whom are thought to need emergency relief.
The UN World Food Programme said it expected to reach about 60,000 people on Sunday, up from 40,000 a day before. A US aid official said he thought US authorities had handed out 130,000 meals, but many more were needed.
The British government is to triple its aid to Haiti to £20m ($37m). The move is to be formally announced at an emergency meeting of EU development ministers in Brussels on Monday.
Roads still blocked
Correspondents say there is a sense of movement at last with the relief effort, although the amount of supplies getting through is still small.
The BBC's David Loyn says the streets of the capital are thronged with homeless people, sleeping in the open and walking for hours for what food and water is available.
Most of the food and water being given out is being distributed informally by local people, correspondents say.
Several agencies or countries complained about not being able to get aid through at the airport, which is heavily congested and has been taken over by the US military.
Medecins Sans Frontieres urged commanders to speed up the landing of aeroplanes carrying medical supplies, after one carrying an inflatable field hospital was turned away on Saturday night.
The head of the US operation at the airport, Col Buck Elton, said there had been 600 take-offs and landings since the US took control on Wednesday, and 50 flights had been diverted.
"What we set up here would be similar to running a major airport... without any communications, electricity or computers," he said.
US troops also said they had set up their first foothold outside the airport to deliver aid carried in by helicopters.
The city's port is badly damaged, and many roads still blocked by corpses and debris.
The Haitian and Dominican Republic governments are planning an alternative 130km (80 mile) humanitarian road corridor to deliver relief supplies from the southern Dominican town of Barahona, the UN said.
We need fuel to bring in supplies and carry the wounded
The UN has warned about fuel shortages, which it says could affect humanitarian operations.
It has also emerged that the US naval base at Guantanamo, Cuba - synonymous with the war on terror - is being used as a staging post for personnel and relief supplies heading to Haiti.
The BBC's Steve Kingstone, at the base, says it has capacity to house up to 10,000 people in tents. While there are no firm plans for that, US commanders say such an evacuation is feasible should it become necessary.
The US and Dutch authorities have said they are speeding up the process of flying orphaned children away from Haiti to adoptive parents abroad.
Six Haitian children adopted by Dutch families arrived in the Netherlands on Sunday and the justice ministry said it was expediting the adoption process and paperwork for about 100 others.
Estimates of how many people died following the 7.0 magnitude earthquake on Tuesday have varied.
The Pan American Health Organization put the death toll at 50,000-100,000, while Haitian Prime Minister Jean-Max Bellerive said 100,000 "would seem a minimum".
A UN official has said aid workers are dealing with a disaster "like no other" in UN memory because the country had been "decapitated".
Three ministers and several senators are reported to have been killed.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.