The government says it needs more time to find any Tamil Tiger guerrillas hiding in the camps, and is suspicious of some agencies which it has accused of helping the rebels.
It plans to resettle most refugees with six months.
"We will try to work hard to keep that promise realised," Mr Ban said. "They need to be resettled as soon as possible."
After talks with President Rajapaksa at his residence in Kandy, Mr Ban said: "The government is doing its utmost best", but added that there was a "wide gap between what is needed and what can be done".
Sri Lanka has appealed for $151m in UN funds to improve the camps.
Mr Ban also called for political reconciliation between the majority Sinhalese and minorities, including Tamils.
"If not history could repeat itself", he said.
Without a political settlement that gives Tamils real rights, UN officials believe the fighting will begin anew, says the BBC's Laura Trevelyan, who is travelling with Mr Ban.
On Saturday, Mr Ban also took a low-level helicopter flight over the coastal area where the final battle was fought.
Aid agencies want greater access to the camps for the displaced
"It was a very sobering visit, very sad and very moving," he said of the scene of the battle.
Sri Lanka officially announced an end to the war this week, after its troops took the last segment of land held by the rebels and killed the Tamil Tiger leadership, including its chief, Velupillai Prabhakaran.
It is thought at least 80,000 people were killed in the war.
The UN says 7,000 civilians have died since January alone, although the government disputes this figure.
At a rally before Mr Ban arrived, Mr Rajapaksa dismissed any attempt to take him to an international war crimes court.
"There are some who tried to stop our military campaign by threatening to haul us before war crimes tribunals.
"I am not afraid. The strength I have is your support. I am even ready to go to the gallows on your behalf."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.