Rebels say Ndayizeye (l) is just a figurehead
The United States and Nelson Mandela have both welcomed the peaceful transition in Burundi, where an ethnic Hutu has become the new president.
Domitien Ndayizeye was sworn in on Wednesday and promised to fight against genocide and to advance the peace process with rebels.
He replaces an ethnic Tutsi, Pierre Buyoya, and reassured worried Tutsis that their security would be guaranteed.
Tutsis, who account for 15% of the population, have traditionally dominated Burundi and some fear revenge under a Hutu president.
Ethnic Hutu rebels, however say that Mr Ndayizeye is just a figurehead, while the Tutsis continue to control the army.
"The transition of the presidency in Burundi is a significant achievement on Burundi's road to a peace," said United States state department spokesman Richard Boucher.
Former South African President Nelson Mandela said that Burundi could show the way forward at a time when "the mighty may take it upon themselves to decide how the world and individual countries should be governed or run", taken to be a reference to the United States-led war in Iraq.
War began: 1993
Tutsis have dominated since independence
"This tiny African country can strike a blow for multilateralism and dialogue if it can prove that negotiations and talks can lead to peaceful coexistence," he said.
African Union Secretary-General Amara Essy said that such peaceful handovers were rare in Africa.
"This is a day of hope for all Africans," he said.
It is the fourth time in 16 years that a Hutu president has been sworn in. Previous attempts at Hutu rule have ended in military take-overs.
This transition is part of a peace deal which is supposed to lead to an end of the 10-year civil war in the country, culminating in elections in 18 months.
But the Forces for the Defence of Democracy (FDD) rebels remain outside the peace agreement and have carried on fighting the war which has left 300,000 people dead.
"This is meaningless. There is no difference between Buyoya and Domitien. It's the same system, the same government," Gelase Ndabirabe, the FDD spokesman told Reuters news agency.
Bujumbura residents live in constant fear of rockets
"We call on the Burundi people not to be distracted by the ceremony."
The BBC's Ishbel Matheson in Bujumbura says that Burundi's new president will be under pressure from both the Hutu rebels and the Tutsi-dominated army.
But she says there may be a change of mood in the country.
Among ordinary Burundians, ethnic divisions seem less important as all have shared in the suffering.
Some young Tutsis say they did not care who ruled the country as long as he brought peace and security.