There is deep suspicion between the Hutus and Tutsi
Although fighting continues between the army and Hutu rebel groups, a new administration, incorporating both the majority Hutus and minority Tutsis ethnic groups, is seen as a vital step towards ending eight years of civil war.
"But no one can deny that the establishment of a... transitional government in this deeply divided country represents a major breakthrough and advance on the road to lasting peace," he said.
"Your country has bled enough. It and its people now deserve peace," he said.
Mr Mandela now steps down as mediator after two years but promised to remain a "moral guarantor" of the peace deal.
Buyoya
Burundian President Pierre Buyoya, who is a Tutsi, will remain in office for the first 18 months before handing over to a Hutu.
Speaking at the swearing in ceremony in front of Mr Mandela, President Buyoya condemned the pressure that had been put on Burundi's leaders.
He said in other parts of the world negotiations begin with a ceasefire deal but in Burundi the opposite happened.
This had resulted, he said, in a peace accord which had not managed to bring peace. People's senate
A new cabinet has been approved by Burundi's Parliament and 14 out of the 26 portfolios will go to Hutus and 12 to Tutsis.
The Tutsis hold the positions of defence, foreign affairs and finance whilst the Hutus keep the ministries of public security and interior.
Under the new arrangements, the legislature would consist of the National Assembly with 60% of seats reserved for Hutus and 40% for Tutsis.
For the first time Burundi will also get a senate which will consist of prominent citizens from both ethnic groups.
But the new government still has to tackle the problem of the two main Hutu rebel groups who have said they would contine to wage war against the Tutsis, who have dominated Burundi politics for nearly 40 years.
Security has been tight in Bujumbura and South African troops are on the ground to ensure the security of the mainly Hutu politicians returning from exile.
Some hardline Tutsis have urged their supporters to attack the South African protection force which is expected to be in Burundi until June next year.
But a spokesman for the South African defence forces said any steps necessary would be taken to protect South Africa's troops.
The war between the Tutsi-dominated government and the majority Hutu rebels was started in 1993 following the death of an elected Hutu president.
It has cost the lives of hundreds of thousands of people, and caused massive suffering.