Mr Mandela now steps down after two years as mediator
A BBC correspondent in Bujumbura says there has been a distinct lack of gunfire.
Although fighting continues between the army and Hutu rebel groups, the new administration, incorporating both the majority Hutus and minority Tutsis ethnic groups, has been portrayed as a vital step towards ending the eight year civil war.
Mr Mandela, who brokered the agreement, told the new government on Thursday that much remained to be done before peace is finally established in Burundi.
"Your country has bled enough. It and its people now deserve peace," he said.
Mr Mandela now steps down as mediator after two years but promised to remain a "moral guarantor" of the peace deal.
The secretary-general of the newly formed, African Union, Amara Essy, said Burundi's new transitional government represents a big step towards peace in the country.
Buyoya
Burundian President Pierre Buyoya, who is a Tutsi, will remain in office for the first 18 months before handing over to a Hutu.
Speaking at the swearing in ceremony in front of Mr Mandela, President Buyoya condemned the pressure that had been put on Burundi's leaders.
He said in other parts of the world negotiations begin with a ceasefire deal but in Burundi the opposite happened.
This had resulted, he said, in a peace accord which had not managed to bring peace.
People's senate
A new cabinet has been approved by Burundi's Parliament and 14 out of the 26 portfolios will go to Hutus and 12 to Tutsis.
The Tutsis hold the positions of defence, foreign affairs and finance whilst the Hutus keep the ministries of public security and interior.
Under the new arrangements, the legislature would consist of the National Assembly with 60% of seats reserved for Hutus and 40% for Tutsis.
For the first time Burundi will also get a senate which will consist of prominent citizens from both ethnic groups.
But the new government still has to tackle the problem of the two main Hutu rebel groups who have said they will contine to wage war against the Tutsis, who have dominated Burundi politics for nearly 40 years.
Security has been tight in Bujumbura and South African troops are on the ground to ensure the security of the mainly Hutu politicians returning from exile.
Some hardline Tutsis have urged their supporters to attack the South African protection force which is expected to be in Burundi until June next year.
But a spokesman for the South African defence forces said any steps necessary would be taken to protect South Africa's troops.
The war between the Tutsi-dominated government and the majority Hutu rebels was started in 1993 following the death of an elected Hutu president.
It has cost the lives of hundreds of thousands of people, and caused massive suffering.