Mr Mugabe earlier administered the oath of office to Mr Tsvangirai at a ceremony in Harare.
Two deputy prime ministers were also sworn in: Thokozani Khupe, the deputy leader of Mr Tsvangirai's Movement for Democratic Change (MDC), and Arthur Mutambara, the leader of a break-away faction of the MDC.
Zimbabwean journalist Brian Hungwe told the BBC that some army commanders attended the swearing-in ceremony in plain clothes.
In the run-up to the March 2008 elections, army commander General Constatine Chiwenga said he would only serve Mr Mugabe.
Political analyst Ibbo Mandaza told the BBC's Focus on Africa programme: "I was hoping to see them [the army] saluting Morgan Tsvangirai. I think in due course they will salute."
The new prime minister must deal with an economy in ruins, 90% unemployment and a cholera epidemic which has killed more than 3,400 people.
Hyperinflation is causing prices to double every day and the country stopped publishing inflation figures after it was last estimated at 231m%. People are using foreign currency wherever possible.
The cholera epidemic broke out in August 2008
More than half the population rely on emergency food aid.
The cabinet in the new coalition government will be sworn in on Friday.
There is deep scepticism about whether it will work, says the BBC's southern Africa correspondent, Peter Biles, in Johannesburg.
At best it will be a transitional arrangement leading eventually to a new constitution and fresh elections, he says.
UK Foreign Secretary David Miliband cautiously welcomed the appointment of Mr Tsvangirai as a "step forward", but was worried that Mr Mugabe remained as president.
"While Morgan Tsvangirai is acknowledging the crowds, behind him is a lurking figure and that figure is President Mugabe, who has tyrannised that country and bought it to its knees," he said.
The international community stood ready to offer additional aid to Zimbabwe but it depended on the actions of the new government, he said.
A final deal on power-sharing was reached in January, after Mr Tsvangirai returned to Zimbabwe following an absence of more than two months for fresh talks with Mr Mugabe.
Earlier negotiations had faltered after the MDC accused Mr Mugabe's Zanu-PF of keeping the most powerful ministries - including the one that controls the police - to itself.
A minister from each party will now share the home affairs ministry.
Other important ministries the MDC will control are health, education and finance.
On Tuesday, Mr Tsvangirai named Tendai Biti, secretary general of the MDC as his choice for finance minister.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.