Mr Zardari - who took over leadership of the Pakistan People's Party (PPP) after the assassination of his wife, Benazir Bhutto - said democracy in Pakistan had come full circle.
He paid tribute to his wife, saying: "I accept the presidency of Pakistan in the name of... Benazir Bhutto. I accept this in her name and in the name of all the martyrs of democracy."
The two presidents pledged to stand together to solve their region's problems.
Pakistan and Afghanistan are like "twins joined," said Mr Karzai. "They are inseparable," he said, adding that they suffered "the same problems, the same evils".
"Afghanistan will be there in each step that you take in our joint struggle for peace and prosperity in the region... in each step that you take in the war against terrorism."
In the past, Mr Karzai has accused Pakistan of not doing enough to stop militants crossing into Afghanistan to launch attacks.
But repairing relations with neighbouring Afghanistan is just one the tasks facing the new president.
His greatest challenge will be how to develop an effective policy to deal with the Islamist insurgency, says the BBC's Barbara Plett in Islamabad.
This will include dealing with an increasingly aggressive American ally, which has stepped up direct strikes against militant targets in Pakistan's border region, our correspondent adds.
'Long live Bhutto'
Mr Zardari's inauguration was attended by diplomats and politicians as well as by high-ranking members of the military, which Mr Zardari and his party have often seen as their enemy, our correspondent says.
It is a very significant day in the history of Pakistan
Mr Zardari's son Bilawal - co-chairman of the PPP - was in the audience with his two sisters Bakhtawar and Asifa and Benazir Bhutto's sister Sanam, who shed tears as the ceremony was about to begin.
Mr Zardari was greeted by applause and a fanfare as he entered the hall. After he took the oath there were cries of "long live Bhutto.
Controversy
Pakistan's president is elected by secret ballots in the national and four provincial assemblies.
Mr Zardari won 481 votes out of 702, far more than the 352 votes that would have guaranteed him victory, leaving his two rivals trailing far behind.
Mr Zardari was thrust into the centre of political power by the killing of Ms Bhutto last December after which he became head of the PPP.
He is one of Pakistan's most controversial politicians.
For years he has been hounded by allegations of massive corruption - although he has never been convicted.
Former Prime Minister Nawaz Sharif took his PML-N party out of the governing coalition last month, accusing Mr Zardari of breaking key promises.
Many in Pakistan fear the country is facing a return to an old-style politics of confrontation.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.
Bookmark with:
What are these?