Mr Zardari - who took over leadership of the Pakistan People's Party (PPP) after the assassination of his wife, Benazir Bhutto - said democracy in Pakistan had come full circle.
He paid tribute to his wife, saying: "I accept the presidency of Pakistan in the name of... Benazir Bhutto. I accept this in her name and in the name of all the martyrs of democracy."
The two presidents pledged to stand together to solve their region's problems.
Pakistan and Afghanistan are like "twins joined," said Mr Karzai. "They are inseparable," he said, adding that they suffered "the same problems, the same evils".
"Afghanistan will be there in each step that you take in our joint struggle for peace and prosperity in the region... in each step that you take in the war against terrorism."
In the past, Mr Karzai has accused Pakistan of not doing enough to stop militants crossing into Afghanistan to launch attacks.
But repairing relations with neighbouring Afghanistan is just one the tasks facing the new president.
His greatest challenge will be how to develop an effective policy to deal with the Islamist insurgency, says the BBC's Barbara Plett in Islamabad.
This will include dealing with an increasingly aggressive American ally, which has stepped up direct strikes against militant targets in Pakistan's border region, our correspondent adds.
'Long live Bhutto'
Mr Zardari's inauguration was attended by diplomats and politicians as well as by high-ranking members of the military, which Mr Zardari and his party have often seen as their enemy, our correspondent says.
It is a very significant day in the history of Pakistan
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.