One reason for the failure to reinstate him is thought to be the fact that he challenged an amnesty given by Gen Musharraf that enabled Mr Zardari to return to Pakistan, on the grounds that Gen Musharraf's own rule could be illegal.
Overturning the amnesty could leave Mr Zardari, the widower of former Pakistan Prime Minister Benazir Bhutto, exposed to corruption charges.
Mr Gilani's announcement, broadcast on television, triggered scenes of jubilation from Mr Chaudhry's supporters outside his home in Islamabad.
The BBC's Barbara Plett, who was at the scene, says the gathered lawyers and activists were in exultant mood, regarding this as the triumph of two years of struggle against both military and civilian governments.
Mr Gilani also said opposition activists and leaders detained over the past week of mounting disturbances would be freed and a ban on demonstrations in the capital and several provinces lifted.
"This is a victory for the people of this country," said Baz Mohammad Kakar, a leader of the lawyers' movement.
"Chaudhry is the first chief justice in the history of Pakistan who has proved himself to be a judge for the people, as a chief justice for the people."
Our correspondent says the development is also a victory for Mr Sharif, a long-time opponent of President Zardari.
He was mobbed by supporters in Gujranwala, about 80km (50 miles) north-west of Lahore, telling them: "We have said that we will restore the judges and the independent judiciary and by the grace of Allah we have achieved it.
"From here, a journey of development will start. From here, a revolution will come."
Our correspondent says that everyone, including Pakistan's Western supporters, will be heaving a sigh of relief that what looked to have been a dangerous political confrontation appears to have been defused peacefully.
The West wants Pakistan to focus on the battle against the Taleban on the Afghan border.
The past two days have provided fresh evidence of the militant insurgency there, with attacks on Sunday and Monday on container terminals near Peshawar in the north-west. The terminals supply Nato and US troops in Afghanistan.
The decision to reinstate the chief justice marks a complete turnaround from the situation on Sunday, when Mr Sharif left Lahore with a convoy of supporters, defying an apparent effort to put him under house arrest.
Iftikhar Chaudhry and other judges were sacked by Gen Musharraf
Riot police had surrounded his Lahore home but after supporters clashed with them Mr Sharif moved past unchecked and left the city.
Several days of rallies around Pakistan were to culminate in a "long march" on Islamabad and a sit-in to press the demand for the reinstatement of the judges.
The government's announcement marks the end for the time being of the power struggle between Mr Sharif and Mr Zardari, says our Islamabad correspondent.
There are indications that other issues that inflamed tensions between the two will be resolved: a Supreme Court decision last month that banned Mr Sharif and his brother Shahbaz from elected office, and President Zardari's decision to put their stronghold in Punjab province under direct rule from Islamabad.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.