Two Palestinians were killed and buildings were damaged during the operation which involved tanks and bulldozers, Palestinian witnesses said.
A BBC correspondent says that Israel was showing it holds the Palestinian leader directly responsible for suicide attacks such as the one on Wednesday which killed 17 people in northern Israel.
A couple of hours after the Israeli forces left, Mr Arafat emerged from the compound flashing the victory sign and pledging defiance.
He was greeted by about 100 cheering civilians and security guards.
"This will only increase the steadfastness of our people," he said.
In Jerusalem, Israeli President Moshe Katsav called on the outside world to break off ties with Mr Arafat, saying he had become a "danger to himself".
For its part, the Bush administration on Wednesday condemned the suicide bombing and strongly criticised the Palestinian leader.
Spokesman Ari Fleischer said: "In the president's eyes, Chairman Arafat has never played the role of someone who can be trusted or who is effective."
Buildings destroyed
Journalists entering Mr Arafat's compound after the withdrawal reported extensive damage, with sections of the exterior wall and some offices demolished.
The column of about 50 Israeli military vehicles has now returned to Israeli army bases on the edge of Ramallah.
In a statement, the Israeli military said its forces had taken control of Mr Arafat's headquarters "in the wake of a wave of Palestinian terrorism sweeping the state of Israel".
It also accused the Palestinian Authority of being "directly responsible for terrorism that originates in its territory".
The BBC's Jeremy Cooke says that Israel will feel it has made a strong point without significantly escalating the intensity of its military operations in the West Bank.
But for the Palestinians, our correspondent says, this will be regarded as another gross and unjust violation of their sovereignty.
In March, Israeli tanks entered Mr Arafat's compound and surrounded his office, trapping the Palestinian leader inside a few rooms without a water supply or electricity for more than a month.
Car bomb
Correspondents say the detonation of a moving car packed with explosives behind a bus in the morning rush hour on Wednesday at Megiddo Junction marked a new and more deadly approach by suicide bombers.
The Palestinian Authority denounced the attack and rejected Israel's charge that Mr Arafat was directly responsible for it.
The bombing, which killed 13 soldiers, three civilians and the bomber himself, was claimed by the radical Islamic Jihad group, which said the bomber came from Jenin.
The group said the attack was timed to coincide with the 35th anniversary of the 1967 Six-Day War, when Israel captured the West Bank and Gaza Strip.
Diplomatic efforts
Israeli Prime Minister Ariel Sharon delayed by one day his visit to Washington to hold more talks on the Middle East situation with President George Bush.
Egyptian President Hosni Mubarak, also on a visit to Washington, is due to meet administration officials on Thursday and Mr Bush on Friday.
Mr Mubarak is trying to press the US to recognise a Palestinian state on the basis of the pre-1967 borders.
Reports say Washington is considering launching a peace plan that would set a timetable for the creation of a viable Palestinian state.
Correspondents say Mr Sharon is keen to avoid being boxed in by a timetable set by the US or a peace process that requires him to make territorial concessions.