But despite their pull-back from the compound itself the Israelis remained stationed nearby and re-imposed a curfew on the town.
Raanan Gissin, a spokesman for Israeli Prime Minister Ariel Sharon, said the troops would distance themselves from the compound in such a way as to prevent several wanted men believed to be inside it from leaving.
Meanwhile Mr Sharon arrived in Moscow for a three-day visit.
Russia is expected to voice its concern to him over the escalation of violence in the Middle East and sound out Israel's position on Iraq.
The Israeli leader arrived with three Russian-born survivors of the Tel Aviv disco bombing in June 2001 in which 21 people died.
'Playing games'
In Ramallah, Mr Arafat said nothing but blew kisses to supporters gathered outside and made a V for victory sign as he was paraded around the compound on the shoulders of his guards.
Earlier, he accused the Israelis of playing games with the international community.
Reading out the text of a UN Security Council resolution passed last week calling for the lifting of the siege, he said the terms of the resolution had not been met.
"The Israelis must withdraw immediately from not only the compound but all the Palestinian cities," he said.
However, some Palestinian officials described the easing of the siege as a victory and the consequence of Mr Arafat's refusal to bow to pressure and leave his offices.
Troops nearby
Israeli tanks began to move away from the compound on Sunday afternoon local time.
But the AFP news agency reported that they returned in smaller numbers later in the evening and announced a curfew over loudspeakers.
Palestinians from outside wandered into the compound, including some young activists who stood near the office and chanted.
But the BBC's Jeremy Cooke says it is not clear how the wanted men will be detected among the crowds when they try to leave.
One guard suggested that some may have already disappeared during the confusion.
Israeli troops besieged the compound after two suicide bombs 10 days ago and have systematically destroyed many of the buildings, leaving just three buildings still standing.
"The siege around Arafat will be lifted, but the wanted suspects will continue to be wanted suspects," said Israeli Tourism Minister Yitzhak Levy.
There are suggestions that a deal like the one to end the siege of the Church of the Nativity in Bethlehem in May could be reached.
That would mean the men would be expelled from the West Bank - probably to the Gaza Strip.
Correspondents say the decision to relax the siege is the result of pressure from the United States.
US President George W Bush reportedly wrote to Mr Sharon at the weekend demanding an end to the siege.
Some reports also say Foreign Minister Shimon Peres had made a veiled threat to resign if the siege was not lifted.