The European security body has around 200 staff in Georgia
The Organisation for Security and Cooperation in Europe says it will close its mission in Georgia early in 2009 because of Russian opposition.
Delegates said Moscow refused to back down during a row over the status of the breakaway regions of Georgia.
Russia insists that the disputed regions of Abkhazia and South Ossetia should be recognised as independent.
US Ambassador Julie Finley described Russia's resistance to reaching an agreement as "appalling".
"There is only one party responsible for what has happened and what is about to happen with the shutting down of that mission - it is the Russian Federation," she said.
The US and its European allies in the 56-nation grouping have not recognised South Ossetia as independent.
Georgian Deputy Foreign Minister Giga Bokeria said Russia was challenging not only his country's sovereignty and independence but also international law and institutions.
"It's basically a statement that the Soviet Union is back," he was quoted as saying by the Associated Press news agency.
The conflict between Russia and Georgia began on 7 August.
Georgia tried to retake its breakaway region of South Ossetia by force after a series of lower-level clashes with Russian-backed rebels.
Russia launched a counter-attack and the Georgian troops were ejected from both South Ossetia and Abkhazia days later.
Russian forces remain in the two regions.
The OSCE - which played a key role in brokering a ceasefire between both sides in August - has worked in Georgia since 1992. Both Russia and Georgia are members of the OSCE.
The observer mission's mandate officially ends on 31 December and the OSCE had been seeking a three-month extension.
Ambassador Antti Turunen, of Finland - the current head of the OSCE - said the two countries were too much at odds to reach an agreement by the end of the year so the organisation had no choice but to pull out.
"We had one side defending the territorial integrity of Georgia and the other the 'independence' of South Ossetia. The sides are so far apart it made no sense trying to bridge the gap before 31 December," he said.
The withdrawal of the estimated 200 OSCE personnel is expected to take several months.
Under the terms of the ceasefire, however, some two dozen military monitors will remain until mid February.