Mr Galbraith had been critical of the Afghan election commission
The UN has recalled one of its senior officials from Afghanistan following a row over the August presidential vote.
Peter Galbraith fell out with top UN envoy Kai Eide and angered Afghan President Hamid Karzai by reportedly calling for a complete recount.
The election has been overshadowed by widespread allegations of fraud.
A UN statement said Mr Galbraith was being removed from his post "in the best interest of the mission". It reaffirmed support for Mr Eide.
Two weeks ago Mr Eide, who leads the UN mission in Kabul, said his deputy had left the country after a row between them. But he denied he had ordered him to go.
PETER GALBRAITH: KEY DATES
1979-1993: Senior adviser to US Senate Committee on Foreign Relations
1993-1998: First US Ambassador to Croatia, and co-author of Erdut Agreement that ended the war in Croatia
2000-2001: Director of Political, Constitutional and Electoral Affairs for the UN Transitional Administration in East Timor
2003: Resigns from the US government to write The End of Iraq: How American Incompetence Created a War Without End
UN sources say Secretary General Ban Ki-moon decided to end Mr Galbraith's mission after it became clear he was no longer able to carry out his work in Afghanistan, says the BBC's Lyse Doucet.
Some Afghan cabinet ministers had said they no longer wanted to work with him.
It is understood that Mr Galbraith would have been kept in his post until after a final ruling on the disputed presidential election - a process that is in its final stages - but leaks emerged in Kabul before Mr Galbraith himself had been informed of the secretary general's decision, Lyse Doucet says.
The statement issued by Mr Ban's office in New York thanked Mr Galbraith "for his hard work and professional dedication".
"The secretary general has made this decision in the best interest of the mission. He reaffirms his full support for his special representative, Kai Eide," it added.
Last week, Mr Eide told the BBC the dispute had been resolved by Mr Galbraith agreeing to leave the country for a while.
He described Mr Galbraith as "a valuable deputy" and said he hoped they could "re-establish a good team and work together".
Mr Eide declined to talk about details of his disagreement with Mr Galbraith, but said the UN should respect the constitutional bodies in charge of the presidential election "to avoid any impression that there is foreign interference".
AFGHAN POLL FRAUD
15 Sep:Election Complaints Commission +chief says 10% of votes need to be recounted
8 Sep: Poll complaints body orders some recounts nationwide
8 Sep: IEC says votes from 600 polling stations "quarantined"
3 Sep: Claims 30,000 fraudulent votes cast for Karzai in Kandahar
30 Aug: 2,000 fraud allegations are probed; 600 deemed serious
20 Aug: Election day and claims 80,000 ballots were filled out fraudulently for Karzai in Ghazni
18 Aug: Ballot cards sold openly and voter bribes offered
The row is between two men who have known each other for a long time but have very different styles, but a UN source said that had not been the only factor in Mr Galbraith's removal, Lyse Doucet says.
It is understood that Mr Ban would not have dismissed Mr Galbraith - who came to the post with US support - without backing from the Washington, she adds.
The US, along with other foreign missions in Afghanistan, appears to want to move on from the election dispute to deal with the country's other considerable problems, she says, but this will anger observers who believe a more robust response is needed to the allegations.
EU election observers have said that about 1.5m votes - about a quarter of all ballots - cast in August's presidential vote could be fraudulent.
They say that 1.1 million votes cast for President Karzai are suspicious.