He wanted to present the evidence to the Afghan Election Complaints Commission for further investigation, he said, but Mr Eide "did not want this information disseminated".
Mr Galbraith said that when he intervened, President Hamid Karzai complained and Mr Eide "decided he would support Karzai, who would be the beneficiary of the fraudulent ballots".
He said Mr Eide had initially "tended to dismiss the fraud".
"He didn't want the UN staff to talk about it, he didn't want us to discuss issues, for example of turnout, with the ambassadors in Kabul because we knew the turnout was very low in the southern provinces although a very large number of votes were in fact being reported from those areas.
"Later, when the evidence of the fraud was inescapable he did talk about it but he's consistently minimised it," he said.
The UN rejected the claims that Mr Eide had played down allegations of electoral fraud and sided with Mr Karzai.
Mr Eide has not offered details of the dispute, but has spoken of the need to avoid any impression of foreign interference in the election process.
EU election observers have said that about 1.5m votes - about a quarter of all ballots - cast in August's presidential vote could be fraudulent.
AFGHAN POLL FRAUD
15 Sep:Election Complaints Commission +chief says 10% of votes need to be recounted
8 Sep: Poll complaints body orders some recounts nationwide
8 Sep: IEC says votes from 600 polling stations "quarantined"
3 Sep: Claims 30,000 fraudulent votes cast for Karzai in Kandahar
30 Aug: 2,000 fraud allegations are probed; 600 deemed serious
20 Aug: Election day and claims 80,000 ballots were filled out fraudulently for Karzai in Ghazni
18 Aug: Ballot cards sold openly and voter bribes offered
They say that 1.1 million votes cast for President Karzai are suspicious.
Mr Galbraith's questioning of the election commission had angered Mr Karzai and several cabinet members, some of whom had said they no longer wanted to work with him.
But Mr Galbraith said the UN had "the mandate to support free, fair and transparent elections".
"That unfortunately imposed on us an obligation to raise the question of fraud in elections which were funded by the international community and supported by the United Nations," he said.
In a statement on Wednesday, Mr Ban's office in New York thanked Mr Galbraith "for his hard work and professional dedication".
"The secretary general has made this decision in the best interest of the mission. He reaffirms his full support for his special representative, Kai Eide," it added.
It is understood that Mr Galbraith would have been kept in his post until after a final ruling on the disputed presidential election - a process that is in its final stages, the BBC's Lyse Doucet says.
But leaks emerged in Kabul before Mr Galbraith himself had been informed of the secretary general's decision.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.