Paddy Ashdown has told the UN secretary general that he no longer wishes to become special envoy to Afghanistan.
Karzai sees Ashdown as too strong a figure, observers say
Lord Ashdown said the job needed Afghan government backing that did not exist, even though President Hamid Karzai had once "wished him to take up the post".
His decision came after Afghanistan said it preferred Gen John McColl, Nato's deputy commander in Europe.
Lord Ashdown - a British peer - served as the UN's High Representative and EU envoy to Bosnia from 2002 to 2005.
Lord Ashdown wrote to UN Secretary General Ban Ki-moon on Saturday to withdraw his candidacy.
I was especially pleased to hear from you that President Karzai also wished me to take up this post
Lord Ashdown letter to UN secretary general
Most computers will open this document automatically, but you may need Adobe Reader
"This job can only be done successfully on the basis of a consensus within the international community and the clear support of the government of Afghanistan," he said in a statement on Sunday.
"It is clear to me that, in Afghanistan at least, the support necessary to do the job effectively does not exist."
The dispute over the appointment - to be made by the UN - comes at a time of strained relations between Afghanistan's President Hamid Karzai and the West, says the BBC's diplomatic correspondent, James Robbins.
The Afghan leader has recently criticised the performance of British troops fighting the Taleban in the restive Helmand province.
He apparently sees Lord Ashdown as too strong a figure, who could look like a rival, our correspondent says.
In his letter to Mr Ban, Lord Ashdown said he had been pleased to be told by the UN chief that President Karzai had agreed to his role - which also had the support of the US.
"However, since our meeting there have been many reports, both diplomatic and in the media, including in Afghan government-controlled newspapers, that President Karzai and members of his government are opposed to my appointment, partially based on my nationality and Britain's role in Afghanistan."
After Lord Ashdown's announcement, the Afghan foreign minister explained why Kabul had changed its mind.
Rangin Dadfar Spanta told the BBC this was because they had to take account the views of Afghanistan's media, its political elites and its public opinion.
Mr Spanta said that in these arenas a number of critical views of Lord Ashdown had already been expressed, which, if the British politician came to Kabul would cause people to view him negatively. "That," the foreign minister said, "would not be good for him or for us."
Without specifying, Mr Spanta said that while serving as UN special representative in Bosnia-Hercegovina, Paddy Ashdown had at times been very controversial.