Gordon Brown's government has been hit by several departures
Lord Malloch Brown, the minister for Africa, Asia and the UN, is resigning from the government at the end of July.
The minister said in a statement that he had always maintained he would "never do the job forever".
He stated that "personal and family reasons" were behind his decision and stressed that his move was not motivated by the political situation.
Lord Malloch Brown said he "greatly admired" the prime minister and continued to support the government.
No 10 would not comment on possible replacements although an immediate announcement is thought unlikely.
Announcing his decision to step down, Lord Malloch Brown said he had entered government "as a professional not a politician".
"My decision to step down at the end of July is not in any way a commentary on the political situation.
"I joined the government at his invitation to help promote his international priorities. It has been a great privilege to do that, and to work with him and with David Miliband."
Lord Malloch Brown was a member of Gordon Brown's Government Of All the Talents - dubbed "goats" by commentators - - non-politicians drafted into government for their expertise in different policy areas when Mr Brown became prime minister in 2007.
Of those, former CBI boss Lord Digby Jones has already quit the government, leaving his post as trade minister last October.
And former Ofcom boss Lord Carter recently signalled his intention to leave his role as minister for communications and broadcasting.
But Lord West, the former First Sea Lord, remains as security minister while renowned surgeon Lord Darzi continues as a minister in the Department of Health.
The prime minister has tried to maintain his goat policy, hiring former M&S chairman Paul Myners as city minister and former bank boss Mervyn Davies to replace Lord Jones.
Lord Malloch Brown worked under Kofi Annan as deputy secretary-general of the UN from April to December 2006 and was previously his right-hand man.
He was also in charge of the UN Development Programme from July 1999 to August 2005.
Downing Street issued a statement, praising the minister's "outstanding work".
"The government has greatly benefited from his exceptional knowledge of Africa, the respect in which he is held by an extensive network of close contacts and his passion for his work," the statement said.
"His support in preparing the G20 London summit ensured that the plans set out enjoyed the widest possible international support."
The government has been hit by a string of high-profile departures in the past month including Communities Secretary Hazel Blears, Home Secretary Jacqui Smith and Work and Pensions Secretary James Purnell.