Environment minister David Miliband has said voters will look back fondly on Tony Blair's time as prime minister, whoever succeeds him.
He said people always tended to want former prime ministers back.
"People will be saying 'wouldn't it be great to have that Blair back because we can't stand that Gordon Brown'," he said on BBC One's Question Time.
He then clarified his remarks to say that Mr Brown would be a "very successful" prime minister.
Mr Blair has announced he will leave Downing Street in the next few months.
Mr Miliband, often tipped as a future Labour party leader, was responding to a question about Mr Blair's continuing leadership.
He said that Mr Blair had a right to choose when he goes.
"I really don't think it matters whether he goes on 4 May, 4 June or 4 July. It's entirely up to him."
He praised the prime minister's "efforts" in Northern Ireland over the past decade, and said that decisions in government had not been paralysed by uncertainty about Mr Blair's departure.
Mr Miliband said: "I predict that when I come back on this programme in six months or a year's time, people will be saying 'wouldn't it be great to have that Blair back because we can't stand that Gordon Brown'."
Asked to explain his statement, Mr Miliband said: "Whoever's in, it's one of those things in government, people have a whack at you.
"That's part of the joy of politics. What I will say is that this country is richer, fairer, more confident than it was 10 years ago and we should be grateful for it."
Mr Miliband said he was not suggesting that Gordon Brown would not be a suitable leader.
"I think Gordon Brown will be the prime minister. I think he will be a very successful prime minister, but I think that in politics it is at election time that people make their judgements."
Other politicians appearing on the show said Mr Blair had hung around too long.
Liberal Democrat Simon Hughes said Mr Blair had a right to be in charge because he had won the last election, but he had "lost the confidence of his own party", with ministers "saying their own thing".
"I afraid that he has suffered from the same fate as Margaret Thatcher did - he stayed too long and he didn't learn the lesson that he should have gone much earlier."
Conservative MP Ken Clarke described the government as "drifting hopelessly".
Referring to constant speculation about the police investigation into honours, Mr Clarke said: "This is just a silly, and rather pathetic end when actually politics, the responsibility of government, means that Tony really should have resigned probably 12 months ago."