Mr Clarke says Labour is facing 'utter destruction' at the polls
Two cabinet colleagues have pledged their support for Gordon Brown after former home secretary Charles Clarke urged a Labour leadership contest.
Schools Secretary Ed Balls dismissed Mr Clarke's call for the prime minister to improve his performance or quit as "Charles being Charles".
Labour deputy leader Harriet Harman said there should not be a challenge.
No other cabinet colleagues have yet publicly defended the prime minister against Mr Clarke's criticisms.
'People of quality'
The former home secretary spoke as Mr Brown prepared to continue his autumn fight back with a speech to business leaders in Scotland.
Mr Clarke told the BBC that the government's performance must improve "significantly" or Mr Brown should "stand down as prime minister with honour and have a proper leadership election and address the proper issues".
Charles' problem is this is not the first time that he has gone for the old Dad's Army 'We are all doomed' without coming up with any real alternative
Nigel Griffiths MP
He also said "many, many, many" people shared his concerns.
Mr Clarke added that, unless he does better, the prime minister should face a leadership contest within months, and there were "five or six people of quality" who could stand.
Mr Balls, one of the prime minister's closest allies, told GMTV: "It's not the first time Charles has made those kind of comments. I think it's Charles being Charles.
"I don't think that's where the debate will be when we get to the next general election."
Ms Harman, often named as a potential challenger to Mr Brown, said: "I don't think that there is going to be a leadership challenge - nor should there be."
And health minister Ben Bradshaw dismissed the former home secretary's attack as "Westminster tittle tattle".
Asked why more cabinet ministers had not publicly backed Mr Brown, he said that might have given Mr Clarke's comments "more credibility than perhaps they felt they were worth".
Former minister Nigel Griffiths also expressed anger, saying: "It is not as if Charles has any alternative policies. In 2007 he and (former health secretary) Alan Milburn set up a think-tank called 2020 Vision.
"It didn't think but it certainly tanked. It folded having produced not one alternative policy."
He added: "Charles' problem is this is not the first time that he has gone for the old Dad's Army 'We are all doomed' without coming up with any real alternative.
"He sort of lobs a grenade into the party and then backs away and he did have his chance ... in 2007 to come up with alternative policies, and he hasn't come up with them."
Mr Clarke has a track record of criticising Mr Brown and was one of only a handful of Labour MPs not to publicly back his bid for the premiership last year.
He has been touted as a potential leader himself, but he earlier ruled out a challenge to Mr Brown.
Asked if he would be prepared to stand against the PM as a "stalking horse" - to flush out more heavyweight challengers - he said: "I wouldn't do that myself - and I wouldn't counsel anybody else to do it either."
Mr Clarke launched his latest salvo against the PM in an article for The New Statesman, writing that Labour faced "utter destruction at the next general election" if it continued on its current course.
He stressed that there was no "Blairite plot" against Mr Brown but there was a "deep and widely shared concern" in the party that Labour was heading for "disaster" coupled with a determination "that we will not permit that to happen".
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.