Tony Blair may not be able to recover his authority and leadership after recent damaging events, former Home Secretary Charles Clarke has said.
In a series of media interviews, Mr Clarke, previously a strong Blair ally, voiced his anger at being sacked.
But Mr Blair told Labour MPs not to lose confidence in the face of the current onslaught.
And Education Minister Jim Knight said ex-ministers "bitching" about Mr Blair were doing Labour "no favours".
Shadow home secretary David Davis said Mr Clarke's comments were a Blairite version of Sir Geoffrey Howe's attack on Margaret Thatcher in 1990 - something Mr Blair denies.
And Liberal Democrat leader Sir Menzies Campbell said: "It is clear that the prime minister's authority is simply draining away."
The prime minister's official spokesman brushed off the criticisms, saying Mr Clarke had not hidden his disappointment on the day he left the Cabinet.
And Mr Blair himself tried to steady nerves in his party at a briefing for Labour MPs, academics and journalists in Downing Street.
He said Labour must not be "frightened by what is a pretty big onslaught at the moment" and must keep modernising.
Acknowledging criticism of his leadership, Mr Blair said: "You will always have a group of people who stand up and say you have betrayed everything we ever stood for.
"In fact, we have done exactly what it said on the tin."
Mr Clarke was sacked as home secretary in May after it emerged that 1,023 foreign prisoners were released without being considered for deportation.
He says he has decided to speak out to restore his political reputation - and to put his "point of view on the record".
Mr Clarke told BBC Radio 4's On The Ropes he had previously thought Mr Blair should say he would quit in late 2008.
That was still "pretty much" his view but he voiced his doubts.
"The best option would be for Tony to recover that leadership and authority and direction and to carry that through over a period of time, in my view," said Mr Clarke.
"Whether he is able to do that - because he has been damaged by recent events - whether he wants to do that, is not a matter for me, really.
"I simply observe there are a lot of doubts about it and I share some of those, that's true."
Mr Clarke told BBC Two's Newsnight he felt the decision to get rid of him was for political expediency rather than long-term reform.
It had destroyed his chances of carrying through his planned shake-up of the Home Office.
Mr Clarke said Mr Blair offered him other jobs but he did not want to take them as he had promised to resolve the Home Office's problems.
"I was angry and frustrated because I felt that this massive task, a great privilege to be asked to be home secretary at the general election, needed to be carried through over a three to four year period and I believed I could do that," he said.
Media 'band wagon'
Mr Clarke said he did not agree with his successor John Reid's assessment, that the Home Office was "not fit for purpose".
"I thought that was absolutely not the case... I think John was wrong to say that," he said.
Mr Clarke also appeared scathing about Mr Reid's decision to consider introducing a Megan's Law system for dealing with sex offenders in the UK, following the News of the World campaign.
He said if that decision had been influenced by the newspaper's campaign "then I would criticise it".
"The home secretary of the day should not simply be running on the band wagon of some particular media campaign," he said.
In response, Mr Reid's official spokeswoman said: "The home secretary has the highest respect for Charles Clarke.
"But different home secretaries have different priorities."
Former Health Secretary Frank Dobson said Mr Blair had become an "electoral liability" who was damaging Labour's prospects of winning the next election, he argued.
But Mr Knight hit back: "Most of my colleagues find some of the people like Frank irritating because we want to talk about policy, about implementing the manifesto and getting on with the job we were elected to do, rather than constantly sniping from the background," said the junior minister.
"Ex-ministers coming on and bitching about the prime minister doesn't do the Labour Party any favours."
And Stephen Pound, aide to Labour chairman Hazel Blears, said Mr Blair was the right person "more than anyone" to renew the party.
"Nobody has the strength or authority that they have on day one. That's just one of the rules of politics," he said.
But he urged Mr Blair to call a halt to any more major legislation, saying there was no appetite for it on the backbenches. "We want to see that legislation work," he told BBC Radio 4's World at One programme.