Pressure is continuing to mount over John Prescott's role as deputy prime minister, as more Labour politicians question his position in public.
Mr Prescott was criticised for "becoming the story"
Labour MP Stephen Pound told the BBC that the electorate had a negative image of Mr Prescott and that "the sell-by date is rapidly approaching".
It follows a weekend of stories and photos of Mr Prescott, who is standing in for Tony Blair while he is away.
But Cabinet colleague Hilary Benn said Mr Prescott was "getting on with it".
He told BBC News that Mr Prescott was the right man to be deputy prime minister and that his role in chairing committees, such as the one tasked with delivering earthquake aid to Indonesia, had been ignored by the media.
The latest controversy began with the publication of photos showing Mr Prescott playing croquet at his country retreat Dorneywood, Buckinghamshire, shortly after taking over the reins from Mr Blair.
His parliamentary aide - the Labour MP Paul Clark - insisted the game had been merely part of a departmental away day to discuss the restructing of his office.
Mr Prescott was stripped of his department in Mr Blair's May reshuffle but kept his salary and perks, sparking anger from opposition parties and some Labour figures.
Prescott was pictured playing croquet (pic by isfphotos.com)
Labour MP for Calder Valley Christine McCafferty told BBC Radio 4's World at One it was unfortunate Mr Prescott had "become the story".
"We need newspapers to be talking about our policies not about what individual ministers are doing in their spare time."
She added: "I think it was Alistair Campbell who used to say that when you become the story it is time to consider your position."
The Labour MP for Ealing, Stephen Pound, told the BBC the public had a negative perception of Mr Prescott and the party was also losing faith.
"There's still a sort of residual loyalty, there's quite a bit of affection, but everybody now recognises that the sell-by date is rapidly approaching," he said.
Derek Wyatt, Labour MP for Sittingbourne and Sheppey, told the BBC: "I don't think it's tenable that he should maintain Dorneywood or the Admiralty flat or the chairmanship of the committees."
1999: As transport minister - already dubbed "two jags" - uses official car to go a few hundred yards to conference
2001: Punches an egg-throwing protester during the General Election campaign
2003: Appears to give journalists a V-sign outside Downing Street
2005: Local reporter says Mr Prescott told him to "bugger off" during an interview
April 2006: Admits to having had an affair with a secretary after newspaper revelations
May 2006: Stripped of his department in reshuffle, but keeps title, salary and perks
May 2006: Photos show him playing croquet at his country retreat while deputising as PM
And Hastings and Rye MP Michael Foster said Constitutional Affairs Minister Harriet Harman would be a "good choice" as deputy leader, adding he wasn't calling for an immediate leadership challenge.
"But what we do need to do is be talking about who we have in the leadership who could do that job," he said.
BBC political correspondent Mike Sergeant said the controversy was difficult for Tony Blair, who desperately wanted to avoid a deputy leader election, as it would inevitably be tied to his own future and would allow old Blair-Brown battles to resurface.
The European Trade Commissioner Peter Mandelson said that Mr Prescott would do what was in the party's interests.
"All I would say about John is that he is a party man to his fingertips and, whatever he does, he'll do what's in the party's interest I'm sure - not his own," he said.
As well as Ms Harman, press speculation about a possible successor to Mr Prescott has included the names of Peter Hain, Jack Straw, Alan Johnson and Tessa Jowell.
Ms Harman has said she believes it is a "necessity" for the next deputy prime minister to be a woman.
Bookmakers are predicting that Mr Prescott is likely to survive his current troubles.
William Hill is offering 1-2 on him staying in post into July.
"Mr Prescott is turning out to be a great survivor and has already defied the odds," said spokesman Graham Sharpe.