Gordon Brown has been told the poll results were a "final wake up call"
Gordon Brown's leadership needs to be "much tougher", billionaire Labour donor Lord Paul has said.
The peer told the Sunday Telegraph that Mr Brown should "exert his authority", and carry out a Cabinet reshuffle.
Speculation over the prime minister's future has followed poor local election results and the Crewe by-election loss.
Former deputy leadership candidate Jon Cruddas told the Independent on Sunday that the results had given Mr Brown his "final wake-up call".
Lord Paul said: "I think he's too gentle. Gordon has to exert his authority further. He must change some of the people [in the Cabinet]."
He said it was possible for Mr Brown to turn round Labour's fortunes but that it would "take time".
"On the other hand time is a luxury that you just can't wait for."
The peer admitted to feeling depressed at Labour's recent difficulties.
"You certainly get depressed. It would be wrong on my part if I said I don't get depressed," he said.
Mr Cruddas said the election results proved Labour was "tone deaf" to the concerns of voters.
"To heal the fractures we have to change," the left-winger said.
"Until and unless we really work through what is going on here we are going to have a big collision with the electorate at the next election."
'The right man'
Cabinet Office Minister Ed Miliband told GMTV he was certain Mr Brown would be leading the party at the next election.
"He's the right man for the times, he's got the values that people will believe in," Mr Miliband said.
"And when it comes to the choice at the general election, when it comes to people seeing what David Cameron stands for - on health, where he doesn't support our extending of opening times for GPs.
"In education, where he doesn't support the expansion of educational opportunity, and on public spending and tax, where he wants large tax cuts for the few - people will see that Gordon Brown and the Labour party are the right people to take this country forward."
Mr Miliband spoke out in support of the prime minister after several of Sunday's newspapers suggested members of the cabinet were concerned about his leadership.
BBC News political correspondent Iain Watson said senior Cabinet ministers were understood to have rejected the idea of calling for unity within the party, because of fears it would have the opposite effect and create further divisions.
He said it was clear that there would be no head-on leadership challenge to the prime minister.
Frank Field, the former minister who led the 10p tax revolt, called for Charles Clarke, Alan Milburn, Stephen Byers and David Blunkett to be recalled to the Cabinet.
He told the Sunday People: "They aren't 'yes-men', they would add weight to his team and they have the strength and experience to ensure policies are properly thought out."
'Expose the Tories'
Schools Secretary Ed Balls said Labour should turn its sights on the Conservatives.
"This is not the time to give up or turn inwards, but to reach out to the public and expose the truth about the Tories," he said.
Senior Labour peer Lord Desai has said the party's only hope of winning the next general election was a "changed, improved" Gordon Brown.
Former deputy prime minister John Prescott defended Mr Brown, saying he was the "one man in our whole system" who could solve the UK's problems.
Ex-foreign secretary Margaret Beckett said Mr Brown must learn from the electorate's "clear desire to see a change of course".
Former environment minister Michael Meacher said new Labour was "dead".