He said Mr Brown concentrated on policies, not on "being a showman".
"If we get the policy agenda right, and if it's sufficiently bold and decisive then the public will take a different look at us.
"At the moment they're being entirely distracted by noises off, by people who are not keeping their nerve."
He said Labour MPs worried about the prime minister did not appreciate that "what is at the root of the public's concerns about the government are the fears they have as a result of the economic recession".
No-one wanted to stand against Mr Brown because the cabinet was "united" behind him, said Lord Mandelson.
Having a third leader in one Parliament would mean "irresistible pressure to hold a general election" before Labour had the chance to sort out the economy and implement its plans for public services, he said.
He's put so many lords in his cabinet he's drifting back into Victorian times
William Hague Conservatives
But Lord Falconer, a close ally of former PM Tony Blair, told the BBC: "I think we are moving moderately quickly towards the need for a change and that change may be a change in leadership."
"What the country wants is a government focused on the renewal of trust in our politics and on the economic crisis. I think at the moment they don't think we as a party can do it.
"We need unity above all. Can we get unity under the current leadership? I am not sure that we can. And I think we need to debate it urgently and I think probably it will need a change in leader".
The continued debate follows a drubbing for Labour at the local elections on Thursday - when they lost the four county councils they controlled and their vote share dipped to an historic low of 23%.
Former minister Nick Raynsford told the BBC's Marr programme the prospects were "even worse" for the European elections, the first results from which are due on Sunday evening.
He said that if there were no change there could be a "disastrous general election" in which Labour would be reduced to "an ineffective and very small opposition party".
Brown: 'You're bound to have ups and downs'
He added that there were a number of possible leadership candidates: "If you are looking at the situation of total disaster facing us, it is not good enough to say 'more of the same'. We have to be radical. We have to think of ways in which we can re-engage with the electorate."
Asked about Labour's leadership rumblings, shadow foreign secretary Mr Hague said the Conservatives were happy to "take any of them on".
"All we can see is a government consumed by its own affairs they're not doing any actual governing. This is all about them and not about the country."
Email 'tittle tattle'
He criticised the reshuffle - which saw Lord Adonis and Glenys Kinnock, who will be made a peer, moved into the cabinet.
"He's put so many lords in his cabinet he's drifting back into Victorian times in terms of less accountability and less democratic government."
Meanwhile, Lord Mandelson dismissed reports of an email he wrote in January 2008, which described Mr Brown as "self conscious" and "angry", as "tittle tattle".
Lord Mandelson said the email had not been hostile to the PM but had simply advised that he be himself.
Influential left-wing backbencher John Cruddas joined those backing the PM. He told the Sunday Mirror: "Everyone knows our government is in trouble. Serious trouble.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.