Mr Cameron said the last two weeks had been terrible for Labour
The government's Commons defeat over the Gurkhas shows Gordon Brown is "rapidly running out of political authority", David Cameron has said.
The Conservative leader said Mr Brown had "misjudged the public mood" over how many Gurkhas may settle in the UK.
Labour lost a vote on the issue after 27 of its MPs rebelled.
Mr Cameron said the defeat capped a damaging few weeks for Mr Brown but the PM said he would not be "diverted" from the vital challenges facing the UK.
Asked if his authority had suffered, the prime minister insisted he was "getting on" with the business of government, focusing on swine flu and the recession among other issues.
"We are dealing with the big issues. We are not going to be diverted," Mr Brown told a press conference after talks with the Iraqi Prime Minister Nouri al-Maliki.
Business Secretary Lord Mandelson has admitted it was "turning into a bit of a week" for Labour but denied the prime minister was losing his authority.
He said Mr Brown was right to be taking the initiative over issues such as reforms of MPs' expenses, on which they will be a series of crunch Commons votes on Thursday.
But former Labour minister Tom Harris said the lost vote was a "humiliation" for the government.
In his blog, Mr Harris wrote that he had backed the government as it deserved "credit" for what it had done for the Gurkhas but conceded Labour had "screwed up".
At his monthly news conference, Mr Cameron said a sequence of events showed Mr Brown's authority was being undermined.
Last week's Budget showed the government was "running out of money" while the Damian McBride e-mail scandal indicated an administration "running out of moral authority", he said.
The government had also been forced into a number of policy U-turns over issues such as Titan prisons, he added.
The Conservatives and Liberal Democrats united to inflict a shock defeat on Labour over the Gurkhas as a large number of Labour MPs either voted against the government or abstained.
In response, ministers will reconsider the restrictions on entry for Gurkhas, which critics say are arbitrary and unjust, and bring forward a decision on current applications.
"The prime minister misjudged the public mood and what was right," Mr Cameron said of the vote.
He urged the government to accept Conservative proposals to reform immigration rules to allow all Gurkhas who served in the British army before 1997 to be able to come to the UK.
Mr Brown, who chaired a Cabinet meeting on Thursday, said Labour had done more for the Gurkhas than any previous government and "wanted to do more".
The Liberal Democrats have also calling for all Gurkhas to get equal right of residence.
In addition to the Gurkhas, Mr Cameron said there were other areas where the two main opposition parties could "co-operate" together, such as the environment and civil liberties.
However, he stressed there were still areas of "big disagreement" between the two parties - notably on Europe.
On the economy, Mr Cameron said it would be the political party which best understood how to cope with the changed economic climate which would deserve to win the next election.
The Conservatives are reviewing all their public spending commitments, with Mr Cameron suggesting a "new era of thrift" in government is needed to restore public finances to health.
But he rejected suggestions that this would leave the Tories having to take "nasty" decisions on spending cuts and tax rises that would effectively derail other proposed reforms.
If people viewed the Conservatives as just "a bunch of accountants" the party would have failed, he said.
The Conservatives had to show their policies for reforming public services to give more power to people were consistent with ensuring the government lived within its means, describing this as an "exciting challenge".