Tony Blair has attempted to move on from David Blunkett's resignation and a big rebellion over anti-terror plans.
He told colleagues during a Cabinet meeting on Thursday "times are tough", a Downing Street spokesman said.
Tory leader Michael Howard told BBC Radio 4 Today's programme that Mr Blair's authority was "vanishing".
On Wednesday, Mr Blunkett quit as work and pensions secretary after breaking rules over jobs. The Terror Bill passed an MPs' vote by a majority of just one.
This was Labour's smallest margin of victory in the Commons since it came to power in 1997.
Mr Blair reportedly told Cabinet colleagues: "Times are tough but they are tough because the government is trying to do the right thing, whether on public service reform, education, health, anti-social behaviour and welfare, or in counter-terrorism."
Mr Howard said Mr Blair's "loss of authority" was similar to that of John Major's Tory government of the mid- to late-1990s.
He had already called Mr Blair a "lame duck" prime minister and claimed his loss of support was becoming a "haemorrhage".
But Mr Clarke told Today that Labour MPs were "far happier than five or six years ago".
Although ministers felt "desolate" about Mr Blunkett's resignation it had been the right thing to do, he added.
The Lord Chancellor, Lord Falconer, said Mr Blair's authority had not dminished, because it had "always depended on the extent to which he connects with the public and their wishes for reform".
BBC political editor Nick Robinson said that, at the moment, it was not "good for your political health to be associated with the prime minister".
Mr Blunkett quit amid the controversy over his two-week directorship of DNA Bioscience before May's election, while he was out of the Cabinet.
He broke ministerial rules by taking that job without consulting an independent committee.
Mr Blunkett, who was previously forced to step down as home secretary in December 2004, said he had gone "to protect the government".
His second resignation follows Cabinet disagreements over education reforms and plans for a ban on smoking in public places.
Meanwhile, Labour's Commons majority was cut from 66 just one over plans to outlaw indirect incitement of terror.
Charles Clarke has asked to delay a vote on plans to extend the time suspects can be held without charge from 14 days to 90 days.
Ministers fear the government could face a defeat unless a consensus is reached.
The plans are opposed by most Lib Dems and Tories and several Labour backbenchers.