John Prescott "clearly looks a fool" over his admission of an affair with his diary secretary and has a "woeful record" in office, David Cameron says.
Mr Prescott is meeting Tony Blair in Downing Street on Monday
The Conservative leader says an inquiry should be held if it is established that the deputy prime minister has broken the ministerial code.
Labour MP Stephen Pound has urged Mr Prescott to "consider his position".
Mr Prescott is meeting Tony Blair on Tuesday afternoon, but No 10 says the talks are only "routine".
"You would be mistaken to read anything into this other than this is routine," the prime minister's official spokesman told reporters.
A public standards watchdog has said it is up to Mr Blair to decide Mr Prescott's fate.
Mr Cameron told BBC Radio 4's Today programme: "His private affair is a private matter.
"Clearly he looks a fool but our job in opposition is to call him to account on his ministerial record.
"I think if he has broken the ministerial code and if he has abused his office in that way, it should be looked at.
"People have already asked questions about the use of ministerial cars and all of that and those questions have to be answered."
But Mr Cameron said it should be for an independent judge of the ministerial code to decide Mr Prescott's fate - and not the prime minister.
He accused the deputy prime minister of being someone who has tried to force regionalisation on communities who did not want it and of bulldozing homes in the north of England, while concreting over the south.
"He has a pretty woeful record," he said.
Mr Pound, usually a loyal Labour MP, said Mr Prescott's affair was "causing huge problems" during local election campaigning but he stopped short of calling for his resignation.
He told BBC News: "People on the doorstep, oddly enough, have been raising the Prescott issue more than the Charles Clarke issue.
"He hasn't made any decision at the moment. He hasn't made a decision to stay or go. The only decision he made was to go to Hull and talk to his wife, which must have been a painful process."
He said Mr Prescott had forced through unpopular regionalisation of public services and building developments, and had failed to deliver effective transport policies.
The prime minister will not discuss the issue, saying it is a "private matter".
But ex-Tory minister Edwina Currie says she thinks the deputy prime minister had no grounds on which to take the Mail on Sunday to the Press Complaints Commission - as he has threatened - over its coverage of the affair.
Ms Currie, who had an affair with John Major before he became prime minister, told the BBC: "If John Prescott really thought he had a case on privacy, he'd go to court."
Meanwhile, Liberal Democrat leader Sir Menzies Campbell told Today the affair had had no impact on the way in which Mr Prescott had fulfilled his public duties.
Derek Simpson, general secretary of manufacturing union Amicus, told the same programme the deputy prime minister should remain in his job.
Sir Alistair Graham, chairman of the Committee on Standards in Public Life, said the prime minister needed to consider if Mr Prescott's private life had affected his public duties and if he should "pay a political price".
Meanwhile, newspapers are continuing to report further allegations about the deputy prime minister.
Two more women have made claims reported in several papers. Mr Prescott has denied any further indiscretions.
Mr Prescott said he "acted stupidly" over his relationship with a secretary
The deputy prime minister has said he "acted stupidly" over his relationship with Tracy Temple, but hit back at claims made by his former diary secretary in the Mail on Sunday.
Showing no signs he intended to resign from the government, Mr Prescott issued a statement in which he said much of her recollections were "simply untrue and are clearly motivated by a desire to maximise financial gain".
Ms Temple, who was paid in excess of £100,000 for her story, said she felt she was used as a "scapegoat" and wanted to set the record straight as she had been "misrepresented".
Ms Temple told the Mail on Sunday she had regular sexual encounters with Mr Prescott in his office with the door open while other civil servants worked outside.
They also had sexual encounters in Mr Prescott's tax-payer funded flat in Admiralty Arch and in the Admiralty Boardroom, she said.