John Prescott should "consider his position" as deputy prime minister, Labour MP Stephen Pound has said.
Mr Prescott said he "acted stupidly" over his relationship with a secretary
Mr Pound said Mr Prescott's affair was "causing huge problems" during local election campaigning but he stopped short of calling for his resignation.
A public standards watchdog has said it is up to Tony Blair to decide Mr Prescott's fate over his admission of an affair with his diary secretary.
The prime minister will not discuss the issue, saying it is a "private matter".
Mr Pound told BBC News: "People on the doorstep, oddly enough, have been raising the Prescott issue more than the Charles Clarke issue.
"He hasn't made any decision at the moment. He hasn't made a decision to stay or go. The only decision he made was to go to Hull and talk to his wife, which must have been a painful process.
"There is a real problem there, and on the doorstep I have to say that is one that is causing huge problems."
Conservative leader David Cameron said it was a "personal" matter but "he looks like a damn fool".
He added: "His record is poor and that's why we should call him to account."
He said Mr Prescott had forced through unpopular regionalisation of public services and building developments, and had failed to deliver effective transport policies.
Meanwhile, a former Tory minister has said she thought the deputy prime minister had no grounds on which to take the Mail on Sunday to the Press Complaints Commission - as he has threatened - over its coverage of the affair.
Edwina Currie, who had an affair with John Major before he became prime minister, told the BBC: "If John Prescott really thought he had a case on privacy, he'd go to court."
Liberal Democrat leader Sir Menzies Campbell told BBC Radio 4's Today programme the affair had had no impact on the way in which Mr Prescott had fulfilled his public duties.
Derek Simpson, general secretary of manufacturing union Amicus, told the same programme the deputy prime minister should remain in his job.
Mr Prescott's biographer, Colin Brown, the Independent's deputy political editor, said Mr Prescott's close friends were horrified by the revelations but that his family were determined he would not be "forced out of office".
Sir Alistair Graham, chairman of the Committee on Standards in Public Life, said the prime minister needed to consider if Mr Prescott's private life had affected his public duties and if he should "pay a political price".
Newspapers are continuing to report further allegations about the deputy prime minister.
Two more women have made claims reported in several papers. Mr Prescott has denied any further indiscretions.
Newspaper claims about the affair have embarrassed the government
The deputy prime minister has said he "acted stupidly" over his relationship with Tracy Temple, but hit back at claims made by his former diary secretary in the Mail on Sunday.
Showing no signs he intended to resign from the government, Mr Prescott issued a statement in which he said much of her recollections were "simply untrue and are clearly motivated by a desire to maximise financial gain".
Ms Temple, who was paid in excess of £100,000 for her story, said she felt she was used as a "scapegoat" and wanted to set the record straight as she had been "misrepresented".
Ms Temple told the Mail on Sunday she had regular sexual encounters with Mr Prescott in his office with the door open while other civil servants worked outside.
They also had sexual encounters in Mr Prescott's tax-payer funded flat in Admiralty Arch and in the Admiralty Boardroom, she said.