Public trust is being undermined by the way alleged misconduct by ministers is policed, says the standards watchdog.
Ms Jowell was cleared of wrongdoing by Tony Blair
Sir Alistair Graham, chairman of the committee on standards in public life, said the controversy over Tessa Jowell has highlighted the problem.
He was "puzzled" about why Tony Blair had failed to change the rules.
Downing Street does not back the watchdog's calls for independent figures, not senior civil servants, to investigate complaints about ministers.
Mr Blair's official spokesman said: "There is no guarantee if we had an independent figure that his decisions would not be criticised in the media."
BBC political editor Nick Robinson said the prime minister thought many journalists would not believe any verdict reached by an independent investigator.
For instance, Lord Hutton's findings on the death of government scientist Dr David Kelly had been criticised.
The BBC has been told that the committee on standards in public life has received "sympathetic signals" from Chancellor Gordon Brown over its proposals.
A spokesman for the chancellor said: "Of course there's a dialogue to be had but any views attributed to Gordon Brown are just speculation."
Sir Alistair is due to meet Mr Brown later this month.
Conservatives and Liberal Democrats support setting up an independent panel.
Sir Alistair said the current policing arrangements were "ad hoc".
"The present system is demonstrably redundant and leads to a loss of public confidence and damage to the standing of the government," he said.
COMMITTEE ON STANDARDS IN PUBLIC LIFE
Set up in 1994 under Lord Nolan to be an "ethical workshop", looking at concerns about the conduct of all public office holders
Recommends changes to improve the standard of behaviour of politicians and public servants but does not investigate specific cases
Current chairman: Sir Alistair Graham, ex-leader of Civil and Public Services Association, ex-head of the Northern Ireland Parades Commission and ex-chair of Police Complaints Authority
The panel proposed by the committee would be agreed between the government and opposition parties after each election.
One member would be asked to step in every time there were claims the ministerial code had been broken.
They would investigate, publish their findings about the facts and the prime minister would then decide what action to take.
Sir Alistair said the Jowell affair and past controversies involving Peter Mandelson and David Blunkett showed the problems of the current system.
Unveiling the committee's annual report, he said: "I am puzzled why the prime minister has not acted on this issue.
"At regular intervals he has been faced with allegations of breaches of the ministerial code in which he and his government have become the centre of a media storm.
"This leads to immense pressure on a minister, whose future will often depend on the vagaries of an ad hoc investigation."
Sir Alistair said the changes could made it fairer for ministers, suggesting that Mr Mandelson had received "rough justice" under the current arrangements.
Cabinet Secretary Sir Gus O'Donnell was called in last week to look into allegations relating to the business dealings of Ms Jowell's husband David Mills.
Mr Blair subsequently cleared Ms Jowell of breaching the code of conduct - because her husband did not tell her about a £344,000 gift he had received.
Sir Gus said the culture secretary accepted her husband should have told her about it.
Ms Jowell would then have told her top civil servant about the gift, so complying with the code, his report said.