Sir James Crosby is no longer advising the government
Gordon Brown is under pressure over his links with former HBOS chief Sir James Crosby, who quit earlier as deputy head of the Financial Services Authority.
The prime minister told MPs Sir James was "right" to quit as he was facing "serious but contested allegations" relating to his time as head of HBOS.
He also said Sir James was no longer an economic adviser to the government.
But the Tories say the PM still has "serious questions to answer" about his past relations with him.
They claim Mr Brown made a "serious error of judgement" in employing Sir James, who made the surprise announcement he was quitting half an hour before prime minister's questions - prompting speculation he was pushed into it.
Shadow chancellor George Osborne told BBC News: "Gordon Brown now has some very serious questions to answer.
"He stood by James Crosby. He appointed him to be a senior regulator. He gave him a Knighthood. Now he appears to have engineered his resignation just half an hour before he came before the House of Commons to face David Cameron at prime minister's questions.
"Gordon Brown has a lot to answer for."
Sir James faces allegations that he ignored warnings from within HBOS - of which he was chief executive until 2006 - that the bank's lending policies were too risky.
A former HBOS employee, Paul Moore, said he had been dismissed by Sir James for alerting the bank to the fact it was expanding too fast, an allegation denied by the bank.
Reacting to his surprise decision to resign as vice-chairman of the FSA, Mr Brown said the move was the "right" one.
Sir James had said there was "no substance" to the allegations but said it was the "right course of action" for him to step aside, added the prime minister.
"In relation to Sir James, these are allegations that he will wish to defend. So it is right that he has stepped down."
Mr Brown said an independent investigation into the whistleblower's allegations by accountants KPMG had found them to be unsubstantiated.
But he suggested it would be appropriate for the Treasury select committee to look at the issue again.
'Error of judgment'
"It is right we investigate serious allegations made about the banking system," he said.
"It is important that the FSA is operating to the best standards possible."
Mr Cameron said Mr Brown had "singled out" Sir James to advise him on key economic issues such as bank regulation, a decision he called a "serious error of judgment".
He told MPs: "Taxpayers have poured billions into this bank and not only was Sir James appointed as one of the top regulators in the country, you have been relying on him for economic advice.
"Sir James Crosby has had the decency to resign. Why can't the prime minister have the decency to admit he got something wrong?"
Sir James wrote two reports for the government, on kick starting the mortgage market and boosting public confidence in the government's controversial ID card scheme.
Liberal Democrat Treasury spokesman Vince Cable said: "It is clear from Paul Moore's evidence that James Crosby was part of the problem, not part of the solution. The FSA needs people with experience, but not the wrong experience.
"It is extraordinary that Gordon Brown ever made such an utterly inappropriate appointment."
He added: "The fact that Sir James has fallen on his sword is beside the point. He should never have been appointed to such an important role in the first place."
In his resignation letter to the FSA, Sir James said the allegations made by Mr Moore had been "independently and extensively investigated on behalf of the [HBOS] board, the results of which they shared with the FSA".
He added: "That investigation concluded that Mr Moore's allegations had no merit."
Sir James went on: "I am full of admiration for my colleagues at the FSA and the work they are doing under extreme pressure.
"As a non-executive director I have an absolute responsibility to ensure that I do not make their task any more difficult.
"Therefore, whilst I am totally confident that there is no substance to any of the allegations, I nonetheless feel that the right course of action for the FSA is for me to resign from the FSA Board, which I do with immediate effect."
BBC political editor Nick Robinson said there was "no doubt that Gordon Brown's political enemies are delighted" at the resignation of a figure involved in banking who was "personally linked" to the prime minister.
During prime minister's questions, Mr Brown also came under fire about the appointment of Glen Moreno as chairman of UK Financial Investments Ltd - the company set up by the Treasury to manage the state's investments at arms's length from ministers.
Until last April, he was a trustee of a Liechtenstein-based fund at the centre of a row over tax evasion.
Mr Brown said Mr Moreno was serving as "acting" chairman.