In a speech to Scottish business organisations, Mr King said that government support for the banking sector had been "breathtaking" at close to £1 trillion.
"Never has so much money been owed by so few to so many. And, one might add, so far with little real reform," he said.
And he added that measures taken by the government and world leaders had so far failed to tackle the key "moral hazard" that if banks and financial institutions knew that they were too big to fail, and that the taxpayer would help them out if they got into trouble.
G20 plans for tighter regulations may not be enough, because if banks knew they would be bailed out if they hit difficulties, they would continue to take risks, he hinted.
"The belief that appropriate regulation can ensure that speculative activities do not result in failures is a delusion."
Mervyn King thinks that with the current approach, we will only have tackled the symptoms of the problem. The ultimate cause will come back to bite us - perhaps rather sooner than we might have thought.
Mr King said banks may have to separate their bread-and-butter businesses - such as holding savings offering loans to households and businesses - which could still be protected. However more speculative practices, so-called casino banking, would not get government protection.
"Encouraging banks to take risks that result in large dividend and remuneration payouts when things go well, and losses for taxpayers when they don't, distorts the allocation of resources and management of risk," he said, adding it was "hard to see why" taxpayer support could not be limited to retail banking.
"Anyone who proposed giving government guarantees to retail depositors and other creditors, and then suggested that such funding could be used to finance highly risky and speculative activities, would be thought rather unworldly. But that is where we now are."
'Powerful and persuasive'
BBC economics editor Stephanie Flanders said that the governor would not have made his comments lightly, wading into the debate by arguing for "separating out the essential, humdrum utility side of banking from the speculative, more casino side that has got us all into so much trouble."
Many, including the chancellor, oppose this plan, seeing it is unworkable, she added.
"But quite simply, Mervyn King thinks that with the current approach, we will only have tackled the symptoms of the problem. The ultimate cause will come back to bite us - perhaps rather sooner than we might have thought."
We should be under no illusion that the path to sustained recovery will be smooth an painless
Banks have been criticised for making risky investments - which in previous years had brought hefty profits and large bonuses for their staff.
However, when these investments went wrong, it contributed to the global economic crisis which saw several banks, including Royal Bank of Scotland and Lloyds, being part-nationalised.
Shadow chancellor George Osborne said Mr King's speech was "powerful and persuasive".
"His analysis of how the government's system for regulating banks failed and how there has been 'little real reform' since is one I share."
Separately, Mr King said the UK economy would grow in the second half of the year.
However, he did not commit himself to saying the economy had come out of recession between July and September. Initial data on the economy's performance during that period is due on Friday.
And he added that while it should become easier for households and businesses to borrow money, "we should be under no illusion that the path to sustained recovery will be smooth and painless", citing high unemployment and lower industrial output.
"We shall all be paying for the impact of this crisis on the public finances for a generation."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.