Mr Osborne said smaller banks were less likely to take excessive risks
A Conservative government would look at whether Britain needs smaller banks, shadow chancellor George Osborne has said in a speech.
Mr Osborne said the eventual sell-off of part-nationalised banks should not necessarily be to the highest bidder but those benefiting the economy most.
Creating larger institutions could encourage more risk because they were "too big to fail", he added.
It would be a "bitter irony" if the sector became "even riskier", he said.
The government has bought large shareholdings in Royal Bank of Scotland and HBOS and nationalised Northern Rock and Bradford and Bingley in an effort to prevent the collapse of the banking system.
It has always stressed it does not want to run the banks as nationalised industries and that it will return them to the private sector at some stage.
Mr Osborne, in a speech to the Royal Society for the Encouragement of Arts, Manufactures and Commerce in London, said Prime Minister Gordon Brown had led people over the last 12 years to believe they were living in "a financial nirvana of stability and growth, only to find it was just a fragile illusion of prosperity built on debt".
It would not be acceptable in future to allow banks to "behave in a way that puts the rest of the economy at risk when it fails".
Mr Osborne added: "We need to think deeply about whether we can sustain banks that are not only too big to fail, but potentially too big to bail."
He also said: "Whether any form of separation between retail and investment banking, even within the same financial group, is feasible or desirable, it is clear that size itself is a risk factor.
"Not only do large financial institutions do more damage when they get into trouble, but their very size and 'too-big-to-fail' status may encourage them to behave irresponsibly and take risks that smaller banks dare not take."
Mr Osborne went on: "In the long term, whilst tighter regulation can strengthen financial stability, we need to balance this with need to maintain and enhance competition between banks and others to protect consumers.
"By dint of its substantial shareholdings the government has a powerful influence over the future structure of the UK banking industry, whether it likes it or not.
"When the time comes to sell off those shareholdings we need to think very carefully before simply selling them to the highest bidder without thinking through the consequences for the wider economy.
"We should look at whether Britain in fact needs smaller banks.
"For it would be a bitter irony if we came out of this crisis with a banking system that was even more concentrated and even riskier than the one we had before it."
BBC business editor Robert Peston said Mr Osborne had meant that "in rescuing RBS, the government has mortgaged all our economic futures to the rehabilitation of this giant bank".
If the next government were to dismantle Lloyds TSB, which bought HBOS recently, it "would be a reputational disaster for Lloyds' management", he added.
The shadow chancellor had also implied "that he would privatise Northern Rock as an independent bank, rather than flogging it to another bank", our correspondent said.
For the government, Treasury minister Stephen Timms said: "George Osborne's efforts to rewrite history are heroic but the fact remains the Tories have been isolated and on the wrong side of the argument from the moment the credit crunch hit Britain.
"Alongside countries across the world, we are part way through a huge boost for our economy, families and businesses - a boost opposed by the Tories, who still think we should do nothing to help people through these tough times."