Some patients get "lost in the gaps" because Wales has too many local health boards (LHBs), Health Minister Edwina Hart has warned.
Edwina Hart said the problems for some patients were unacceptable
With a review due to report by the end of the year, Ms Hart has indicated that the number should be cut from 22.
In an interview with a health magazine, she said it was not right that patients were being "batted back and forth".
The Welsh Assembly Government said it was committed to simplifying the system and reducing bureaucracy.
Ms Hart, who became health minister last May, signalled her thinking in an interview with the magazine of patients' group Hafal.
Wales's LHBs cover the same areas as its 22 local councils, and receive a large slice of the NHS budget to commission - in other words, to buy - services from health providers such as hospital trusts, GP practices, and dentists.
But Ms Hart said in the interview: "There are too many commissioning bodies. We need to do something about it.
"People are getting lost in the gaps between providers of services, getting batted back and forth between various places and that's not acceptable."
Hafal chief executive Bill Walden-Jones told the BBC's Politics Show in Wales: "I think it's likely that there will be an announcement before long: that we can't proceed with having 22 LHBs for a population of only 2.9m. That will be an enormous change."
Critics claim LHBs are too small and there are too many of them to negotiate in the best way for patients.
Richard Lewis, secretary of the British Medical Association (BMA) in Wales said: "Very often you've got smaller LHBs with different priorities, for instance, so it's difficult to concentrate on key issues and get a co-ordinated approach to the way services are commissioned."
On Wednesday the Conservatives will call in the Welsh assembly for all 22 LHBs to be scrapped and replaced with an all-Wales commissioning body.
Tory health spokesman Jonathan Morgan said: "It will get worse of course now that the health minister is going to be merging some of the health trusts and creating these super hybrid trusts.
"The problem with that is that the rationale for keeping these mini-LHBs will simply disappear because if they can't compete with NHS trusts now, they certainly won't compete with the new bigger trusts."
Plaid Cymru is in coalition with Labour in the assembly government, and its health spokesperson Helen Mary Jones said it was clear the situation could not continue.
"There are too many organisations, too much money being spent on administration and of course those are resources that we can't use in front line care."
But Kevin Sullivan, of the Welsh NHS Confederation, which represents LHBs and trusts, said complete abolition would be a "retrograde step".
"Everyone knows they (LHBs) aren't perfect, everyone knows there are areas where they need to improve, as they themselves will acknowledge, so the challenge is how to do that without throwing the baby out with the bathwater."
Ms Hart was unavailable for comment, but an assembly government spokesperson said it was committed to ending the internal market in health care, simplifying the system and reducing unnecessary bureaucracy.
Health economist Siobhan McLelland said Ms Hart's suggestion was no surprise, as the coalition document agreed between Labour and Plaid said they would get rid of the internal NHS market by 2011, and the LHBs were the purchasing side of that.
But Ms McLelland said she thought Ms Hart had never liked the LHB structure begun by Jane Hutt, a previous health minister.
"And she's going to do what politicians always do when they come into the health service," she said.
"They reorganise structure as a proxy for making real changes and developments in the service."