Brian Gibbons said councils faced a challenge to deliver better services
Local authorities in Wales must significantly improve services or face a possible cut in their numbers, the local government minister said.
Brian Gibbons told the BBC Politics Show restructuring local councils would be an option if targets were not met within the next four years.
Dr Gibbons said he would prefer the 22 local authorities to work together and with external organisations.
This he said would improve services within existing structures.
"It could mean restructuring but I don't think that we would say that is automatically the end point," Dr Gibbons said.
"The big challenge for local government is - with the existing structures, working jointly with health, with other external organisations, working jointly across organisational boundaries.
"Can they deliver improved services for their citizens and service users?"
However Professor Steve Martin from Cardiff Business School said the reduction would not be effective as different-sized councils need to offer different services.
"Simply to reorganise and have a number of larger councils is to miss the problems that we need to address in local councils. So its not a radical enough solution, if you want," Prof Martin said.
Councils could work together to deliver waste management service
"The optimum size for delivering education is a very different sized council to the one that maybe is the best size for delivering waste management.
"So really what we need is a service-by-service solution.
"In some cases we will want several councils to cooperate together, to work in partnership perhaps to deliver a waste management service," he added.
"In others you'd want even smaller councils than we've got now, maybe operating at a neighbourhood or a very local level."
The Welsh Assembly Government has already outlined plans to reduce local health boards from 22 to eight.
The plans, under consultation, would also spell the end for the internal market in the NHS with direct assembly government funding.
The last major local council reorganisation was in 1996 when the Welsh county councils - created in 1974 - were abolished.