In September it was announced that the two units would stay open
Every consultant brain surgeon in Wales has signed a letter calling for all Welsh neurosurgery to be based at a single site.
This is despite assembly government plans to develop two centres at hospitals in Cardiff and Swansea.
Writing to the deputy chief medical officer, Professor Mike Harmer, the surgeons say a single site would provide a "safe and effective service".
Prof Harmer said the consultants' views would be considered.
In September, Health Minister Edwina Hart announced a "clinical network" would allow services to stay at the University Hospital of Wales, Cardiff and Morriston Hospital, Swansea.
It meant specialist staff would work at the two sites, keeping them both open.
This followed a review led by Professor James Steers, a consultant neurosurgeon from Edinburgh.
"We welcome the view from the neurosurgeons," said Prof Harmer. "This will be considered as part of the ongoing implementation of the Steers report recommendations."
Dr Alan Axford of Hywel Dda NHS Trust is to lead a group looking at how to implement that decision in the new year.
Patients in north Wales will continue to receive treatment at Walton Hospital, Liverpool.
But Richard Hatfield, a consultant neurosurgeon at the University Hospital of Wales, Cardiff, said sustaining two units was "not viable".
He warned that non-emergency treatments may have to be cut back before the end of the year if the situation remained the same.
"The way things are going at the moment even by after Christmas we'll be having difficulty providing a full service," he told BBC Wales' Dragon's Eye programme.
"The difficulties are really taken in the form of having enough junior staff to provide 24 hour services and something will have to give.
"A sustainable future can only be achieved with a single neurosurgery unit."
Professor Ann Rosser from Cardiff University's School of Biosciences said she feared political interference was jeopardising the work of research teams there.
"From the point of view of developing really high quality international research there's a critical mass that's required and we really need a centre of excellence here in Cardiff," she said.
In 2006, a review by Health Commission Wales recommended complex surgery should only be carried out in Cardiff and warned that keeping two sites open would be unsustainable.
But plans to close the neurosurgery unit at Morriston Hospital were put on hold after a high profile campaign.
Conservative health spokesman Jonathan Morgan said Ms Hart should "start listening to the experts".
"This idea that we can continue in the medium and long term by having two sites is a nonsense and I think the minister has to accept that only one site is viable in the long term," he said.
Liberal Democrat Peter Black said there was a danger services in both cities would "wither on the vine" if the concerns of consultants were not addressed.
"I have argued from the beginning that there needs to be a single neurosurgical service and that Swansea is the logical place to site it," he said.
"It has good communications links, is geographically central and together with the Bristol service will ensure that the whole of south Wales is within easy travelling distance of a service."
Abertawe Bro Morgannwg University NHS Trust said it fully supported the recommendations of Prof Steers' report.
"We will be working with Dr Alan Axford to implement the recommendations of the report, and with Cardiff and Vale NHS Trust to develop the services for all patients, right across south Wales," it said.
"While Mr Axford's work with the implementation group is ongoing it would be inappropriate for the trust to comment any further."
Responding to Mr Morgan's and Mr Black's comments, a spokesperson for Ms Hart said: "The minister has invested a lot of time and effort in these issues to ensure that there is a good service provided to all the people of Wales.
"Unfortunately, she finds it all too predictable that opposition members would seek to make mischief in this way."