The proposals have divided opinion in Aberdeen
The majority of people who took part in a consultation on plans to create a civic square in the heart of Aberdeen opposed the plan, putting it in doubt.
Results of the consultation showed 55% of 11,943 formal submissions received were against the City Square project, with 44% supporting it.
Proposals for the £140m development came with a pledge of £50m in funding from businessman Sir Ian Wood.
Sir Ian said he was now "less than hopeful" the project could proceed.
Arts body Peacock has separate plans for a £13m centre in the same area which have already been approved.
There was an eight-week consultation into the Aberdeen City and Shire Economic Future (Acsef) plan.
Respondents to the consultation said the most important facility for the area was formal gardens, followed by a contemporary arts centre and a cultural centre.
Sir Ian said: "I'm obviously disappointed.
"Although marginally negative, I believe there was a significant upsurge in backing towards the end of the consultation.
"I am now less than hopeful this project can go ahead."
Aberdeen City Council leader John Stewart told BBC Scotland the council would consider options for the whole city centre - including the gardens - on 19 May.
Sir Ian said: "I am prepared to leave my financial offer on the table until these deliberations.
Sir Ian Wood has pledged £50m in funding for the project
"However, unless Aberdeen City Council at that meeting decide that the project should proceed with their backing, and that they will step in and take a key leadership role along with Acsef, my financial offer will be withdrawn.
"The democratic process must decide."
Acsef chairman Tom Smith said: "The public have said they want change. They believe the gardens are underused and inaccessible.
"Significant new green space and a cultural centre must be part of this change that would give us a more attractive and safer city centre and kick-start the wider regeneration of the city centre.
"The findings reveal that opinion is divided over the City Square project, Acsef is very aware of and respects the concerns and issues raised by the public."
He said Acsef was now seeking leadership and support from the council to progress the project to the next stage.
"Rejection of this £140m investment will damage Aberdeen city's reputation," Mr Smith.
"We will be known as the city which turned down a £50m private sector donation towards an ambitious and much-needed public infrastructure development at a time when other cities throughout the UK would have grasped it with open arms."
A Peacock spokesperson said: "Peacock welcomes the feedback from members of the public.
"We are delighted to see that all of the major elements that people want to see - greater accessibility, green space and cultural facilities - will be delivered in Peacock's existing plans for Union Terrace Gardens at a fraction of the cost to the public purse and the environment.
"Given that on these figures 55% of those voting said they did not want the City Square, and the fact that people's desires will be met in the Peacock plans, we hope that the go ahead will now be given for the creation for Peacock's major new cultural centre which will regenerate the existing gardens and give the people of Aberdeen what they have asked for."
Bob Collier, chief executive of Aberdeen and Grampian Chamber of Commerce, said: "The result of the City Square consultation is a split decision.
"There is sufficient evidence in the detail, including stronger support from younger age groups; strong support for green space; support for a contemporary arts centre, and clear evidence of a desire for change, to encourage the project team to go to the next stage of an international competition having listened carefully to the views of the people."