The IT department has been relocated to Slough Trading Estate
Slough Town Hall has been turned down from becoming a Grade II listed building by culture chiefs.
The building was described as not being of "sufficient architectural or historic interest" to merit protection.
The move has been met with anger from campaigners trying to save part of the building, dating back to 1937, who are set to appeal against the decision.
Slough Borough Council said it could invest £2m into the town that it would have spent restoring the building.
It added its priority would be to remove the dilapidated 1970s part of the building, while treating the 1930s section with "sensitivity".
In a letter, the Culture Secretary Ben Bradshaw said the town hall was a significant civic building designed by two of the leading architects of municipal buildings of the era.
But he added that the building itself was not remarkable for its planning, features or technical aspects.
A petition of 1,200 signatures was handed to Downing Street by the Campaign to Save Slough's Heritage.
It called for the building to be listed and to be put into community use for future generations to enjoy.
Martin Gibbons, from the group, said it was "shocked" and "disappointed" that the decision had been turned down.
He said: "Our campaign group is launching an appeal against the minister's decision."
The original building, off Bath Road, was built in 1937, a year before Slough officially became a town.
An extension was built during the late 1970s which has since become dilapidated, although the campaign group is not looking to retain that part of the building.
Ruth Bagley, chief executive of the borough council, said the decision was useful in order to have clarity on the future of the town hall.
"There are several possible uses for the town hall site and this ruling gives the council flexibility to make the best decision to satisfy critical community needs and still be sensitive to the history of the site," she added.
The council recently decided to relocate the building's IT room out of the town hall to a site on the Slough Trading Estate.