Plans to build the stadium at the Maze have proved controversial
Some civil servants have tried to stop the Maze stadium project, Culture Minister Edwin Poots has said.
He was speaking after Belfast City Council announced a shortlist of five potential alternative stadium sites.
Mr Poots said there had been opposition to the Maze for a long time within the corridors of power.
John Corey of Nipsa, the union which represents civil servants, said it was "unfair and wrong" for the minister to publicly criticise his members.
He said civil servants were "bound by a strict code of ethics" which included an obligation to "provide ministers with information and advice based on evidence".
The First Division Association, which represents top civil servants, described Mr Poots' comments as "cowardly".
Its general secretary Jonathan Baume said it was "unacceptable for ministers anywhere to attack their civil servants, when they couldn't answer back".
Mr Poots said: "There have been those within the civil service who have been opposed to the project at the Maze right from its conception.
"I would have to say that a lot of the work being done within the civil service has been to stop it, and that has been the case both before devolution and during devolution.
"Ultimately, members are going to have to make up their minds on this issue."
£100m price tag
Last month, it emerged that the projected cost of building a stadium at the former Maze prison site was thought to be substantially higher than its reported £100m price tag.
One source told the BBC the cost was at least tens of millions of pounds higher.
The site for a national stadium in Northern Ireland has divided political opinion.
Sinn Féin has said the party will not support the idea unless it is on the site of the former Maze Prison.
However, some unionists oppose the Maze site because of plans for a conflict transformation centre at the stadium.
The Gaelic Athletic Association, Irish Football Association and Ulster Rugby have all confirmed they would play games at the Maze venue.