Prison officers at a West Yorkshire jail have been advised in a report not to wear St George's Cross tie-pins.
The report said race relations were a weak area in the prison
The wearing of England's national symbol by staff at Wakefield Prison could be "misinterpreted", said the report's section on race relations.
Chief inspector of prisons Anne Owers said she had been told the pins were bought in support of a cancer charity.
But one of her formal recommendations was that "staff should not wear unauthorised tie-pins".
Ms Owers said race was a "weak area" of prison policy.
Her report highlighted "discrepancies in the representation of black and minority ethnic prisoners in some key areas, such as use of force, segregation and the few available jobs".
Brian Caton, of the Prison Officer's Association, said: "If the only problem the chief inspector found was tie-pins carrying the Cross of St George, which is after all the English national flag, then there can't be a lot wrong with Wakefield prison.
"Staff at this establishment are among the most professional in the Prison Service and deal with some of the most difficult, damaged and dangerous inmates, including some of the country's most high-profile criminals."
Soham killer Ian Huntley, last week handed a 40-year tariff for murdering Holly Wells and Jessica Chapman, is among inmates at Wakefield, where serial killer Harold Shipman killed himself in January last year.
In her report, Ms Owers said: "However, Wakefield is clearly a prison on the move.
"But there is a great deal of movement still required in order to make it a fully effective prison, able to engage properly with the serious and difficult offenders that it holds."