Prison service leaders are being criticised amid claims they are relocating an office from Corby because the town is too white.
90 jobs will be lost from Corby's Crown House
One of the factors behind the centre moving from Corby to Leicester, with the loss of 90 jobs, is because the city boasts a "diverse workforce".
Other reasons given for relocation from Northants is more people in Leicester hold degrees than those in Corby.
The local Conservative Association has attacked the decision.
Crown House has handled prison contracts for catering, transport and prisoner escorts for more than 30 years.
Staff do not yet know when their jobs will end.
Almost 94% of the population in Corby are white British, compared with just under 60% in Leicester.
Philip Hollobone, Conservative MP for Kettering, said: "I think most people would like to see the best people employed at the Prison Service whether they are black, blue, green, white or whatever colour.
"In Corby there is a very loyal workforce, with many young mums who work on a part-time basis.
"They are being told the office is being relocated to Leicester, which is bad enough, but to be told they are too white is completely ridiculous."
The Prison Service said the taxpayer would benefit from the move as key factors influencing relocation to Leicester were the closer proximity to more prisons, better transport links, the availability of more professional staff and a more diverse workforce.
A Prison Service Spokesman said: "The potential sale of the property has given Her Majesty's Prison Service the opportunity to look at what provides the best option for the Prison Service and therefore the taxpayer in the medium to longer term.
"The key factors that influenced our decision to relocate to Leicester were the closer proximity to a greater number of prison establishments, better transport links, the ability to recruit professional staff and the ability to attract a more diverse workforce."