Action is needed by prison authorities to tackle racism after complaints from ethnic minority inmates.
The prisoners at Lancaster Castle jail, Lancashire, accused governors of passing them over for release on temporary licence.
But after investigating the claims, HM Prison Inspectors said the statistics did not back up the allegations.
However, the report stated more could be done to boost the confidence of ethnic minority inmates, who make up 5% of the prison's population.
Prison governor Barry Rossiter said the inmates' previous experience of racism may influence their view of prison authority decisions.
He said: "If you come up against racism by authority then you are going to expect it in decisions made by institutions."
In line with the report's recommendations prisoners are now represented on the prisoner forum where race relations issues can be raised.