The no-confidence vote follows a report on HMP Whitemoor
Prison officers at the top security Whitemoor Prison have "no confidence" in their governor, a union has said.
Staff at the Cambridgeshire prison passed a motion saying they were dissatisfied with Steve Rodford, the Prison Officers Association (POA) said.
The no-confidence vote comes six months after an internal review by Prison Service investigators raised concerns about Whitemoor.
Mr Rodford said only 23% of prison officers were involved in the vote.
"To put this into context, the POA currently has a membership of approximately 500 members of whom 115 voted against having confidence in the Governor," he said.
"This vote of confidence was in relation to revised shift patterns, underpinned by minimum staffing levels, in line with national policy."
A report, published in May following an inquiry in February, said that Whitemoor officers "appeared reluctant" to challenge "inappropriate behaviour" for fear of "doing the wrong thing".
The report also said the high number of Muslim inmates at Whitemoor made some staff "anxious and apprehensive" and it called for an "intelligence assessment" of possible Muslim prisoner "activities".
Mr Rodford asked for an investigation after five prisoners committed suicide between November 2006 and December 2007.
"Prison officers at Whitemoor have passed a motion of no confidence in the governor," said a POA spokesman.
"This is an unusual situation at a high-security prison like Whitemoor. But there is no suggestion of any industrial action.
"They are hoping that this will result in Prison Service officials at a higher level taking action to address their concerns."
He said Whitemoor found it hard to recruit officers and the jail was understaffed.
But local union officials would be left to deal with the problems and there were no plans for national POA leaders to become involved.
Mr Rodford added that Muslim prisoners were not "a law unto themselves" at Whitemoor.
"Robust monitoring of all prisoners takes place at Whitemoor irrespective of their religion or affiliations," he added.
"Whitemoor does have groups of prisoners who have formed themselves into gangs.
"It is clear that this gang culture is a reflection of issues that exist outside of the custodial environment.
"Prisoners will inevitably bring their social structures into prison when convicted."