The prison has 773 offenders serving life or long term sentences
Prisoners at a specialist training jail are spending too long in their cells and are not getting enough purposeful activity, according to inspectors.
But Anne Owers, Chief Inspector of Prisons, added that HMP Swaleside, on the Isle of Sheppey, had "extremely good staff-prisoner relationships".
"Swaleside has to manage a challenging population of serious offenders... we found it to be a very safe prison."
The prison currently has 773 offenders serving life or long-term sentences.
The announced inspection of HMP Swaleside was carried out in April, two years after the last full inspection.
Publishing the findings, inspectors said they found more than 40% of prisoners locked up during the "core day", and learning and skills required better strategic management.
"This weakness needs to be addressed if Swaleside is to become a first-rate prison," Ms Owers said.
But inspectors also said anti-bullying, suicide and self-harm prevention arrangements at the prison were good, and prisoners "felt significantly safer" than at comparable prisons.
However, black and minority ethnic prisoners were more negative about the prison than white prisoners, and illegal drugs were a problem. The prison was said to be working hard to reduce both supply and demand.
The Category B training prison is one of three jails managed as a cluster on the Isle of Sheppey.