Almost 6,000 people with IQs below 70 are estimated to be in UK prisons at any one time - many of them with learning disabilities, a report says.
Report says prisoners may be unable to access education schemes
Research for the Prison Reform Trust says such low IQs will affect their ability to cope with prison life.
But it says there is no routine screening to identify their needs and they are unlikely to benefit from schemes aimed at curbing re-offending.
The government says it is already reforming offender education.
Changes include a requirement for individual needs assessments.
The report was based on the findings from a study carried out by the University of Liverpool, at the request of local NHS commissioners, in Liverpool men's Prison, Styal women's prison and Hindley, an institution for young offenders.
One prisoner with learning disabilities told the researchers: "Being in prison is frightening. People shout a lot. It's noisy. You don't know what's happening to you."
'Dropped in a maze'
Report author Dr Pat Mottram said the average IQ of prisoners was 87, while a sizeable minority had very low IQs indeed.
"Many will struggle to make sense of their experience of imprisonment," she said.
"It is important therefore that the prison regime, in particular education, health and social care and rehabilitation, takes this into consideration."
The Prison Reform Trust said the report might shed further light on the "very high" rates of re-offending.
Its director, Juliet Lyon, said it raised important questions about how such people got caught up in the criminal justice system in the first place and whether those responsible for special education, social care and family support could have done more.
"As things stand men, women and children with learning difficulties and learning disabilities in the criminal justice system are dropped in a maze with no exit, left to wander between police station, court and prison."
Skills Minister Phil Hope said the government had already set in train major reforms of offender education.
The new Offenders Learning and Skills Service required learning providers to assess inmates' individual additional learning needs and put in place additional support.
Teachers had access to in-service training to keep their practice with regard to learning difficulties and disabilities up to date.