The number of people recalled to jails in England and Wales for breaching release conditions has risen 250% since 2000, says HM Inspectorate of Prisons.
One man hanged himself after he learned he would be returning to jail
Some 11,081 inmates were recalled in 2004/05, compared with 3,182 in 00/01.
Chief Inspector of Prisons Anne Owers said the rise was "staggering" and jails needed to be more aware of the support recalled prisoners might need.
The government said a range of measures were already in place to improve the recall process.
One prisoner who served nine months of an 18-month sentence hanged himself after being ordered to return to jail.
The man, known as Mr B, had received "a slip of paper, with no explanation" stating that he would have to serve four more years.
The full dossier explaining his situation and right to appeal arrived the following day.
Those released on licence can be recalled for breaking their conditions.
And offenders freed early with electronic tags can be recalled for breaking their curfews.
"The growth in the number of prisoners recalled to prison over recent years has been staggering," said Ms Owers, who also said offenders were not given enough information to explain their recall.
"This review suggests that prisons have been struggling to keep up and those recalled do not always receive appropriate information, care and advice," she said.
Ms Owers added: "There was little awareness that recalled prisoners might be feeling traumatised or aggrieved about their recall and in need of support."
Home Office minister Baroness Scotland said: "The government is determined that offenders who are on licence should understand that licence conditions must be complied with and failure to do so will result in swift and effective enforcement action.
"We have already implemented a range of measures to improve the recall process."
Prison Reform Trust director Juliet Lyon said the findings caused concern.
She said: "The system is setting people up to fail and recycling them through prison at great cost to themselves and the public purse.
"Prisons up and down the country are being crammed with recalled prisoners with no clear idea why they are there, how long for, or what to do with them."
Shadow prisons minister Edward Garnier said: "Prisons are clearly not capable of dealing adequately with these returned prisoners.
"The consequence of denying prisoners adequate rehabilitation will be felt by us all when they are released."
Liberal Democrat home affairs spokesman Alistair Carmichael said the report painted "a picture of chaos" in relation to rehabilitation.
"The government must take responsibility for this mess and for its failure to get to grips with re-offending."