Indefinite custodial sentences are "fundamentally flawed" and should be used only in exceptional cases, the Howard League for Penal Reform says.
More than 3,000 IPPs have been given in two years
It says the sentencing option introduced in 2005 is threatening to overwhelm the overcrowded jail system.
Ministers are already reviewing the way the sentences work after successful High Court challenges by prisoners.
Inmates say many jails fail to provide chances to show they do not pose a risk to the public - a condition of release.
Under indeterminate sentence for public protection (IPP), offenders convicted of a wide range of offences are given a minimum tariff.
However, they must take courses and undergo assessments to prove they are no longer a danger to the public before they can be released.
The Howard League interviewed prisoners about the sentences for its report, which it says reveals "systematic failings" in the system.
Howard League director Frances Crook said: "The IPP is a disastrous example of the government failing to plan for the consequences of an ill-thought out policy.
"This sentence threatens to overwhelm a prison system that is already on its knees.
"If the government's attempts to appeal the recent High Court judgements fail, then hundreds of prisoners classified as 'dangerous' will be released from custody.
"Either way, the public is not being well served."
A report by the Prison Reform Trust last month said more than 3,000 IPPs have been passed in just two years.
One prisoner told Howard League researchers: "I have found it really hard to adjust to my sentence.
"I would have preferred a fixed sentence that is longer, but at least I would have a light at the end of the tunnel."
The group's document called for "discretion and a fact-sensitive approach to sentencing" if the IPP remains in use.
It said: "The legislation should be amended so that the IPP is used in exceptional circumstances, rather than as a standard prescription.
"The Sentencing Guidelines Council should provide guidance on the criteria for exceptional circumstances.
"Courts should be given discretion so that where 'there is a significant risk to members of the public of serious harm' the IPP may be imposed, rather than must."
The Ministry of Justice said it was considering the concerns surrounding IPPs, which are under review.
In the meantime, the government has allocated £3m to help prisons holding IPP prisoners to complete assessments, it added.