Prison sentences of less than a year should be abolished because they do not work, a motion passed by the Prison Governors' Association (PGA) says.
The motion adds that short-term sentences do not reform criminals, and contribute to record overcrowding.
Some 65,000 out of 100,348 prisoners sentenced in 2008 were given sentences of 12 months or under.
The Magistrates' Association said governors have a duty to manage sentences and should not "meddle".
The PGA national executive committee passed the motion at their annual conference, which condemns the record rise in prisoner numbers as a "failure of penal policy".
Andy Tighe, BBC home affairs correspondent
Until now it has mainly been liberal-minded prisoner reformers who questioned whether this represents value for money in the case of offenders who receive shorter sentences.
But one senior prison governor told me that scrapping such sentences is the only pragmatic solution to the dual problem of rising prison numbers and "efficiency savings" imposed by the Ministry of Justice.
Otherwise, he warned, offenders with mental health and drug or alcohol problems will simply have to be locked up for ever-longer periods. No chance, in other words, of getting the kind of help they need to turn their lives round.
But not everyone agrees. The Magistrates' Association, who would effectively lose their powers to send anyone to jail, say even short sentences make sense for repeat offenders who breach community penalties.
It says short jail sentences are ineffective at reforming criminals and should be abolished in favour of community punishments.
Last week there were 84,354 prisoners in custody, despite 2,500 inmates being released more than two weeks early from their prison sentences every month.
The government has pledged to increase prison capacity to 96,000 by 2014.
Paul Tidball, president of the PGA, says prison is not the right place for the mentally-ill and drug addicts, and more spending on fewer prisoners would be more effective in tackling serious offender behaviour.
"We really can't afford, literally, to be locking people up unnecessarily, ineffectually and so pointlessly. More on less would be a saner approach," he says.
"For instance, women who were in for multi-shoplifting and their real problem is a drugs problem, so putting them into prison, and ejecting them from a short prison sentence a few months later isn't actually achieving much at all, except the big bill for the taxpayer."
Magistrates' Association chairman John Thornhill says governors have a duty to manage sentences and should not be "meddling in our duty to decide a proportionate sentence".
He said: "In reality magistrates send very few offenders to prison and only as a last resort."
CRIMES PUNISHABLE BY UP TO 12-MONTH SENTENCE
Theft from a shop, involving significant intimidation or threats - six weeks (starting point) Community Order to 36 weeks (sentencing range) seven years (maximum sentence)
Burglary in a building other than a dwelling involving goods valued at £2,000 or more but less than £20,000: 18 weeks (starting point) Community Order to 12 months (sentencing range) 10 years (maximum sentence)
Assault occasioning actual bodily harm involving pre-meditated assault resulting in minor, non-permanent injury: 24 weeks (starting point) 12-36 (sentencing range) five years (maximum sentence)
Cruelty to a child involving short term neglect or ill-treatment: 12 weeks (starting point) Community Order to 26 weeks custody (sentencing range) 10 years (maximum sentence)
Source: Sentencing Guidelines Council. Guidelines are for first-time offender aged 18 or over who pleaded not guilty. Case circumstances may make it appropriate for provisional sentence to fall outside sentencing range
He added that at any one time, only 5% of the prison population had been sent there by magistrates.
Ministry of Justice figures for the first quarter of 2009 show that the average sentences for theft and handling stolen goods, fraud and forgery and motoring offences were all under 12 months.
Prison reform groups say the proposal shows the "depth of feeling and despair" amongst prison governors faced with a rising number of inmates, but warn it could see more, longer sentences.
"The proposal could lead to 'up-tarriffing' where sentencers were determined to send an offender to prison," says Geoff Dobson from the Prison Reform Trust.
"But I sympathise with the argument that short sentences lead to the same people coming in and out of prison - it's the old 'revolving door' argument," he adds.
The Ministry of Justice says it has taken steps to toughen up community sentences which it acknowledges can be more effective for less serious offenders, but it adds that it is important for judges and magistrates to have a range of options open to them when sentencing.
"We are clear that prison is the right place for the most serious, violent and persistent offenders, and we will always provide enough prison places for those who should be behind bars," a spokesperson said.