Ministers say thousands of knives have been seized during search operations
Only one person was given the maximum four-year jail term for carrying a knife last year, figures have revealed.
In the year to September, one defendant out of more than 3,700 caught with a knife in England and Wales was given the four-year sentence.
Shadow home secretary Chris Grayling, who uncovered the figures in parliamentary answers, said the "wrong messages" were being sent out.
Ministers say more knife carriers are being jailed and fewer given cautions.
The figures revealed that of the 3,713 people sent to prison for carrying a knife, more than half were jailed for three months or less.
Only one in six received a jail sentence of more than six months.
The maximum term was doubled to four years in February 2007 but has barely been used despite a string of high-profile stabbings and public fears over the dangers of knives.
No-one was jailed for the maximum term in 2007.
Mr Grayling said: "It's demoralising for the victims of knife crime if there is an expectation of a tough sentence for offenders which doesn't materialise in reality.
"We have to create a real disincentive for people who carry knives, and the fact that the supposed tough sentences just aren't happening sends out all the wrong messages."
Much of the concern over the proliferation of knife crime has revolved around youths, many who say they carry blades for security.
In 2008, 43 youngsters aged 10 to 19 were stabbed to death. So far this year, 15 in the same age group have died from knife wounds.
Last year, the Home Office responded to the concerns with an awareness campaign - backed by England footballers David Beckham, Rio Ferdinand and David James - and ploughed £2m into 10 police force areas to combat the problem.
The cash paid for 85 extra metal detecting search arches and 566 "wands" for use in anti-knife operations, education programmes in schools and sting operations against traders who sell knives to under-18s.
The government said last year that more than 2,200 knives had been seized during targeted stop-and-search operations.
It also changed the guidelines for police and prosecutors to extend the "presumption of prosecution" for anyone caught carrying a knife to those aged 16 and 17.
It had previously only applied to people aged 18 or over, with younger offenders more often being cautioned.
Despite these changes, justice minister David Hanson said sentencing in individual cases was a matter for the courts.
He said: "Our figures show more people are going to jail for carrying knives and that they are getting longer sentences."
A recent poll conducted for the BBC found a majority of the public back mandatory sentences for anyone caught carrying a knife.
Pollsters found 62% of those questioned said knife offenders should automatically be jailed.
A similar proportion, 61%, said current sentences were not strong enough to deter knife-wielding thugs.