Only 1.1% of the drivers who admit to using a mobile phone behind the wheel are getting caught, the Liberal Democrats have suggested.
It is thought that 6.7m people could be driving whilst on the phone
Figures released by the Home Office show that 74,000 drivers were caught and fined in 2004 in England and Wales.
The Lib Dems combined those figures and an insurance firm's survey on driving habits to calculate that 6.7m drivers dial and drive.
The survey showed 21% of drivers said they used a mobile phone while driving.
Liberal Democrat transport spokesman Alistair Carmichael said a lack of traffic police may have led to the numbers being so low.
"Driving while on the phone can be extremely dangerous, and it is concerning that so few drivers are being caught," he said.
"Much tougher punishments must be issued to discourage drivers from driving whilst on the phone, as it is the cause of countless serious accidents."
According to Home Office minister Tony McNulty, police in the London area issued the most fixed penalty notices, with 13,581 in 2004, followed by Greater Manchester and the West Midlands.
The recent survey by Natwest Car Insurance that suggested 21% of drivers regularly used a phone while driving surveyed 2,349 people.