Using a phone while driving was made illegal in 2003
The number of drivers caught using mobile phones at the wheel has increased by nearly a third, according to Ministry of Justice figures.
Police in England and Wales issued 164,900 fixed penalty notices in 2006 - up 38,100 on the previous year.
Using a hand-held phone while driving became illegal in 2003.
Of 12.7m driving offences dealt with by police or local authority penalty charge notices, the largest group was "obstruction, waiting and parking".
The biggest group dealt with by police action was 2m speeding drivers, representing 40% of all the offences they dealt with. However the figure was eight per cent lower than 2005.
Driving using hand-held mobile - £60 fine + 3 points on licence
Driving safely whilst using hands-free - no penalty
Driving in distracted fashion whilst using hands-free - £60 fine + 3 points on licence
The total number of driving offences dealt with by police action or penalty notices was 12.7m, down three per cent on the previous year.
More than 2,000 drivers were fined by magistrates for mobile phone use, and one was prosecuted in crown court.
The offence of using a phone at the wheel contributed to an ongoing rise in the number of careless driving offences, which has risen from 86,400 in 2003 to 233,600 in 2006.
The ministry's report said the figures showed a large number of police forces were taking action against phone users.
Having been given powers to deal with mobile phone offences, the police are putting them into action, said Chief Constable Steve Green, head of road policing for ACPO.
"These figures show that we're out there actively prosecuting people who use their phones in the car. If you use your phone there is a very good chance that an officer will see you, stop you, and prosecute you for that offence," he added.
Chief Constable Green added that enforcement was just part of the battle.
He stressed the need for ministers to continue with "education, with publicity, to make sure that people know that if you're using your phone in the car, be it hands-free or hand-held, then you are less capable of handling that car than a drunk".
Norman Baker, Liberal Democrat transport spokesman, fears that the message is not getting through.
He said: "It is alarming that the number of drivers who are using mobile phones while driving has risen, despite the obvious dangers.
"Talking on a hand-held mobile while driving must become as socially unacceptable as drink-driving has," he added.
The statistics also show that the number of penalty charge notices issued by traffic wardens are going up. They issued 7.8 million notices in 2006, 62% of the total.
The annual figures cover the number of motoring offences and breath tests dealt with by the police in England and Wales.