Sports people should not be banned for using "social" drugs, sports minister Richard Caborn has suggested.
Sailor: Admits error of judgement, but insists he wasn't cheating
Mr Caborn told MPs the key anti-doping issue in sport was whether drugs enhanced performance or not.
The police should be left to deal with athletes caught taking recreational drugs, he told the commons science and technology committee.
The Conservatives said Mr Caborn's comments were "damaging" because top sports people were role models.
Cannabis, cocaine and other illegal substances are on the World Anti-Doping Authority's banned list.
But Mr Caborn said the drug authority's job was to stop athletes using drugs to enhance performance and it should not be policing the "social aspects of life".
It should be down to the police to deal with athletes caught using recreational drugs - provided they had not been used to enhance the athlete's performance, he argued.
"We are not in the business of policing society. We are in the business of rooting out cheats in sport. That's what Wada's core function is about," Mr Caborn told the committee.
He said there was "a discussion" inside the international anti-doping body about how it should treat athletes caught taking recreational drugs.
But, he added, its code was based on three principles - "performance enhancement, harm to the athlete and harm to the sport".
And he said: "I would give far more weight to the performance-enhancing of the those three.
"And I would also look very seriously at the list, to take off what I believe are some of the social drugs."
His words came as Australian Rugby Union player Wendell Sailor withdrew his appeal against a two year drugs ban after testing positive for cocaine.
The player insisted he did not cheat but had succumbed to the temptation of a "so-called party drug" and would pay a heavier price than most.
A number of British footballers have received bans for taking recreational drugs.
In November 2004, former Chelsea striker Adrian Mutu was given a seven-month ban from football after testing positive for cocaine.
Mr Caborn also told MPs Britain would not introduce a new law before the 2012 Olympics to make doping in sport a crime.
"We think it would be disproportionate to what we want to achieve.
"What we have done over the last three years is to give political support.
"We do not want this to be overtaken by politicians. It's very important that sport should deal with its own misdemeanours."
Mr Caborn also insisted there were no conflict of interest in governing body UK Athletics policing drug-use and running the elite athletes programme, which was focused on giving athletes an "edge".
Committee chairman Phil Willis said the US and Australia had both separated out their policing and training regimes.
But Mr Caborn insisted the British system had "Chinese walls" preventing conflicts of interest and there was no evidence of abuse.
Shadow sport minister Hugh Robertson, for the Conservatives, said: "Richard Caborn's comments are wrong and appear directly to contradict UK sport's very clear stance on drug taking by elite athletes.
"The performance of an elite athlete in any sport will be affected by taking recreational drugs and top sportsmen and women act as role models for many young people.
"I hope that Tessa Jowell will disown these illogical and damaging comments."