Fewer than a third of the 2.7 million people claiming incapacity benefit are legitimate claimants, a government welfare adviser has said.
Mr Freud says most claimants should be looking for work
David Freud, an investment banker, said up to 185,000 claimants work illegally while on the benefit.
He told the Daily Telegraph it was "ludicrous" medical checks were carried out by a claimant's own GP.
The Department for Work and Pensions said the number claiming incapacity benefits was at its lowest since 2000.
"But we agree with David Freud that there are many more people who could and should be supported to move off benefits and into work," a spokesman added.
"We are implementing his review and have already committed to replacing incapacity benefit and introducing a new medical test that places the emphasis on what work a person can do, rather than what they can't."
Mr Freud published a report on welfare last year which was highly influential on reforms outlined by new Work and Pensions Secretary James Purnell this week.
Shadow work and pensions secretary Chris Grayling said every claimant should have an independent medical check.
"Those fit to work should have their incapacity benefit withdrawn immediately and those with the potential to work should get proper support to help them back into employment," he added.
Liberal Democrat spokesman Danny Alexander said: "Millions of people on incapacity benefit want to work and would be doing so today if the government had put the right support in place."
Mr Freud, who said the system was "a recipe for getting people on to [incapacity benefit]" has recommended private firms be paid "bounties" to get claimants off incapacity benefit and into jobs.
He said there was a "classic conflict of interest" with GPs carrying out claimants' medical checks, saying: "They're frightened of legal action."
Meanwhile, compared with unemployment benefit, incapacity claimants received more money and were not "hassled".
"The system we have at the moment sends 2.64 million people into a form of economic house arrest and encourages them to stay at home and watch daytime TV. We're doing nothing for these people," he told the paper.
'Conflict of interest'
Kate Green, chief executive of the Child Poverty Action Group, said the most recent official figure for incapacity benefit fraud suggests it is below 0.5%.
HAVE YOUR SAY
Making it harder to claim will put people off leaving the benefit once on it, something our gormless leaders fail to understand.
James Smith, Watford
She said: ""Ministers will surely be alarmed that the man charged with major reform of the welfare system and family security rights gets basic facts wrong about benefits that he could find out in a second with a Google.
"His suitability must be under question for the task Work and Pensions Secretary James Purnell has set him."
Claimants were assessed by independent doctors - rather than their own GP - after 28 weeks, she added.
Since the 1980s, there have been claims that successive governments have allowed the incapacity benefit-roll to grow in order to keep down the more politically sensitive count of the unemployed.
"When the whole rot started in the 1980s we had 700,000 (claimants). I suspect that's much closer to the real figure than the one we have now," Mr Freud said.
Recent figures showed that more than 500,000 people under 35 are now claiming incapacity benefit.
About 40% of recipients are claiming for mental health problems, some 250,000 because of stress-related illness, while others cite alcoholism, obesity or eating disorders.
Mr Freud said a new system, with private firms and voluntary organisations paid by results in getting claimants into lasting jobs, and those who refuse to co-operate having benefits docked, could be in place within five years.
He told the paper it would be "economically rational" to pay as much as £62,000 to a company which managed to place an incapacity benefit claimant in a job which lasted three years or more.
Incapacity benefit costs the Treasury about £12bn a year.