Labour's record on dealing with failed asylum seekers is "a tale of serial failure and serial incompetence", shadow home secretary David Davis says.
The MPs want more failed asylum seekers deported
He spoke after a committee of MPs said failed asylum seekers were not being removed "anywhere near fast enough".
Many people refused asylum know there is little chance of being caught - a situation the Public Accounts Committee says is "extremely serious".
But Prime Minister Tony Blair said: "The progress has been enormous."
In its report, the committee warns that it would take between 10 and 18 years to clear the existing backlog of failed asylum seekers.
More electronic tagging or detention must be considered to cut the backlog, it says.
The report comes after senior immigration judge Mr Justice Hodge said last month that judges felt they could make little difference because too many of the migrants they expelled were not sent home.
The Immigration and Nationality Directorate estimates there is a backlog of between 155,000 and 283,500 people due to be removed from the UK.
By last September 1,350 failed applicants were being removed every month - although this was still below the number of people being refused asylum.
The committee criticises the directorate for being "vague" about how many failed asylum applicants are in the country.
And it says the agency does not know where most of them are, including 400 criminals released from prison into the community.
The committee's Tory chairman Edward Leigh said: "Failed asylum applicants are in increasing numbers staying in this country knowing that there is very little likelihood they will be apprehended and removed."
Backlog of failed asylum seekers due to be removed from the UK: 155,000-283,000
Time the PAC says it will take to clear the backlog: 10-18 years
Voluntary departures from UK in January and February: 770
Time taken to decide asylum application: 90% made within two months under new rules
He called for a clear target for seriously reducing the removals backlog with more staff put on the front line.
"The integrity of the UK's asylum application process is at stake," argued Mr Leigh.
Shadow home secretary Mr Davis told BBC Radio 4's Today programme it was clear the government did not have the situation under control.
"It is a story I'm afraid of serial failure and serial importance," he said.
"The fact is that they lose track of people - I mean they only know of 155,000 of the probable 280,000 that are still here.
"Of those they only know the address of a quarter of them. How on earth can you deport people if you don't know where three quarters of them are?"
However, Mr Blair said the numbers of asylum seekers were down to "a fraction" of what they were a few years ago because of the measures taken by government.
"We are now removing five times as many failed asylum seekers as when we came into office," he said, following talks with Swedish Prime Minister Goran Persson.
But Mr Blair conceded that there was still "more to do... to make sure that we handle the applications quickly, get the people who are failed out of the country".
It was also essential to get "the right court decisions" to remove failed asylum seekers, he added.
Home Secretary Charles Clarke said the government was "winning the battle" regarding removals of failed asylum seekers from the UK.
Immigration Minister Mr McNulty said he was "pretty sure" the backlog of people awaiting removal was nearer 155,000 than 283,500, but he admitted he had "no idea" what the exact figure was.
He claimed the MPs' report was based on 2003/04 figures, which include people from the EU accession states - some of whom may have returned home or regularised their position.
But he said the government was "either doing or seriously considering" many of the committee's recommendations.
Asylum cases were now being more tightly managed so claims were settled quickly and accurately, he said.
And 770 people had voluntarily left the UK in the first two months of 2006 through a government assistance scheme.
Detention and tagging were being used more where they were appropriate, he added.
Lib Dem home affairs spokesman Nick Clegg said: "This is further evidence of the Home Office's inability to administer the asylum system."
The Refugee Council - the largest group working with asylum seekers in the UK - says arresting and detaining them is the most expensive option and unnecessary.