Home Secretary Charles Clarke is facing increasing pressure to resign after he revealed five immigrants released from prison had gone on to re-offend.
Mr Clarke insists he is the right person to sort out the problem.
They are among hundreds of foreign prisoners freed at the end of their sentences without being deported.
Police and immigration officers are undertaking a series of raids to detain some of the higher risk offenders.
Opposition MPs are calling for Mr Clarke to quit. Tory David Davis said there had been a "massive failure".
Liberal Democrat president Simon Hughes told BBC News 24: "Somebody has got to put things right...but it does not have to be him (Charles Clarke).
"In politics, if things go wrong then somebody has to carry the can and that is the person at the top."
The Home Office says more than 200 police and immigration officers are taking part in the raids across Britain, to track down 63 high-risk foreign offenders, following initial raids overnight.
They are among the 1,023 prisoners released into the community without being considered for deportation over the past seven years - 79 of whom had been in jail for serious offences.
It comes after the home secretary said on Friday that five of those released had reoffended and had been convicted for offences relating to drugs, violent disorder and inflicting bodily harm.
Two have also faced rape claims, with one case dropped because of lack of evidence.
The other case, which is still under investigation, is alleged to have taken place in August 2005.
This was after a report had been published calling for more speed in dealing with overseas criminals nearing the end of their sentences.
Action 'too late'
Mr Clarke has so far dismissed fresh calls for his resignation.
But shadow home secretary David Davis, speaking on BBC Radio 4's Today programme, said it was time for Mr Clarke to quit.
He said the home secretary was told "effectively for the fourth time, but told in terms 10 months ago" about the problem and should then have taken action.
Labour voices have been among those calling for Mr Clarke to resign.
1,023 freed without being considered for deportation
79 of them originally jailed for more serious crimes, including 13 of them for murder, manslaughter, rape or child sex offences
Five of them known to have committed offences since release - including violent disorder, drug crimes and grievous and actual bodily harm
Deportation action now started for 63 of the 79 more serious criminals
Officials have decided nine of the offenders should not be deported
Labour MP Lindsay Hoyle told BBC News: "The public out there expect to be protected and what they're saying is, 'Hang on a minute, if these people have had seven months, 10 months to put this right, why didn't they? Why did releases continue?'"
Meanwhile another Labour MP, Graham Stringer, echoed Mr Hoyle's sentiment.
"As the facts change, then it's likely that Charles's position will become untenable," he told BBC Radio Four's PM programme.
He added: "I think the prime minister, in not accepting his resignation, is being cruel and unfair to the home secretary. Charles is a man of integrity: he did the right thing in the first place, and he should have been allowed to go."
However fellow Labour MP Ian Gibson told BBC Two's Newsnight the home secretary had the "next week" to sort out the issues.
Labour MP Stephen Pound told News 24 that Mr Clarke should be "part of the solution" to the problems.
George Howarth was another party colleague of Mr Clarke's who publicly backed the embattled Home Secretary, saying that he was the right man to fix the problems.
Mr Clarke said the 79 serious offenders had all been identified on the police national computer. All but seven had now been considered for deportation, with action started in 63 cases.
Six have already been detained pending deportation.
Among them was Kyle Bester from Zimbabwe, who was convicted of rape but not deported upon release. He is being held at Belmarsh top security prison.
His father, Andre, said Mr Bester was appealing against his rape conviction and had been told by the Home Office that he could remain in the UK until it was heard. He thinks his son's arrest has been prompted by political events.
"I think it all boils down to Charles Clarke and what has been going on in the media lately and they are just looking for a scapegoat," he said.
Prime Minister Tony Blair refused Mr Clarke's offer to resign earlier this week.