More than 300 wrongly released foreign prisoners, including a murderer, are still being sought, John Reid has said.
John Reid says he is making progress at the home office
They were mostly "minor" offenders and three of the four murderers had now been arrested, the home secretary said.
He told MPs he was making progress in reforming the department but said he was "not the wizard of Oz".
Mr Reid's predecessor Charles Clarke was sacked after more than 1,013 foreign prisoners were released without being considered for deportation.
Mr Reid told the home affairs committee that loophole had now been closed.
And of those mistakenly released, "129 or 130 have actually been deported, about two thirds of them, 750 or thereabouts, have been considered for deportation" said Mr Reid.
"About a third of them, the police are still looking for, but of those, they are minor offenders in the main.
"And when I came here the last time, I think there was four very major offenders, at risk, out there, four murderers.
"As of last night there were two, and as of today there is one, because earlier this morning we detained one of the two remaining very serious offenders. We arrested them this morning."
He said "we will try to deport all of them", but it was ultimately in the hands of the courts.
He insisted he was making progress with reforms, with asylum claims at a 14-year low and the department making £1.9bn in efficiency savings, 16 months ahead of its target.
But asked if he could guarantee there would not be other scandals such as foreign prisoner release - or corruption at the immigration and nationality directorate - he said: "I can promise that I will do my best to give leadership.
"I can't promise to be the wizard of Oz. I am not coming here today saying everything is perfect, I am saying progress is being made."
Mr Reid denied claims by the government's spending watchdog the Home Office has no idea how many people work for it.
The National Audit Office criticised record keeping and financial controls at the department, saying it had lost control of its payroll and run up a £246m "overdraft" with the paymaster general.
Sir John Bourn, the Comptroller and Auditor-General, gave only qualified approval to the department's annual accounts after last year failing to approve them at all because of flawed financial records.
Sir David Normington, the Home Office's top civil servant, admitted there had been "difficulties" with record keeping at the department.
"It's quite a technical issue about our ability to reconcile our human resource records with our payroll. There were some difficulties with that, which we have now put right.
"It is not the case that we do not know how many staff we employ from one month to another. It's about 77,840, round about that, at the moment."
He said there were still "underlying problems" but this year's accounts were a "step forward from last year".
Mr Reid also told the committee "tough choices" had to be made at the Home Office because, in common with other departments, it faced a spending squeeze between 2008 and 2011.
Some projects had already been cancelled, said Mr Reid, but he was still determined to increase prison places by 8,000.
And he defended plans to part-privatise the probation service, saying "new ideas" were needed because re-offending rates remained "obstinately too high" despite a big increase in investment.
On immigration, he said he had wanted to limit low-skilled migrants from Bulgaria and Romania, after the government got its predictions "wildly wrong" when the first eight East European countries joined the EU in 2004.
He said he wanted the new Migration Advisory Council to consider "wider" factors, not just the impact on Britain's labour market.