At least five of the foreign prisoners freed without being deported have gone on to commit more serious crimes.
Charles Clarke has apologised after the prisoners walked free
Home Secretary Charles Clarke said the five had been convicted since release for offences relating to drugs, violent disorder and inflicting bodily harm.
One of the five had also been accused of rape but there had not been enough evidence so far to prosecute.
The Tories and Lib Dems say the latest news reinforces their calls for him to quit - but Mr Clarke says he will stay.
1,023 freed without being considered for deportation
79 of them originally jailed for more serious crimes, including 13 of them for murder, manslaughter, rape or child sex offences
Five of them known to have committed offences since release - including violent disorder, drug crimes and grievous and actual bodily harm
Deportation action now started for 63 of the 79 more serious criminals
Officials have decided nine of the offenders should not be deported
"I believe I have the ability, knowledge and talent to lead the Home Office in the best way to deal with those issues and that is what I intend to do," Mr Clarke told BBC News.
He once again apologised for the problems - and directly to the victims of the crimes committed by the criminals since their original release from prison.
The Home Office was now in the middle of "dramatic change" to enable it to meet the challenges of the modern world, he argued.
In a letter to the Speaker of the House of Commons, Mr Clarke said 79 of the 1,023 foreign prisoners freed without being considered for deportation since 1999 had been imprisoned for serious offences.
Thirteen of them had originally been convicted of the worst offences: murder, manslaughter, rape and child sexual offences, he said.
Mr Clarke said all of the 79 offenders had been identified on the police national computer and all but seven of them had now been considered for deportation, with deportation action started in 63 of them.
He also said police were still examining a small number of cases involving other released offenders - including a rape allegation and another sexual offence claim.
And more arrests were expected over the weekend, with six of the offenders already detained so they can be thrown out of Britain.
Mr Clarke said his department was now focusing "ruthlessly" on its priorities of tackling terrorism, crime, reoffending and managing asylum and immigration.
"The truth is that real and profound change does take time and often reveals matters that have been hidden or remained dormant in an organisation," he added.
For the Conservatives, shadow home secretary David Davis said Mr Clarke's statement made his position worse.
"This reinforces the need for Mr Clarke to go," he told BBC News.
"The very fact that he did not even start this process until earlier this week - he knew 10 months ago - is a major failure in his duty to protect the public."
The Lib Dems say the statement confirms their worst suspicions.
Party home affairs spokesman Nick Clegg said: "Let's be clear that those five serious offences would not have occurred but for the abject failure of the Home Office in the first place.
"No secretary of state should stay in his post when serious offences have been committed by people who could and should have been removed from the country."
A Populus poll for the BBC's Daily Politics show, conducted before the statement about the new offences, suggested that 63% of people thought Mr Clarke should quit.
Tony Blair refused Mr Clarke's offer to resign earlier this week.
Turning a blind eye?
New claims that immigration officials were effectively told to ignore the problem of released foreign criminals will add to Mr Clarke's troubles.
Conservative MP Rob Wilson has a letter from an immigration official saying the government told them to concentrate on dealing with asylum seekers instead.
In the letter, seen by the BBC, the long-serving official says: "Immigration officers have been forbidden from going out on operations, or to deal with immigration offenders arrested in police stations, unless they are failed asylum seekers".
Mr Wilson says he also received an e-mail from a senior police officer in Reading saying: "We have a whole raft of people currently sentenced to terms of imprisonment who are illegal immigrants who we have asked the immigration service to deport after sentence.
"However immigration will not track the sentence and invariably these people will be released and be back in Reading."
The Home Office has so far not commented on the allegations.