Home Secretary John Reid is said to be furious after he had to apologise to MPs for giving them the wrong figures on foreign prisoners.
John Reid has admitted to another Home Office mistake
Mr Reid told MPs on Tuesday that four murderers, as well as 23 other "most serious" offenders, were in detention.
But now he says one murderer, a rapist and a child sex offender had actually already been released on bail by the Asylum and Immigration Tribunal.
A senior civil servant has been removed from his post pending an inquiry.
Sources close to Mr Reid say he only discovered the mistake on Tuesday after asking specific questions.
Staff from the Immigration and Nationality Directorate (IND) went to the bail hearings to oppose the prisoners being released on bail but they lost.
That information was not then passed on to the home secretary who is also said to be "perplexed" at the decision to grant bail.
The revelations come after Conservative leader David Cameron said Tony Blair should take personal responsibility for the "shambles" at the Home Office.
The latest mistakes are embarrassing for Mr Reid, who told the Commons home affairs committee on Tuesday that he was overhauling an immigration system which was "unfit for purpose".
He warned the MPs all figures he gave should have a "health warning" because in his first two weeks in the job he had found Home Office figures had always seemed liable to change.
Now he has written to the MPs, as well as Commons Speaker Michael Martin, to say some of the figures were indeed wrong.
He wrote: "To be put in a position where information was wrongly given to your committee - in spite of the caveats that I put around that information - is not acceptable and I apologise to you and your committee."
Mr Reid says he has asked officials to "re-check" all the facts given to MPs on Tuesday.
Home Office officials stress the bail decisions were made by judges and were opposed in all cases and all further bail applications would be contested "as vigorously as possible".
Mr Reid has asked his department to contest all further bail applications "as vigorously as possible" and ask for the most stringent bail conditions.
The spokesman said: "The home secretary has asked officials to ensure that all those released on bail are immediately subject to the probation licensing arrangements appropriate to their case."
In his letter, Mr Reid says the permanent secretary at the Home Office and the director general of the IND have taken "immediate management action".
The Home Office says there have not been any sackings but one senior civil servant has been removed from his post while an inquiry is carried out.
A second senior official has been moved from his post as part of a reorganisation.
John Denham, chairman of the Commons home affairs committee, said it was "quite extraordinary" that Mr Reid had given the wrong figures.
Mr Denham said from his experience it was the first time information on such a high profile case had been held back from a minister.
Shadow immigration minister Damian Green said the latest twist meant there were more dangerous people on Britain's streets than previously thought.
The Conservative frontbencher said: "When a home secretary warns that nobody can believe Home Office figures, and gets the figures wrong himself, it shows a complete failure of administration."
Liberal Democrat home affairs spokesman Nick Clegg said a task force from the National Audit Office should be called in to find out what was going on.
"It is unprecedented for the Home Secretary to admit that he cannot guarantee the accuracy of his statements to Parliament," said Mr Clegg.
Earlier, the foreign prisoners crisis was the focus of a prime minister's questions session so stormy the Speaker threatened to suspend it.
Mr Cameron said Mr Reid had contradicted the prime minister's claim that the immigration service had been transformed.
Mr Blair said much progress had been made but more needed to be done.
He said Labour had inherited a system from the Conservatives where asylum applications took 22 months to decide and failed asylum seekers were not deported.