The shadow immigration minister says the government was wrong to house convicted criminals at immigration detention centres.
Riot police were called to the centre after fires were started
Damian Green's criticism comes after 14 offenders are still at large after fleeing Oxfordshire's Campsfield Detention Centre after a disturbance.
The convicted criminals, who were among 26 who fled, had served their sentences and were awaiting possible deportation.
Border and Immigration chief Lin Homer denied the centre's regime was unsafe.
"We've recently looked at the regime in Campsfield and we are putting in place a number of improvements with the centre operator," said Ms Homer, chief executive of the Border and Immigration Authority.
"There's no suggestion that the regime is unsafe but there's always room for us to learn from incidents and we and the centre operator remain very committed to learning those lessons and implementing any changes and improvements that we can."
But shadow immigration minister Mr Green said: "We need immigration detention centres as part of the process of removing people who have no right to be here, but what we shouldn't be doing is mixing up immigration offenders with other criminals and that's where the big failure lies."
Meanwhile, the chief executive of the Refugee Council, Donna Covey, told BBC News of her concerns on Monday.
"There's a very volatile mix of people in these detention centres and as I say they include many asylum seekers who've committed no crime.
"Recognised international bodies like the UNHCR [The UN Refugee Agency] have constantly called for countries like the UK to develop alternative detention during the asylum process."
On Sunday a spokeswoman for the Home Office said the 14 escapees, who fled on Saturday night, were foreign prisoners.
'Frustration and misery'
Superintendent Robin Rickard - in charge of the major police search - has urged members of the public to contact police immediately "should they see anyone they believe could be one of those involved".
Campsfield, at Kidlington in Oxford, was converted to a detention centre holding 200 male asylum seekers in 1993 and is run by the American company GEO.
No-one from the company was available for comment.
Local Liberal Democrat MP Dr Evan Harris has called for action at the centre which has seen a series of disturbances, including a mass hunger strike last week.
Dr Harris said: "It is not surprising that seemingly indefinite detention without charge or without conviction leads to frustration, misery and potential unrest."
Imam Sajid, a former chairman of the Joint Council for the Welfare of Immigrants, has urged the escapees to give themselves up but has also called for a change to the way immigrants are treated.