Minister Chris Mullin has said "any decent democrat" would be concerned over the detention without trial of foreign terror suspects at Belmarsh.
Mr Mullin is a Foreign Office minister
Law lords have ruled the detention, under anti-terrorism laws, is in breach of human rights legislation.
Home Secretary Charles Clarke this week said he was hoping to reach deals so suspects could be safely deported home.
Mr Mullin told ePolitix.com those held could choose to leave the London jail if they also left the UK.
He added: "But they have to find a country willing to take them and because of their former activities or the present situation of their own country, they aren't able to do that.
"This is how they come to find themselves where they are."
Mr Mullin - who serves under Jack Straw in the Foreign Office - said there were several safeguards built into the powers that permit the detentions such a requirement to renew the terror legislation on a regular basis.
"I don't think anyone can take satisfaction in this situation but at the same time we can't have people who have been mixed up one way or another in terrorism allowed to just walk the streets."
The detention without trial of a total of 12 men at Belmarsh, London, and Woodhill, Milton Keynes, was ruled unlawful by Britain's highest court in December.
Those affected by the ruling are from Algeria, Tunisia, Egypt and Jordan.
Critics of plans to deport them fear they could be tortured or executed if sent home.
Mr Clarke has said that he is seeking "memorandums of understanding" between overseas governments and Britain to ensure suspects would not be subjected to the death penalty on their return.