Leading lawyer Gareth Peirce has alleged there has been chaos over the use control orders on terror suspects.
Abu Qatada is one of those who has been detained
Peirce, who represents several men under control orders, says the conditions under which they must live have not been made clear.
They were being held without trial until last week when anti-terror laws, including control orders, were passed.
The Home Office has said "teething problems" were inevitable but they are being solved.
The control orders replaced detention without trial for 10 men thought to be a terror risk, but who cannot be deported because they might face death in their home countries.
It has also been alleged that detainees could not get through to a helpline set up by the Home Office.
Ms Peirce told BBC News: "There is profound unease at the vagueness and lack of definition.
"One man, this weekend, was genuinely worried that if he went outside to put his dustbin out, that he would be in breach of his order.
"The same man is more worried that he was locked up three years ago on evidence he didn't know and has never been told.
Gareth Peirce is known for speaking out over human rights
"If he goes out now - as he is permitted to do 12 hours a day - he doesn't know who he is allowed to speak to and who he isn't, what he is allowed to do or not do that will cross another line that might lead to him being detained."
Another lawyer for a former detainee said he had been left without food or help after his release.
Natalia Garcia, who represents one of the former detainees, said: "He was released from Colnbrook with no money and no food at all.
"He was taken to his accommodation address. There was no food there. He couldn't make any telephone calls out, so he couldn't seek assistance to get food.
"Even if there had been a line for him to ring out from, nobody would have been able to bring him food because nobody is allowed to come to his house unless they have been previously cleared by the Home Office to do so.
"The emergency contact line for the Home Office wasn't working and still isn't working, so he wasn't even able to telephone them to tell them he had no food."
One of the detainees, who had been held in Broadmoor high security hospital, had a breakdown after been moved to a flat and had to be admitted to a psychiatric hospital, the Guardian reports.
Another suspect P, who has no hands, had not been given a specially-adapted phone he would be able to use, the paper added.
A Home Office spokeswoman said: "This is a large and complex operation and it was always possible that there would be some teething problems.
"However, we are satisfied that these are now being solved."
The spokeswoman said the helpline was never intended to be 24-hour and will only staffed during office hours.