Each of the four have been denied a passport
The government has refused to issue two more British men freed from Guantanamo Bay with passports, it is reported.
Their solicitor says Moazzam Begg and Richard Belmar were tortured and the decision was based on false information obtained during their detention.
Martin Mubanga and Feroz Abbasi - freed from the camp at the same time as the pair - have also been denied passports.
They received a letter from the government suggesting they would target UK or allied interest from abroad.
Home Secretary Charles Clarke used the Royal Prerogative - a set of historic powers enjoyed by the monarchy which have been passed to politicians - to issue the bans.
Other prerogatives include the declaration of war and the calling of elections but they have only been used to withdraw passports 13 times since 1947 - the last being in 1976.
Solicitor Gareth Peirce, who represents Mr Begg and Mr Belmar, said letters from the home secretary indicated he had made his decision based on information obtained while they were in US custody.
She also disputed the information's validity and insisted her clients had been tortured by US authorities.
"At no time of course was any moment of their custody lawful," she said.
"Each was kidnapped by US forces in Pakistan, neither was made the subject of any process of extradition or deportation, both were tortured in Bagram airbase and tortured in Guantanamo Bay.
"Their entire captivity was and was meant to be inhuman and degrading.
"We have asked the Home Secretary for disclosure in its entirety of the information upon which he relies and the circumstances in which it was obtained."
Ms Peirce said she "deplored" the "unjustifiable" decision to refuse passports, but hoped to discover what the UK government knows about the activities of American staff in their detention camps.
Mr Begg, 36, from Sparkhill, Birmingham was arrested by the CIA in the Pakistan capital of Islamabad in February 2002.
Mr Belmar, 25, from London, was first detained, in Pakistan before 11 September.