Police searched a number of addresses in the early morning raids
Home Secretary David Blunkett, who authorised the round-up, said he had given the go-ahead after "careful and detailed consideration" because he believed they were a threat to national security.
Immigration officers, backed up by police, raided addresses in London, Bedfordshire and the West Midlands.
Under new measures - part of the government's anti-terror legislation passed last week - foreign-born terror suspects can be detained without trial.
Secure units
It is one of the most controversial aspects of the Anti-terrorism, Crime and Security Act and has been attacked by civil liberties campaigners.
It requires part of the Human Rights Act to be set aside and for the home secretary to deem the UK to be in a state of emergency.
It is thought that those held on Wednesday, who are now being detained in secure units around the country, are either of Middle Eastern or North African origin.
Mr Blunkett said he would not name them and that they were free to leave the UK at any time.
A spokeswoman for Bedfordshire Police confirmed two men were detained at two separate addresses in Luton.
She said a "handful" of police officers provided back-up to Immigration Service officials during the operation.
She said: "We were there to support the Immigration Service in a keep-the-peace type role.
"There were a handful of officers and it was just a polite knock on the door."
The BBC's Margaret Gilmore said all suspects were foreign nationals, but it was believed some had exceptional leave to stay in the country because of the risk posed to them if they return to their own country.
They were suspected of having links to the al-Qaeda network but could not be arrested under normal UK law because the police did not have enough evidence they were involved in terrorist activity.
'Unjust'
MI5 is believed to have drawn up a list of about a dozen names several weeks ago and submitted them to the home secretary for approval.
The suspects can be held for six months, after which their cases will be reviewed by an immigration appeals tribunal.
The new laws, which were passed on Friday, enable police to hold non-UK citizens without trial where deportation is not possible.
Director of the campaign group Liberty, John Wadham, said the detentions were "utterly unjust" and again pledged to challenge the legislation in the courts.
"Arrests under these powers stamp all over basic principles of British justice and the European Convention of Human Rights - even the government admits that," he said.