The UK government is breaking the law by locking up too many asylum seekers before their claims are dealt with, Amnesty International says.
Asylum seekers can be detained at any time during an application
A report by the human rights group estimates 25,000 asylum applicants were held in detention centres during 2004.
It says many of these were vulnerable, such as pregnant women, families with children and torture victims.
The Home Office said detention powers were essential to ensure effective immigration controls.
Home Office minister Tony McNulty told the BBC's Today Programme that he disputed the figures Amnesty had come up with - but he conceded that people needed to be in the system for as short a time as possible.
This was true "whether they need to be removed or go on to secure refugee status," he said.
Asylum seekers can be held at any time during their application to stay in the UK.
Ministers say the powers are needed to prevent people from absconding in the run-up to removal from the UK, and the government has tripled the number of detention spaces since 1997.
But Amnesty says the government's use of detention for asylum seekers is a "lottery", dependent on the availability of beds rather than whether it is necessary or appropriate.
It says some people had been held without a final decision on their application for up to two years, a move it says is illegal because the detentions did not appear to be leading to removal from the UK.
Kate Allen, the UK director of Amnesty, told the BBC there was no evidence that people were going to abscond.
"All the people that we talked to and met in the duration of our research were abiding by Home Office reporting restrictions - they had no intention of absconding.
"There were families with young children, there were pregnant women, there were women who had been raped.
"There were people who'd been in the most appalling of situations, and there was no evidence that these people were about to abscond.
"And the criteria, as I say, was just so broad that literally anybody could be taken into detention at any moment".
The government's detention programme has been beset by problems including a riot that destroyed one centre, multiple incidences of self-harm, sporadic protests and a small number of suicides.
Amnesty said its interviews with detainees found some people were being held for long periods with apparently little chance of removal from the UK for a variety of complex reasons.
It added that more asylum seekers selected for a "fast-track" decision were now being detained for all of their application, a concern echoed earlier this month by a human rights report from the Council of Europe.
A Home Office spokesman said the Amnesty report would be studied in detail, and that the government welcomed the opportunity to have a "sensible debate" about asylum issues.
"The power to detain an individual is an essential part of protecting the integrity and effectiveness of our immigration controls," said the spokesman.
"It is also essential that those we do detain should be treated with humanity and dignity and we are committed to ensuring that this is the case."
But Maeve Sherlock, chief executive of the Refugee Council, said the report's evidence was stark.
"People are being detained arbitrarily, without proper consideration of whether it is appropriate or indeed necessary to do so," she said.
"Today marks the beginning of Refugee Week, when we celebrate the positive contribution refugees make to the UK and also reflect on why people seek sanctuary here.
"It can't be right to lock people up simply because they have asked for safety here - seeking asylum is not a crime."