The decision to bring in ID cards for foreigners before bringing them in for Britons may lead to "de-facto racial profiling", MPs and peers have warned.
Home Office expect detection rates to rise as a result of changes
The Joint Committee on Human Rights says black and other ethnic minorities may "disproportionately" be asked to prove their UK immigration status.
From next year biometric immigration documents are being introduced for all non-EU nationals in the UK.
Ministers say colour or race can never be the reason for an immigration check.
In a letter to the committee Home Office minister Liam Byrne said the main idea of the biometric immigration documents was to allow people to confirm their immigration status easily.
Lack of detail
He said it would be unlawful racial discrimination for anyone being asked to prove - to someone such as a police or immigration officer - their immigration status because of their racial appearance.
"Therefore race or colour can never be the basis of the officer's 'reasonable suspicion' that someone is not a British citizen," he wrote.
But despite his response the committee warned: "The fact that such a document exists for non-nationals and can be requested to prove entitlement to services makes it highly likely in our view that members of black and minority ethnic communities in the UK will be disproportionately required to prove their immigration status."
The committee said it was impossible to assess the scheme's impact on the right to privacy because of a lack of detail in the Bill.
It said, for instance, it was vital that the Home Office did not allow race or ethnicity to be used to decide in what order it phases in the cards, the committee said.
Categories of people who will have to have the cards in future range from non-EU students to people who have married a British citizen and want to live in the UK.
The Home Office said there would be a "rational" basis for deciding the order in which the cards would be rolled out from next year.
That order would be based on things such as keeping the numbers manageable in the early years to selecting immigration categories where there is evidence of abuse of the system.
Mr Byrne said the UK Borders Bill would improve UK security and border control, would link into the proposed UK national identity scheme and would cut down on forgery and fraud.
The biometric documents would also reduce the scope for illegal working and lead to a "significant increase in the detection of illegal applicants" for state benefits, he said.
It is currently planned that identity cards would start being rolled out to British citizens from 2009.
It will not be obligatory for British people to have an identity card, although the government has said it may seek to make them mandatory if it wins the next election.
Neither non-EU nationals nor British people will have to carry their biometric identity document with them, although they will be needed to access a range of services.
The Conservatives and the Liberal Democrats both oppose the identity card scheme.