Established British families should be given priority over economic migrants for council housing, government minister Margaret Hodge has said.
She has called for a rethink of social housing policy, to take account of length of residence, citizenship and national insurance contributions.
Social housing was limited and British families had a "legitimate sense of entitlement" to their own homes.
But Labour MP Keith Vaz said many would find Mrs Hodge's comments "offensive".
Industry Minister Mrs Hodge said rules should "promote tolerance rather than inviting division".
She told the BBC she was aware it was a difficult issue, but she was trying to listen to her constituents' concerns and wanted to start a debate.
She said she understood the reasons why genuine refugees needed access to public resources, but said most new families were economic migrants who had chosen to come to Britain.
"In exercising that choice as an economic migrant, should they then presume to have automatic access immediately to public social housing?" Mrs Hodge said.
"Of course it's true that we need to develop more social housing," she added.
"But however much social housing you do create, you will nevertheless have to take decisions to ration what will always be a finite resource."
She said white, black and Asian British families on low incomes, who had lived in an area for several generations, could not get their own homes and all felt there was an "essential unfairness" in the system.
"They feel that they've grown up in the borough, they're entitled to a home, and that sense of entitlement is often overridden by a real need of new immigrant families who come in, perhaps locked into private accommodation, poor accommodation, overcrowded."
She won some support from Labour chairman, and deputy leadership candidate, Hazel Blears, who agreed there was a need "to tackle these tough issues."
"You have got to look at allocations policies to show that they are fair," she told BBC One's Sunday AM programme.
But fellow deputy leadership contender Jon Cruddas, a neighbouring Labour MP to Mrs Hodge, said there was a danger of "racialising arguments over housing allocation".
Instead there was a need to "focus on the need for a greater supply of decent, affordable homes".
But Liberal Democrat local government spokesman Andrew Stunell said there were 1.5 million families on council house waiting lists, but the government kept selling houses off.
'Build more homes'
"The first thing to do is start building social housing again, not to blame immigrants for the catastrophic government failure to tackle the issue," he said.
Nancy Kelley, head of international and UK policy at the Refugee Council, said: "The way to counter some of the views put forward by far-right parties is not to follow their lead."
She stressed that asylum seekers were not entitled to council housing and arrivals from new EU states had restricted access to benefits.
Mrs Hodge, who was born in Egypt, has warned before that many of her constituents in Barking, east London, were angry at the lack of housing.
She was also criticised for saying, before the council elections in 2006, that as many as eight of 10 white families were tempted to vote for the British National Party. The BNP went on to become the second largest party on Barking and Dagenham council.
Mr Vaz, chairman of Labour's ethnic minority taskforce, said: "Many people will find what Margaret Hodge has said offensive.
"As a minister in a government committed to equality she has the capacity to deal with any unfairness - if indeed it exists.
"No minister should use the language of far-right groups which only assists them in their objectives."