There is a "myth" the system is unfair, ministers say.
There is no evidence that new arrivals in the UK are able to jump council housing queues, an Equality and Human Rights Commission report says.
Once they settle and are entitled to help, it adds, the same proportion live in social housing as UK-born residents.
The prime minister has said guidance is to change so local people get priority for social housing in England.
Housing minister John Healey said the changes were designed to tackle the "myth" that the system was unfair.
The changes, he said, would not alter the policy of awarding council housing on the basis of need or the fact that economic migrants from outside the EU cannot apply for housing for the first five years after settling in the UK.
But councils would get more "leeway" to deal with specific housing pressures in their areas.
COUNCIL HOUSE RULES
Since 1996, councils have had to give "reasonable preference" to certain groups in allocating housing.
These are the homeless, those living in overcrowded accommodation and people who must move for medical reasons.
Since 2002, it has been illegal for councils to exclude anyone who does not have a local connection.
However, councils can take into account length of residence and family links when choosing between applicants within the preference groups
The British National Party has campaigned heavily on the issue, claiming that British people are being short-changed for housing in favour of newly arrived immigrants.
Mr Healey said it was "wrong" to say the government had been forced into action to counter the BNP's arguments after it succeeded in getting two Euro MPs elected.
But he added that he wanted to "nail the myth" that certain groups were losing out in terms of housing allocation.
"It is largely a problem of perception," he told Today.
"The report shows there is a belief, a wrong belief, that there is a bias in the system."
Although the rules would not change, Mr Healey said new guidance to councils will enable them to help people who have been waiting the longest or those, in rural areas, who have strong local or family connections.
"There is more we can do to give local authorities more freedom and scope to give more preference," he said.
He added that changes to the system of allocation were "no substitute" for building more council houses.
Ministers say they will build an extra 20,000 affordable homes over the next two years, in addition to 90,000 already announced, but critics say this is inadequate given the four million people now on a waiting list for homes.
The Commission's report - based on figures from the 2007 Labour Force Survey - was carried out by the centre-left Institute for Public Policy Research think tank.
According to the study, 64% of people who arrived in the UK within the last five years live in private rented accommodation.
Just 11% of new arrivals get help with housing - almost all of them asylum seekers.
But after five years, when many immigrants are able to get residency and become entitled to government help, one in six live in social housing - exactly the same proportion as those who were born in Britain.
1.8% of social tenants have moved to the UK within the past five years.
87.8% were UK-born while 10% were foreigners living in Britain for more than five years.
Source: IPPR report
Mr Phillips said ministers and social housing providers needed to work together to address the issues that resulted in perceptions about immigrants benefitting unfairly.
He added: "We have to recognise that people's perceptions are powerful, so it's vital that social housing providers and policy makers work to foster understanding about what is really happening on the ground.
"Much of the public concern about the impact of migration on social housing has, at its heart, the failure of social housing supply to meet the demands of the population.
"The poorer the area, the longer the waiting lists, therefore the greater the tension."
On 29 June, the prime minister told MPs he wanted to allow councils in England to give additional preference to locals.
That was seen as a broadside to the BNP, which has claimed immigrants are able to get more help with housing.
BNP leader Nick Griffin said the housing shortage was caused as much by the number of asylum applicants paid to live in private accommodation as migrants getting council houses and the two could not be seen in isolation.
He said foreign migrants had been "pushed to the front" of housing queues in recent years, saying: "This is not a perception. This is a fact."
Views on council housing: "Local people get nothing"
The Conservatives have accused Labour of "spin", adding that the forthcoming equality bill would make it illegal for councils to discriminate in favour of local people.
"Labour's cumbersome laws tie the hands of local authorities, compounding the problems of a growing housing waiting list and aggravating community tensions," said its housing spokesman Grant Shapps.
"Housing waiting lists are at a record high level because of Labour's historic failure to build enough social housing."
The Lib Dems said the government was trying to "grab headlines" rather than deal with the underlying issues.
"Gordon Brown has been caught out openly pandering to a misconception," said its housing spokeswoman Sarah Teather. "The real problem is the queue not the method for queue-jumping."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.