Mr Phillips said talk of an immigration cap was pointless
More help is needed for areas where there is a "white underclass" which has been "neglected" by existing equalities policies, Trevor Phillips has warned.
"In some parts of the country the colour of failure is not black and brown... it's white - especially in some rural areas," he told the BBC.
"We fail to deal with it at our peril," the equalities commission chief warned.
"There are people who are waiting with an answer and it's a political answer and it's a nasty one," he added.
Mr Phillips, chairman of the Equality and Human Rights Commission, had earlier told a CBI conference on migration that resentment of immigrants could increase as a result of the economic downturn.
He used the example of a mother who loses her job and then sees "a clever, young Latvian with three degrees doing the job she would like to do".
"It doesn't take a rocket scientist to work out how she's likely to feel. She then sees an overworked nursery teacher with a class of 30 with 15 languages. We know who she's going to resent."
Mr Phillips also criticised those talking about a potential cap on immigration to the UK - calling it "pointless" and possibly dangerous.
He said that talk of a "numbers game" was "holding out a promise that can't be delivered for a problem that does not exist".
He said the British population would not hit 70 million but could not be capped anyway because EU citizens have the right to settle in the UK and millions of Britons abroad could return at any time.
Immigration minister Phil Woolas said last week that the population would not rise above 70 million in Britain.
Addressing the same conference in London, he said "let me make clear - the government is not proposing to cap migration... but we need to recognise that the labour market might change".
Mr Phillips also said that large numbers of East European migrants were returning home as the economy slowed - helping the UK "export" its unemployment.
He said that if they were not leaving the country the unemployment rate could be heading for three million, rather than two million.
Shadow home secretary Dominic Grieve told the summit the new points based immigration system was "a step in the right direction".
"However, a points based system without an upper limit is pointless.
"Whilst we cannot restrict inward EU migration, this would allow us to control migration from outside the EU, which is around two-thirds of the foreign nationals arriving in the UK each year."
Mr Grieve said annual net migration from outside the EU had risen 144%, from 88,000 in 1997 to 215,000 in 2006. The total net migration from outside the EU was two million since 1997, he said.