Jon Cruddas has attacked Immigration Minister Liam Byrne over his "extraordinary" response to Lib Dem calls for an illegal migrant amnesty.
Jon Cruddas is a "big supporter" of Gordon Brown
Saying migrants "should go home" put Labour on the "wrong side of the debate" on immigration, said the MP.
He said Labour had to address people's concerns on population changes without trying to "outdo" the far right.
Mr Cruddas, who was second in Labour's deputy leadership contest, was speaking at a conference fringe event.
He told the Bournemouth meeting, which was organised by the Immigration Advisory Service and Strangers into Citizens, would be a key issue at the next election.
"I am a big supporter of the changes that are occurring in the Labour Party, the change of leader in the past year," said the Dagenham MP.
"In Manchester, barely a year ago, the feeling in the air was almost toxic. There was a real nastiness around and the feeling then was whether we could retrieve the party. We were in freefall.
He said Gordon Brown had succeeded in signalling a "new type of political engagement".
'Off the radar'
"But the one area that we haven't cracked is this area of extraordinary demographic change."
He said "the velocity of change in some of our communities is quite extraordinary", but it was happening "off the radar" because official statistics were too slow to react.
This meant some communities were left short of resources and migrants were more vulnerable to exploitation.
He said the issue of the exploitation of migrant workers was "beginning to move centre stage despite the politicians".
And ideas such as an amnesty or "regularisation" - for illegal immigrants who have settled in the UK and are contributing to society - as proposed last week by Lib Dem home affairs spokesman Nick Clegg were starting to gain ground.
But the government was not helping by adopting the language of the right.
"I was pretty disappointed when I saw the tenor of the debate that followed the Liberal discussion of 10-year regularisation at their conference," said Mr Cruddas.
"Liam Byrne, the immigration minister, his response was they should go home. And these people are jumping to the front of the queue for jobs and services, which I thought was an extraordinary response."
Mr Cruddas, who supports a two-year work permit for people who had already been working in the UK for four years or more, said the government had to stop thinking the only way to take on the right over immigration was to move to the right.
"As soon as this starts percolating around, we are on the wrong side of this debate".
He said any amnesty for illegal migrants had to be part of a wider effort to combat rogue employers who were responsible for "extraordinary" exploitation.
He called for a "real time demography to allow resources to be allocated in real time, to help communities who disproportionately take the strain as we undergo the most profound demographic changes".
"The only way public policy-makers can resist that dog whistle of the far right is to deal with those fundamental material insecurities themselves."
He said his Dagenham constituency, in east London, would be on the BNP's frontline at the next election, with the party topping 40% of the vote in some local council wards.
"I don't see the solution to the far right as to try and outdo them in the language they use. The solution is to offer an alternative analysis and remedy."
Giving his reaction last week to the Lib Dem proposals, Liam Byrne said: "I believe those here illegally should go home - not go to the front of the queue for jobs and benefits.
"That's why we're now deporting someone every eight minutes and doubling our frontline enforcement resources."