The committee said the government had to do more to address concerns
Rapid immigration has damaged community relations in parts of England, a report by the Commons communities and local government committee says.
In three areas with high immigration - Peterborough, Burnley, and Barking and Dagenham - community cohesion is among the lowest in the country, the MPs say.
The report said there was "significant public anxiety" over issues such as pressure on public services.
Ministers said action was being taken to minimise the impact of immigration.
In their report, Community Cohesion and Migration, the MPs say many migrants make "significant contributions" to local communities - working in the NHS or other public services.
But it said there was "significant public anxiety" in some areas about immigration, which it warned "cannot simply be dismissed as expressions of racist or xenophobic sentiments".
Some concerns arose from "practical concerns" - such as overcrowded accommodation and pressure on public services - such as sharp rises in the numbers of primary school children who do not speak English well.
In the three areas visited, community cohesion - measured by how many people believe those from different backgrounds get along - is among the lowest in England, the report said.
It said public services were under pressure because government funding was being based on "flawed population data".
The committee criticised plans for a levy on migrants' normal visa fees to help fund public services like policing which it said were likely to generate "very little" money.
Press reports suggest it could be only £15m, something the committee says would be "a drop in the ocean".
Immigration Minister Liam Byrne said: "We do think it is right that newcomers are asked to pay a little bit more for public services.
"Actually we propose that the fund should raise tens of millions of pounds every year.
"This vital cash will allow us to channel money quickly to public services wherever there is a short term pressure from migration."
But the committee said it might be unfair, as not all immigrants would pay into the fund. EU citizens and anyone moving within the UK would not have to pay.
Instead the government should "immediately establish a contingency fund capable of responding effectively to the additional pressures which may be put on local government services from migration".
The Local Government Association has repeatedly called for a £250m contingency fund to be made available for under-pressure councils.
But Dr Dhananjayan Sriskandarajah, of the Institute of Public Policy Research, said he was disappointed by the committee's "negative conclusion".
"We're risking turning everything that migrants do into a problem and forgetting that they are dynamically contributing to the local economy and to the country because they are working and paying taxes," he said.
He said only better population figures and a funding system that could respond quickly would "solve the challenges".
For the Conservatives, shadow minister Baroness Warsi said the report was an "indictment" of the problems caused by the government's "failure to control the numbers of migrants coming into this country" and by their "inability to know where migrants are living and to fund local authorities accordingly".
The Conservatives say it shows annual limits on economic migration are needed to ensure communities and public services can cope.
A spokesperson for the Department for Communities and Local Government said the government was already taking action to manage migration to "maximise the benefits and minimise the impacts".
These included a £50m "cohesion fund" to support councils and extra funding to help manage the "transitional impacts" of migration - such as £10m as for schools experiencing increases in pupil numbers.