Voters are worried about illegal immigrants who are "undermining" the jobs market and health and safety measures, John Reid has said.
Immigration must be effectively controlled, Mr Reid says
The home secretary said people were not opposed to incomers but were angered by a sense of "unfairness".
Immigration had to be "effectively controlled and fair" to maintain public confidence, Mr Reid told GMTV.
On Saturday, the right-wing think-tank Civitas said people had been allowed to enter the UK at a "reckless pace".
Mr Reid said: "Most people aren't opposed to foreigners; they're opposed to unfairness.
"So what they worry about is people coming here illegally, people working here illegally, undermining the minimum wage, undermining the terms and conditions, safety and all of the good terms that protect workers and consumers.
"If that is being done people can see the benefits of immigration.
"If it isn't being done, people worry about it and we have to address that worry by making sure that immigration is effectively controlled and fair to people in this country as well as those who come here."
Civitas has warned that immigration could lead to the political break-up of Britain and that it may have reached a "tipping point" beyond which it could no longer be seen as a single nation.
Its pamphlet - A Nation of Immigrants? - said the "seemingly reckless pace and scale" of immigration was bound to cause concern for people who saw the UK as a model of tolerance and freedom.
Shadow home secretary David Davis called on the government to put a cap on those coming to the UK.
The Home Office said it had already announced a tough new points system aimed at immigrants.