Tony Blair has ordered a "top-to-bottom" review of the immigration system just days before the biggest enlargement in the EU's history.
The UK will not house migrants who are not working
He told business leaders that new restrictions on the right to welfare benefits would help tackle abuses, especially after EU expansion.
But Tory leader Michael Howard accused the PM of acting in a "blind panic".
"You could not have a better example of how the government has let us down than on immigration," he said.
In his speech to the CBI conference in London, Mr Blair insisted that migrants from eastern Europe who do not work will not have access to council houses and will be prevented from "benefit-shopping."
With 10 new states joining the EU on 1 May - increasing its numbers to 25 - Mr Blair said the UK would be neither "fortress" nor "open door".
He told business leaders: "Migrants will not be able to access social housing unless they are here legally and are working.
"No-one will be able to come to the UK from anywhere in the enlarged European Union simply to claim benefits or housing.
"There won't be support for the economically inactive. The same goes for migrants from elsewhere in the world... they must be self-sufficient."
People who argued migration is out of control, or that the UK is taking more people than other countries were "simply wrong", he said.
Migrants made a "huge contribution" to the British economy and public services would be "close to collapse" without them, he said.
Restrictions will apply to all countries in the European Community Association Agreement, which includes the 10 new EU states such as Lithuania and the Czech Republic but also other hopeful nations like Romania and Bulgaria.
NEW EU NATIONS:
Mr Blair said quotas for low-skilled workers getting work permits to enter the agricultural, hospitality or food processing sectors would be reduced.
Currently there are 25,000 places on the Seasonal Agricultural Workers Scheme, with many of those filling them coming from countries about to join the EU.
Mr Blair said concerns over migration could not just be dismissed as racism, and that a "top-to-bottom" look at the system was necessary.
'Abuse and absurdity'
"There are real, not imagined abuses of the system, that lead to a genuine sense of unfairness.
"They can accept migration that is controlled and selective.... but they will not accept abuse and absurdity.
Mr Blair has paid particular attention to immigration in recent weeks
"All who come to work and study must be able to support themselves."
Mr Blair said immigration had "suddenly become very high on the agenda" of local communities and he urged sensitivity in the handling of the issue.
"We need few reminders about what happens when the politics of immigration gets out of hand," he told the conference.
'No soft touch'
However, shadow home secretary David Davis accused Mr Blair of a "panicky response to a problem which has been long in creation".
"This is a desperate attempt to try to recover the initiative on an area of policy which has been a complete shambles from beginning to end," he told BBC Radio 4's the World At One.
But Work and Pensions Secretary Andrew Smith rejected the charge, saying the prime Minister was using a "commonsense approach which is good for Britain".
"It will boost the economy whilst getting across the message loud and clear that we are not a soft touch for people who want to fiddle the benefits system. They won't be able to."
Earlier, European commissioner and former Labour leader Neil Kinnock claimed there would be no influx of European migrants - but argued some restrictions were justified.