The MP for Birkenhead told the BBC's Today programme it may have been argued when the economy was enjoying a boom, that there was a case for an "open immigration policy" - although he did not accept that.
But he said: "When we're moving into a recession, the length of which we do not yet know, the immigration policy suitable for a boom is totally unsuitable for a recession."
He said the key was to "break the link" between people coming to the UK to work and gaining citizenship, which increased the population.
Sir Andrew Green, chairman of Migrationwatch, which argues for balanced migration, told BBC Five Live Mr Woolas's remarks showed a significant development in the immigration debate.
"I think this could be a significant turning point. I think the economic crisis has shown up the weakness of uncontrolled immigration.
The immigration policy suitable for a boom is totally unsuitable for a recession
Frank Field MP
"This is the very first time that a government minister has recognised the link between immigration and population. The government have been in denial about that for years."
But the chairman of the Commons Home Affairs Select Committee, Keith Vaz, told the Times he would be "astonished" at a Labour immigration minister "in effect changing the policy".
The Labour MP added: "His predecessor and the home secretary have made it very clear they do not support a quota."
Keith Best, chief executive of the Immigration Advisory Service, told Today there was a danger Mr Woolas's comments could be misinterpreted.
From what Phil Woolas is saying this morning he appears to agree with us now
Shadow immigration minister Damian Green
"This is what's going to come across as 'We don't want migrant workers'."
An economic downturn would mean fewer people would be attracted to the UK for work anyway, he said - but some skills shortages would still exist even in recession.
Shadow immigration minister Damian Green said it appeared Labour was following Conservative policy.
"We've argued for an annual limit, the government has argued against it. From what Phil Woolas is saying this morning he appears to agree with us now - that's fine."
Chris Huhne, the Lib Dem's home affairs spokesman, argued the UK did not just need a population policy, but a regional policy as well.
"Some parts of the country still have declining or stable populations, while the South East is now reaching the limits of what can be sustained with its water resources," he said.
A Home Office spokesman said the points-based system allowed it to "raise or lower the bar" according to needs.
He added: "Our tough new Australian points system, plus our plans for newcomers to earn their citizenship, will reduce overall numbers of economic migrants coming to Britain, and the numbers awarded permanent settlement.
"Crucially, the points system means only the migrants with the skills Britain needs can come - and no more.
"Had the points-based system been in place last year, there would have been 12% fewer people coming in to work through the equivalent work permit route."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.