Conservative leader David Cameron says the current level of immigration to the UK should be "substantially lower".
Mr Cameron said population was not a left or right-wing issue
Currently 200,000 more people move to live in the UK than leave each year.
Mr Cameron told the BBC that figure should be cut by reducing the numbers of people moving to work in the UK from outside of the European Union.
He declined to give a precise figure, but said action was needed because of the recent increase in immigration and the pressure on services and society.
Mr Cameron had earlier delivered a speech in which he called for the UK to have a population policy that would also take in an ageing population and growth in single person households.
He denied a move to right-wing issues saying it had to be considered "in the round" in calm language.
Last week the ONS (Office for National Statistics) estimated the UK population would rise by 4.4m, to 65m, by 2016 and to 71m by 2031.
In a speech to the Policy Exchange, the Conservative leader said a coherent population strategy was needed, adding that demographic changes were not just "an inevitable part of the modern world and that we'd better just get used to it".
He said he did not want to "pull up the drawbridge" on all immigration, but bring down net immigration to a "sustainable level".
"We must recognise that in an advanced, open economy there will be high levels of emigration and immigration. But what matters is the net figure, which I believe is currently too high," he said.
He said immigration had to be considered alongside other pressures including people living longer, and more people choosing to live alone.
"As a country we simply haven't planned for the effects of our changing and growing population," he said.
"Our current level of population growth and atomisation is unsustainable. Immigration is too high. Family breakdown is too high. Unsustainable demographic change makes it harder to build the opportunity society I want to see."
He said government estimates were that net immigration into Britain was between 190,000 and 200,000 a year.
He spoke out as the UK Borders Bill, which will tighten the government's grip on immigration with compulsory biometric ID cards for non-EU nationals and greater powers to immigration officers, completed its final Commons hurdles before becoming law.
Projections for the UK population put it at 71m by 2031
Mr Cameron said economic migration from outside the European Union, should be subject to annual limits. And people from new EU countries should be subject to controls on access to the labour market - as the government has done for Romania and Bulgaria.
Other pledges were to set up a border police force with powers to track down and remove illegal migrants and to raise the minimum age for spouses coming to Britain to 21, and ensure they can speak English.
Mr Cameron told the BBC that it was probably wrong to have linked immigration and asylum together in the past.
He said that part of the attempts to cut economic migrants from outside the EU would involve making it a priority to get people "off benefits and into work".
Asked whether he was abandoning the political centre ground and moving back to core Tory values, he said: "I just don't accept that actually having a proper understanding of population, demography, household formation or poverty is somehow a left wing issue or a right wing issue.
"I think if you want politicians that want to try and make positive changes in our country, improve the quality of life, strengthen our economy, deal with all the problems I've talked about , you have to address this issue.
"What matters is are you addressing it in a way that is sensible, rational, reasonable and comprehensive and I think we are."
But immigration minister Liam Byrne said Mr Cameron's plans were a "smokescreen", as 80% of immigrants to Britain last year were from the European Economic Area.
"If David Cameron wants to have a serious conversation about these issues, he should start by offering some detailed policy proposals and not merely rehash platitudes that we have heard from him before," he said.
"He talks of a limit on immigration numbers, but nowhere does he say what this would be."
Lib Dem home affairs spokesman Nick Clegg said it was "fantasy politics" and said Mr Cameron was "pandering to the right wing in his own party and claims that immigration numbers should be cut without having the faintest clue as to how that would happen".
UK Independence Party leader Nigel Farage said British Conservatives in the European Parliament had been "the greatest enthusiasts of open borders to Eastern Europe, which is where the huge rise in immigration has come from".
Habib Rahman, chief executive of the Joint Council for the Welfare of Immigrants, said Mr Cameron seemed to be "blaming the rate of migration for a host of problems from lack of housing to congested transport".
"If the Conservatives are really serious about stemming non-EU migration, David Cameron should have committed the party to specific objectives on reducing global social injustice and human rights abuses."