Tony Blair has told MPs the UK has no policy on controlling the size of its population.
The UK population topped 60m last year
Appearing before the Commons liaison committee he said there was a migration policy but no population policy.
And he argued it was impossible to give clear facts, rather than opinions, about the benefits or disadvantages of migration to Britain.
Labour MP Tony Wright said a debate was needed or the danger from right-wing extremists would get worse.
During his appearance, Mr Blair was also grilled about services such as schools and hospitals coming under strain in some areas because of an influx of East European migrants.
Mr Wright said the UK's population had topped 60 million for the first time last year and was expected to rise 12% in the next generation.
The rises were equivalent to having a new Oxford, a new Middlesbrough and a new Ipswich every year, he said.
And migration was the main driver, he told the prime minister.
Mr Wright urged Mr Blair to set up a commission to give a cost and benefits analysis about different levels of population, which was being driven up mainly by migrants.
A proper debate was needed, he said.
"If we don't do these things, we all know the potential for nasty right-wing extremists doing something about them is there all the time and is going to get worse," warned Mr Wright.
Asked if the government had a population policy, Mr Blair replied: "No, but we do have a migration policy obviously."
He agreed with an MP's suggestion the issue was "political dynamite".
And he said it was difficult to give objective facts on the benefits and "disbenefits" of migration.
But he argued: "Migration on the whole is positive and with benefit to countries but it needs to be controlled."
"Most people in the country are not racist and just think there ought to be some rules," added Mr Blair, "I think it is the rules that are the problem".
Biometric visas and identity cards were a key way of ensuring that people who overstayed in Britain could be deported, he said.
Mr Blair said all countries were battling with the problem of mass migration.
Twelve million people visited the UK every year for legal reasons such as trade and tourism - the challenge was to ensure they did not overstay, he argued.
The prime minister also challenged the view that migration was the main driver of population growth.
Birth rates were also important, he said. While these were higher in some communities, that did not mean the people in those communities were in Britain illegally.
The questions come after former Labour minister Frank Field told the BBC News website that politicians were living on "borrowed time" in failing to tackle immigration.
Mr Blair refused to estimate the number of illegal immigrants in the UK - something a study referred to by the government last year estimated could be up to 570,000.
He was asked about the consequences of deporting hundreds of thousands of people from London and the South East, where they were working in a range of jobs.
He replied: "What's the consequence of saying that ever if someone is an illegal migrant, you are going to allow them to stay?
"The consequence is you are going to get a lot more."
Immigration Minister Liam Byrne recently said he had not ruled out the idea of an amnesty for illegal migrants - but Mr Blair's comments suggest he opposes the idea.
Explaining the problems in tracing and deporting people who overstayed their visas, Mr Blair said: "Until there is a proper system of identifying people, it's going to be very, very difficult to do."
Councils under strain
Council chiefs in Slough last week complained they did not have the money to provide basic services because government statistics underestimated the scale of migration.
Labour MP Gwyneth Dunwoody told Mr Blair that large influxes of migrants from the new European Union member states were putting schools and housing under strain in some areas.
She asked which minister was responsible for coordinating efforts to help councils cope with the pressures.
Mr Blair said the Home Office and Department for Communities and Local Government worked closely, but other departments were called in to help with other issues. Some local councils had asked for more money and there were constant negotiations over funding levels, he said.