Tony Blair has said he feared public concern over immigration would lose Labour the 2005 general election.
Mr Blair stressed the economic benefits of migration
The then Tory leader, Michael Howard, made immigration and asylum one of his five key election issues, claiming the system was "out of control".
Speaking earlier on Thursday, Mr Blair said he feared it was the issue that would "topple" New Labour.
But he claimed government action to curb illegal migrants asylum applications had eased public concern.
Tory strategists have since blamed the party's focus on immigration as one of the factors that led to Mr Howard's defeat - even though opinion polls suggest it continues to be a major public concern.
During the 2005 campaign, Mr Howard said he wanted annual quotas for both immigration and asylum and he used an election poster to suggest it was "not racist" to be concerned about the issue.
At the time, Mr Blair accused the Conservatives of raising immigration "in a profoundly unpleasant way" and of trying to stir up people's fears.
Speaking earlier, at a conference on progressive government, Mr Blair said: "I have fought long and hard to stop immigration being the issue that toppled New Labour."
"In the last election it might have done. It would have been tragic and a horrible mistake for the country.
"Migration has benefited Britain. We should be and actually are, proud of our diversity.
"But I know we could never hold to that position unless we were prepared to introduce tough measures on illegal migration, cut numbers of unfounded asylum claims and seek to deport those who threatened us."
Mr Howard's successor as Tory leader, David Cameron, has attempted to soften the party's language on immigration, stressing the potential economic benefits, while retaining a commitment to a quota system.
He has said the party would assess the impact of current record levels of immigration on local services such as schools and hospitals as well as economic factors.
"Then we can come up with an overall limit on immigration, which we suspect would be lower than current level of immigration," Mr Cameron said earlier this week.
A Tory spokesman said the party did not want to comment on Mr Blair's speech or the 2005 election, as it was "looking to the future".