Refugees are suffering a lack of protection because of "asylum fatigue", says the United Nations.
Internal displacement: A major issue, the UNHCR warns
UN High Commissioner for Refugees Antonio Guterres warns that public confusion over asylum has led to growing intolerance and racism.
Launching a major survey of crises, Mr Guterres said global numbers of refugees had fallen to a 25-year-low.
But he said public confusion meant people who needed help were seen as a public threat.
And he warned against any moves to ditch the international principle of protecting people fleeing persecution.
In the State of the World's Refugees, the UN's refugee agency (UNHCR) says global numbers of refugees have fallen dramatically thanks to improving conditions in states including Afghanistan, Angola and Sierra Leone, among others.
But the survey stresses that despite improvements in the international picture, there are at least 33 protracted crises where more than 25,000 people have been refugees for five or more years.
Taken together, these account for more than half of the world's refugee situations.
While there were now some 9.2m refugees - the lowest figure for 25 years - internal displacement had also been growing because of civil wars and ethnic conflicts, said Mr Guterres.
Mr Guterres said the international system that protected people fleeing persecution had reached a "critical juncture".
This historic deal was under pressure from government responses to mass migration in an increasingly open, globalised world.
Public confusion over asylum and refugees and mixed sentiments over economic migration had led to growing intolerance, racism and fundamentalism, he argued.
Many people believed asylum and terrorism to be linked, said Mr Guterres, despite the fact refugees were "the first victims of terrorism".
Mr Guterres warned states not to turn their backs on the 1951 Convention, amid "asylum fatigue" and claims the system is no longer working.
"If you try to change the [refugee convention] the probability of it changing to something worse is bigger than it changing into something better," Mr Guterres told the BBC News website.
"There are opportunities for a modern interpretation of the convention, as we have seen in how the convention has been applied in the courts to protect the rights of refugees. It can be flexible for the future."
"In the past few years, asylum issues and refugee protection have become inextricably linked with the question of international migration, particularly irregular migration," he said.
"Asylum seekers are often using the same illicit channels as illegal migrants, and as a result are denied access to asylum procedures that states are obliged to provide under international law.
"While the UNHCR recognises it is the prerogative of states to control their borders, such measures should not preclude the right of those in real need to adequate procedures that states are obliged to provide under international law."
Mr Guterres said that internally displaced people, who had no international legal protection, remained the international community's "biggest failure".
A separate UN-backed report suggests that the huge growth in internally displaced people was down to government repression, the changing nature of war and the restrictions on claiming asylum over borders.
National armies or militias were playing a major role in forcing people to flee within the borders of their own country, the report said, highlighting the crisis in Darfur, people displaced in battles over drug-trafficking routes in Colombia and clashes between monarchists and Maoist rebels in Nepal.
"People who would otherwise seek safety in neighbouring states are more frequently compelled to remain within the borders of their own country, most often in similar conditions as refugees," Mr Guterres said.
"Two long-running conflicts in Africa - the Democratic Republic of Congo and Sudan - alone accounted for an estimated 7.5 million internally displaced people in 2005."