The UNHCR says this amounts to Conservatives whipping up fear to try to win votes.
The manifesto says that racketeers are "flooding our country with bogus asylum-seekers", and accuses Labour of being a "soft touch" on the issue.
It says that each household in Britain is paying £160 a year to cover the cost of processing asylum-seeker claims.
The UNHCR says this amounts to Conservatives whipping up fear to try to win votes.
The UN agency has also criticised Labour for its silence on the issue.
It said the government had done nothing to combat the current climate of hostility towards asylum-seekers.
Record arrivals
Arrivals of new immigrants hit a record 100,000 last year and the hijacking of an Afghan plane to Stansted Airport brought debate over asylum-seekers to a head.
The Labour Party has already introduced a series of new measures in an effort to cut the growing numbers.
People seeking refuge will have their benefits paid in the form of vouchers only usable at certain shops.
The changes also include moves to disperse asylum-seekers around the country, and fines for truck drivers caught bringing in illegal immigrants.
The government has been under increasing pressure, amid concern over the system being abused.
Tories hit back
Conservative Home Affairs spokesman David Lidington said the UNHCR had made a "foolish" intervention and he stood by the wording of the manifesto.
He said: "Most people in Britain are very willing to welcome in genuine refugees fleeing persecution, but those form only a small proportion of the tens of thousands of people who we now have in Britain making claims of political asylum.
He denied the language in the manifesto was emotive or exaggerated.
He said: "We have asylum applications running at record levels and a backlog of cases piled up at the Home Office of 103,000 at the latest count and an unknown number of people who have failed their asylum assessment, who've gone to ground in Britain. This is causing quite widespread and justifiable resentment and anger.
"What we're doing is giving expression to the genuine feelings of anger among very large numbers of British people at the fact that our immigration controls and our tradition of giving hospitality to genuine refugees is currently being abused, and that the Labour government has taken insufficient action to bring this crisis to an end."
He called for a system which allowed genuine refugees to have their cases dealt with quickly.