The UN says the numbers who have already returned are huge
Pakistan needs to revise a plan to repatriate 2.4 million Afghan refugees by the end of 2009, the UN refugee agency (UNHCR) has said.
The head of the UN refugee agency in Pakistan, Guenet Guebre-Christos, told the BBC that the plan was unworkable and would create instability.
Pakistan said it still hopes most of the refugees will return by the end of next year.
It says Taleban militants blamed for attacks often shelter in refugee camps.
The UNHCR has already agreed to Pakistan's plan to shut down four camps that Islamabad says pose a security threat.
Many of the refugees fled during the Soviet occupation of Afghanistan in the 1980s.
Most live in towns or cities, rather than camps.
Pakistan announced the plan to repatriate the refugees early in 2007, when relations with Afghanistan were particularly tense.
Repatriations are supposed to be voluntary, but many refugees say they do not want to return.
"Since 2001 almost five million Afghans have returned from Iran, Pakistan and other places and for a country that has been ravaged by war for the last 30 years, rebuilding is a very slow and painful process," Ms Guebre-Christos said.
"Some areas in Afghanistan are not yet very secure, so assuming that all Afghans must return immediately is a fallacy.
"Therefore we count on the government of Pakistan to continue to be a generous host, and to review the situation constantly."
The UNHCR argues that "people are not commodities" and that the plan needed to be "revised and reviewed".
It said that refugees who were repatriated would likely go back to Pakistan as illegal immigrants.
"Because of the porous border they are likely to return," Ms Guebre-Christos said, "and they will simply inflate the number of illegal migrants in Pakistan.
"On the other side, these are also the poorest of the poor, and they could join other undesirable elements, and fight back."
But Pakistan's commissioner for Afghan refugees, Imran Zeb Khan, said his country should not have to "carry the burden" of the refugees alone, and would still try to get close to its target for 2009.
"We will give it our best shot," he said.
Millions of Afghan refugees have already returned from Pakistan, Iran and other countries since the fall of the Taleban in 2001.