The BBC's David Loyn in Islamabad said that in one camp, there was a blockade of displaced people protesting that they had not received the identification cards they needed to secure food and cash to help them rebuild their livelihoods.
Each family has been promised 25,000 rupees ($306), but most people have not yet received the money because of administrative difficulties.
Some 20 trucks and buses transported 108 families from Jalozai camp back to their homes.
"I am happy we are going to our home," Sher Zaman, 60, told the Associated Press. "The days were rough, the weather extremely hot for us, but we thank the government who served us well. We want peace in our area."
But some people said they were wary of returning home until the security situation was clear.
"The security situation is still bad over there. If we go there and something happens again, what will be our options?" asked Meraj Uddin.
Government officials expect people to move in larger numbers in the coming days, especially as the word goes out that the trip is safe, our correspondent says.
The first families have begun their journey home
On Tuesday, 800 families are due to be sent back to Swat, officials say.
Felipe Camargo, of the UN High Commission for Refugees, said some areas were now "considered clear and safe" for return - but said it was vital that displaced people were informed about the situation there.
"We have signed an agreement with the government... of North West Frontier Province to ensure that the willingness of the voluntary return is maintained and that people are well informed about what the conditions are in the areas of return," he told the BBC.
He said the US and World Food Programme would provide food aid to the returning families.
The first batch of returnees are from the Landakai-Barikot sector of the main road leading into Swat's main town of Mingora. This was one of the districts worst affected by fighting between the military and the Taliban.
Unlike these shops, most buildings in Mingora are largely unscathed
Reports from that district say there has been no fighting for nearly three weeks despite frequent curfews and house searches by the army.
The return is being overseen by the substantial military presence established in the Swat, Malakand and Buner regions after Taliban militants were dislodged.
Once people have been moved from the camps, the army will begin returning people who have been living in schools and other places since they fled the fighting.
Much of the infrastructure in the Swat region was severely damaged in the months of fighting.
Power and water supplies have been shattered and the reconstruction is expected to take many months.
The BBC's Syed Shoaib Hasan, who recently visited Mingora, said the town was largely intact, with markets and residential areas still standing.
But the security situation remains uncertain and supplies are critically low, he says.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.