They are nervous, frequently tugging on their beards. Neither man wants to give his name.
Former Taleban members say there is little appetite for the fight
"It is very dangerous for us," says one. "Both here and in Pakistan."
They are former members of the Taleban, allowed to return to their homes in Khost under a low-key reconciliation initiative here, involving the US military and local Afghan officials.
There are cautious hopes such efforts could help end the Taleban insurgency that has affected much of eastern and southern Afghanistan since 2003.
The message from these two men is that it is already fading.
"The majority of the Taleban are tired," said one.
Yet still only a handful have responded to these US and Afghan offers and come in from the cold.
The US military's initiative is known as the "allegiance programme".
It is aimed at lower level figures, the "rank and file Taleban", as Colonel Gary Cheek, US commander for Eastern Afghanistan, puts it.
In return for promising to give up violence and pledging support to the government of Hamid Karzai, they are granted an amnesty.
If someone approaches US troops asking to join the scheme "we'll take some data down on him", explains Col Cheek, "and make a formal declaration".
"Then we'll send him on his way and if he's good to his word, he'll be all right."
They are given an ID card, which they can show if they should be arrested in the future by US or Afghan security forces.
Senior Taleban leaders are excluded - though exactly which ones has still not been made clear.
But American commanders have gone further with this reconciliation drive. In some cases, they have released militants from US detention.
In his area, Col Cheek has set free a man allegedly linked to several bomb attacks. The man has subsequently been appointed as a local police chief in one eastern province.
Col Cheek admits he had doubts about the move, but "as it's turned out, it's been a very positive measure - the security's much better there, the populace are very pleased with how he is doing".
More controversially, he has released another man directly implicated in an attack that killed five people last year in Paktika province, including a popular local doctor.
Col Cheek admits cases like this call into question the whole basis for detaining other suspected militants.
The US military has an estimated 500 people still in custody at its Bagram and Kandahar bases - all held without charge.
But he says these releases have only happened on the recommendation of local Afghan officials, as part of their own reconciliation efforts.
"We are working with the government and we are pretty much doing what the government would want us to do."
But things have changed, he adds: "Maybe a year or two ago this might not have happened."
Nonetheless, it is a dramatic change in strategy.
However, the military has little to show for its efforts since it announced the programme three months ago.
In eastern Afghanistan, just five militants have been signed up, according to Col Cheek.
Streams of visitors come to the al-Qaeda memorial in Khost
Nationwide, US commanders say they have had about 30 in total.
But with the number of Taleban attacks down significantly, US officers insist the hardline movement is a declining force.
The Khost provincial governor, Mirajuddin Pathan, agrees and says through his contacts he believes "there are many Taleban who want to come home".
A large number of them are from the Pakistani tribal areas just across the border from Khost.
This province is a good place to test the mood. Nestling against the Pakistan border, it was one of the Taleban's strongholds and has seen some of the worst violence of the past two years.
Al-Qaeda had one of its main bases here until 2001. Some 40 of its members killed in a US air strike are buried just outside the city at a place which has become an unofficial memorial.
When the BBC visited, there was a steady stream of locals arriving to pay their respects to people they regard as martyrs.
The grave sites - some marked simply with the words "Arab Martyr" are festooned in colourful pieces of cloth.
Many of the plots are covered with rice and seeds, placed there as offerings.
The graves bring miracles, some say.
"My daughter couldn't walk," one old man told me. "But now after bringing her here three times, she is fine again."
Yet several Taleban figures the BBC spoke to in Khost said support for continuing the battle against the Americans is waning.
One dismissed the idea put about by some militant leaders that this is a "jihad" or holy war, like the struggle against the Soviet invasion.
"It is like the difference between sky and land," he said. "What the Americans are doing here is completely different to the Russians."
Another said: "The Taleban are still organised, but they cannot overthrow the government or cause any serious trouble."
He had been living in the town of Miram Shah, in the tribal agency, or region, of North Waziristan.
However, his friend said that although other Taleban still there want to return, they are still not convinced it is safe.
"One reason so few have come back is because so many of our friends are still in Bagram and Guantanamo Bay. If more people were released, they will believe the process is real," he said.
Several former Taleban have begun serving the Afghan state
"But the majority of the Taleban are tired, if there are proper talks with the government most will give up their weapons."
But he said they are also scared of possible reprisals by hardliners in the tribal areas, including members of al-Qaeda.
And that is why many say that even if reconciliation efforts gain more momentum, there will not be peace.
In statements, Taleban leaders have vowed to keep the attacks going.
And at Camp Salerno, the main US base in Khost, troops are still preparing for more clashes.
Change of tone
But what both US commanders and Afghan officials in Khost say is really needed is for President Hamid Karzai to set out an official nationwide policy on reconciliation.
The Afghan leader has in the past said he would welcome back any Taleban not involved in serious crimes.
It is now thought only 30 senior figures, including Taleban leader Mullah Omar, would be kept out.
But months after the idea was first discussed, no-one knows for sure and that uncertainty has held reconciliation efforts back.
With the Taleban still deeply reviled in some quarters - particularly among the Shia Hazaras who suffered particularly under Taleban rule - it is a highly sensitive issue for Mr Karzai.
Colonel Cheek says he understands the difficulties for the Afghan leader.
But he says: "As an American, with the American experience with our civil war, our reconstruction was based on malice toward none and charity to all, healing the wounds of conflict."
That is a very different message to the one the Americans were sending when they first arrived here, when all Taleban were irredeemable terrorists.
Now they are hoping that this more conciliatory approach could eventually be the key to bringing the Taleban insurgency to an end.