Up to 200 Britons are said to be fighting for the Taleban
Mr Livingstone said that although he did not agree with them, he believed any Britons thought to be fighting with Taleban forces should not be charged with treason.
The former Labour MP said they went to Afghanistan because of the "genuine injustice" facing Palestinians in Israel and the West Bank.
But the leader of the Greater London Authority's Conservative group, Bob Neill, dismissed Mr Livingstone's argument as "rubbish".
He said: "The British law is clear. They are committing treason and they should be duly charged with that on their return."
'Martians on the moon'
Reports have put the numbers of Britons fighting for the Taleban as high as 200, although the Home Office has estimated "a handful" and Mr Livingstone about a "dozen".
In an interview with the Daily Telegraph Mr Livingstone compared them to people "who believe there are martians on the moon".
He also described the Taleban as "flipping mad religious nutters".
But the London Mayor said the West must help restore peace to the Middle East to stop the genuine concerns of some people being exploited by terrorists.
He said: "You could destroy al-Qaeda and chop up bin Laden into little bits, but unless you resolve that festering sore there'll be another terrorist along next year.
"The terrorists feed off a genuine injustice," he said.
'Terrorist cause'
Mr Neill rejected the argument: "Those British citizens who have chosen to go and fight for the terrorist cause in Afghanistan have made their bed and I'm afraid they have to lay in it."
Dr Zaki Badawi, president of London's Muslim College, said it was a "myth" that British Muslims were fighting for the Taleban.
He said: "Anyone fighting against this county is a traitor and I feel this very strongly.
"If you are a citizen of this country you have to abide by the law of this country."
However, Dr Badawi continued: "I don't think there are any Muslims from this country fighting on the side of the Taleban. This is a complete myth.
"There are lots of stories about it and every time they claim somebody is fighting then we discover them somewhere in Pakistan."