A charity has criticised George Osborne after he appeared to suggest Gordon Brown could be "faintly autistic".
Mr Osborne laughs during an interview before controversy arose
The shadow chancellor, who denied the charge, was criticised by the National Autistic Society after a light hearted exchange at a Tory fringe meeting.
Mr Osborne had been recalling his ability to retain odd facts, when the journalist hosting the event joked he might have been "faintly autistic".
In reply, Mr Osborne said: "We're not getting on to Gordon Brown yet."
The National Autistic Society said using the term to mock a person could cause "deep distress".
Carol Evans, a director of the charity, said: "Any pejorative use of terms relating to autism can cause deep distress and hurt to people affected by the condition.
"We as a charity are keen to raise awareness in order that these terms are not used lightly by commentators.
"To use such terms as a criticism of someone's social skills only perpetuates the confusion that surrounds the condition."
According to the charity, some 535,000 people across the UK are affected by autism.
On Monday, Mr Osborne rejected, on Sky News, claims that he had suggested Mr Brown was autistic.
The Tatton MP said: "I didn't say that actually. I was asked if I was slightly autistic."
His response referring to Mr Brown had been "merely" moving the conversation on, he said.
Pressed to say if he thought the chancellor was slightly autistic, Mr Osborne said: "No absolutely not. Autism is a very serious condition."
Mr Brown has been chancellor since 1997
He added: "Do I think that Chancellor Gordon Brown needs better relations with his colleagues, let alone with the opposition front bench? Most certainly."
A source close to Gordon Brown said the comment was "grossly offensive" and urged Mr Osborne to apologise.
The source said: "This isn't offensive to Gordon Brown but it is grossly offensive to the thousands of people affected by autism and their families that their condition should be used by George Osborne as a term of political abuse and he should apologise."
For the Lib Dems Norman Lamb MP said: "This may have been a casual off-the-cuff remark from George Osborne, but that doesn't stop it being offensive.
"Opposition politicians have reason enough to criticise the Chancellor without having to descend to these ill-informed comments.
"This won't do much to help the attempts to portray the Conservatives as a caring, sharing, modern party."
Writer Nick Hornby, whose 13-year-old son Danny has autism, and who helped co-found TreeHouse, an educational charity for children with autism, said: "George Osborne doesn't seem to have noticed that most people over the age of eight no longer use serious and distressing disabilities as a way of taunting people."