While the 6-3 ruling does not affect the overall legality of the death penalty in the United States, it will force the 20 states that still allow the execution of the mentally handicapped to change.
"We are not persuaded that the execution of mentally retarded criminals will measurably advance the deterrent or the retributive purpose of the death penalty," Justice John Paul Stevens wrote in his majority opinion.
The court had previously declared the execution of the mentally handicapped legal in 1989.
But since then, the number of states with the death penalty to ban capital punishment for inmates deemed mentally disabled has risen to 18 from just two.
It was not known how many inmates on death row could be considered mentally handicapped - a term generally used to refer to people with IQ scores below 70. (A score of 100 is considered average.)
But the ruling was likely to mean that lawyers for many death row inmates will request reviews of their client's cases.
The specific case before the Supreme Court was that of Virginia man Daryl Renard Atkins, who was convicted of killing a man - by shooting him eight times - after robbing him of $200 in 1996.
Mr Atkins' lawyers said he had an IQ of just 59, and had never been able to live independently or work.
The Supreme Court agreed that he should not face capital punishment because the mentally handicapped are less likely to understand the severity of their crimes, and are less morally culpable.
Death penalty opponents have also argued that the mentally disabled are more likely to confess to crimes they did not commit.
As many as 35 of the more than 750 people executed in the United States since the Supreme Court reinstated the death penalty in 1976 are believed to have shown signs of mental handicap.
'National consensus'
Opponents of the death penalty welcomed the ruling on Thursday.
"There is no doubt that there is now a national consensus on the issue," said Richard Dieter, head of the Washington-based Death Penalty Information Center.
"Concerns remain about many other aspects of the death penalty, but at least today we have stopped a practice that most Americans and the rest of the world finds abhorrent," he said in a statement.
The Supreme Court's three most conservative members - Chief Justice William Rehnquist and Justices Antonin Scalia and Clarence Thomas - dissented.