Hampton, who pioneered the use of the vibraphone instrument and played with some the world's greatest jazz musicians in his band, died of heart failure at Mount Sinai Hospital.
He had suffered two strokes in 1995 and had suffered ill health in the past few years.
Fellow musicians paid tribute to Hampton, who played with jazz greats from Louis Armstrong to Charles Mingus and Dinah Washington and also performed at the White House for US presidents such as Richard Nixon and Ronald Reagan.
"Some guys become complacent and they feel like they got it made, but he never did," said Clark Terry, the trumpeter in his band.
"He was always in there with a vision to win."
Early start
Jazz historian Phil Schaap said his death left a great hole in the music world.
"With Hampton's death, we've drawn closer to losing part of the origins of the early jazz era," he said.
Former New York Mayor Rudolph Giuliani called Hampton a "true American legend" who would be sorely missed by music lovers around the world.
Hampton did not have a birth certificate but commonly dated his birthday as being April 20, 1908, when he is thought to have been born in Louisville, Kentucky.
Raised by his mother in Birmingham, Alabama and Chicago, Illinois - one of the birthplaces of jazz and the blues - Hampton began his music career banging on drums given to him by a nun while still in school.
He later switched to the vibraphone, recording with Louis Armstrong before joining the Benny Goodman band in the 1930s.
'Ambassador of music'
The Benny Goodman quartet broke new ground by performing on stage as a multi-racial quartet at a time when blacks and whites jammed or recorded, but rarely performed together.
"He shattered the walls of racism with Benny Goodman," politician Charles Rangel, who had been friends with Hampton since the 1960s, told Reuters news agency.
From the 1950s, he formed his own group and found lasting success, playing for successive Republican presidents as a long-time Republican supporter, and becoming a United Nations "ambassador of music" in 1985.
He also set up music scholarships for underprivileged students, aided the building of low income housing for the poor and elderly in Harlem, New York and developed a music school - now named after him - at the University of Idaho.
One of the Harlem projects was named after his wife, Gladys, who he was married to for 35 years before her death in 1971. They had no children.