Adler was famous for his musical collaborations
US-born Adler, who had been in showbiz for an astonishing 73 years, had been battling cancer and recently suffered a series of health setbacks.
He died last night at St Thomas's Hospital in London surrounded by his family.
His former manager, Jonathan Shalit, said: "Only three weeks ago we were talking about him doing a concert in China."
"He was very active until the end, that was one of the things which made him such a remarkable man."
The musician's last recording was a duet of Young at Heart with Cerys Matthews from the Welsh pop band Catatonia.
She joked at the time that she preferred older men.
He was known for his original collaborations with musicians such as George Gershwin, Kate Bush, Sting and composer Vaughan Williams, but also his own virtuoso performances.
Showbusiness friends, including Elton John, Sting and Sir George Martin, had sent messages in recent days to the legend.
Jazz trumpeter and broadcaster Humphrey Lyttleton told BBC News Online what an inspiration Adler had been.
"He made wonderful music, you just can't get away from it, the sound he got out of the harmonica was as great as Yehudi Menuhin could get from a violin," he said.
"He could express himself equally well in pop, jazz or classical music."
Self-taught
Lawrence Cecil Adler, who was Jewish, was born in Baltimore, USA on 10 February 1914.
He taught himself the harmonica and began to play professionally at the age of 14.
Adler moved to the UK in 1949 having been forced to leave the States after accusations of pro-communist sympathies during the McCarthy era.
The slurs ruined his career in the US, though he was widely respected for his refusal to accuse other musicians or acquaintances.
Besides his musical fame, Adler also appeared in a number of films - playing himself.
He was also known as a prolific letter writer, his correspondence to the satirical magazine Private Eye becoming legendary.
He also wrote an autobiography - It Ain't Necessarily So - in 1985, and worked as a food critic for Harper's & Queen.
The 1953 movie Genevieve brought him an Oscar nomination for his work on the soundtrack, although his name was originally kept off the credits because of the McCarthy blacklisting.
Adler is survived by four children, two grandchildren and two great grandchildren.