John Profumo will forever be linked to the 1960s scandal that rocked the establishment, but he has been praised for his "inspirational" charity work.
After his resignation, Mr Profumo spent 40 years helping charity
His friend Lord Deedes said he had "atoned" for his mistakes with 40 years of work helping east London residents.
Prime Minister Tony Blair said Mr Profumo "underwent a journey of redemption" and had helped many people.
Ex-Tory leader Iain Duncan Smith said he would remember Mr Profumo, who has died at 91, for his "remarkable work".
Within days of his forced resignation from the Cabinet in 1963, Mr Profumo had walked into Toynbee Hall, an east London charity, and asked to help with the washing up.
That visit was the beginning of an association which lasted more than 40 years, until his death.
"Many will automatically remember his spectacular fall from grace in the Christine Keeler affair. But I will remember his remarkable work after that in the East End of London," Mr Duncan Smith said.
"For years after he stood down as a frontline politician, he dedicated his life to helping the many devastated people in London.
"His enormous efforts will have changed the lives of many people over the years and he will be sadly missed by them and many of us who knew the other Jack Profumo."
Mr Blair said: "He was a politician with a glittering career who made a serious mistake but who underwent a journey of redemption and who gave support and help to many, many people.
"He should be remembered with a lot of gratitude and respect for what he achieved in his later life."
He added: "He and his family showed a lot of dignity and honour."
Lord Deedes told the BBC: "The fact is what he did, and continued to do until quite recently, was a very long stint of social work for the poor of east London.
"And if that isn't considered to be sufficient atonement for the mistake he made, then there's no such thing as forgiveness."
John Profumo helped the charity Toynbee Hall for more than 40 years
A spokeswoman for Toynbee Hall said Mr Profumo had been its longest serving volunteer.
Mr Profumo - known as Jack to his friends - initially undertook fairly menial tasks, but went on to be the charity's chairman for 10 years and later its president.
He became a "model fundraiser" relying on his existing links and natural charm but also creating new connections, the charity said.
Toynbee Hall credits him with turning round the "poor financial situation" it faced around 40 years ago.
Chief executive Luke Geoghegan said Mr Profumo was an "inspiration to us all".
"His tireless commitment to the organisation's development, and particularly fundraising, continued to the end," he said.
"More recently, he oversaw the revival of the residential volunteering scheme, the expansion of our legal advice service and the establishment of a new advice service for the financially excluded.
"The social policy lunches that were set up under him attracted a wide range of people at the very top of their professions to come and speak.
"He took an active interest in the work of all, providing services for the local community, and helped to create a very special form of friendship amongst all the people here.
"All who came into contact with Jack will have a very special memory of him."
'Sense of fun'
John Parkin was Mr Profumo's private secretary when he became a junior minister at the War Office.
He told the BBC News website he had fond memories of an "extremely pleasant, good fun person" whom he had liked very much.
He said: "Jack had a sense of fun about him - he wasn't very tactful, he didn't need to be. He was very wealthy and quite incautious, he was often flippant at the wrong time."
Mr Parkin believes that although Mr Profumo made a "terrible mistake" in his political life, the scandal "was blown out of all proportion" and was "not as serious as it was made to look".
Political historian Anthony Howard described Mr Profumo as a "man of charm" but a "lightweight politician".
Mr Howard said that "technically" the Profumo affair did not bring down Macmillan because he retired through ill-health.
But he added: "It did do tremendous damage - the country went hysterical.
"But then for 40 years or more Jack Profumo, first of all with his wife, devoted himself to social work in the East End."
He added Mr Profumo would never talk about the scandal.