One of his victims was William Foxton, a 65-year-old retired British army major, who shot himself in a Southampton park in February after discovering Madoff had cost him his life savings.
He had worked in the world's conflict zones - Oman, Bosnia, Kosovo and, Afghanistan. In 1999 he was awarded the OBE for his work in Kosovo.
His son, Willard Foxton, 28, went on a 14,000-mile journey to New York, Boston, Florida and California after his father's death.
"I wanted to take all the medals my father had won for gallantry and throw them into Bernie Madoff's face, to make Madoff know the sort of man he killed," Mr Foxton said.
"But as my feelings and my passions cooled, I realised that what I really wanted to do was to just find out more about Madoff.
"I wanted to find out about this titanic crime he'd perpetrated and I wanted to work out exactly how my father had become entangled in all of it."
Mr Foxton found some of Madoff's victims had witnessed him in action.
Two weeks after her husband died, Norma Hill headed for the Lipstick Building, one of Manhattan's iconic skyscrapers and the home of Bernard L. Madoff Investment Securities.
He put his arm over my shoulder and he looked at me and he said to me, don't worry, everything is going to be fine
Norma Hill, Madoff victim
She wanted to discuss the investments her husband had made with Madoff.
"He appeared to be a really nice, kind man, sort of like anybody's grandfather.
"He put his arm over my shoulder and he looked at me and he said to me, don't worry, everything is going to be fine."
Mrs Hill believed Madoff and kept her money with him for 21 years, but now, after losing nearly all her pension and savings, she will be forced to sell her house.
Paul Allen, 89, also invested with Madoff.
Last December, just a week before the conman was arrested, Mr Allen put in a call to Madoff's offices because he was worried about the credit crunch and was thinking about cashing in his investments. He was transferred straight to the boss.
"He got on the phone immediately and he said, 'well, why are you doing this after all these years and with the success we've had?'"
Even though Madoff knew his Ponzi scheme was imploding, "he didn't sound distressed at all", Mr Allen recalled.
"He should have been on the Shakespearean stage, he's a great actor."
Mr Allen chose to stay with Madoff, and a week later his investments were wiped out.
Madoff did not just rely on his personal charm but used his social connections, particularly within New York's Jewish community, to ensnare victims - and cash.
Madoff's assets worth more than $100m (£71m) may be seized
Madoff's con hit the Florida island of Palm Beach, a winter retreat for the East Coast's millionaires and billionaires, particularly hard.
Madoff was a member of the exclusive Palm Beach Country Club, and it is thought that a third of its members invested in his Ponzi scheme.
Martin Gruss was one of the lucky ones who did not invest but he was at the club in the aftermath of Madoff's arrest.
"There were maybe a dozen people in the dining room, hardly anybody was there, and the people that were there looked like they had been hit in the face with a frying pan," he said.
Madoff was trusted by his victims partly because of his sterling reputation on Wall Street.
Alongside his fraud, he ran a legitimate share-trading business that provided the perfect cover.
Ed Nicoll, who has run several Wall Street trading companies, has known Bernie Madoff for early 30 years.
At one stage, Madoff was chairman of the giant NASDAQ stock exchange.
Mr Nicoll believes that Madoff's establishment credentials helped his fraud stay undetected for decades - right under the noses of the regulator, the Securities and Exchange Commission.
"They were very close to Bernie and they trusted Bernie," he said.
A third of Palm Beach Country Club's members had invested with Madoff
"And that trust that Bernie managed to create with the Securities Exchange Commission blinded them."
Bob McMahon, is a computer expert who worked for a year at Madoff's legitimate share-trading division.
The company's offices were split across three floors of the Lipstick Building, and the fraud was run from the 17th floor, and it was rare for employees from the rest of the office to be allowed on to that floor.
But one day Mr McMahon was given access.
"It was as if 17 were a cave, the rooms were dark, there were piles of paper in boxes stacked nearly to the ceiling, and all of this old technology," he said.
The 17th floor offices housed green-screen terminals, dot-matrix printers and computers from the 1980s.
"Whenever I asked, 'one day it's going to die, what are you going to do then?', the response that I got was that it was Bernie's problem, Bernie will make that decision, we don't question it."
"Hindsight being 20-20, now we know it was a criminal enterprise."
The old technology was in fact being used to create the mountains of fake account statements that were sent to Madoff's Ponzi investors, and the computers were never upgraded.
Ex-employee Bob McMahon found an office filled with 80s computers
"This would have required outside consultants to come in, and in doing so the fraud would have been found out," Mr McMahon said.
Willard Foxton also tracked down a Madoff family friend, John Maccabee, who first met Madoff over fifty years ago.
"I never saw anything particularly special about Bernie," Mr Maccabee recalled.
"It was only in those years when he began making money that he came to life in a way."
Mr Maccabee believes he knows what turned an ordinary middle-class boy from the New York borough of Queens into the world's biggest conman.
"It's a man who was running from mediocrity and winding up like his parents.
"He wanted out of that. The middle of the middle of the middle of the middle class."
See Willard Foxton's full investigation on The Madoff Hustle, This World, BBC2 at 1900 BST on Sunday, 28 June.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.