Described by friends as neat, attentive and loyal, he remained in the family's employ despite his arrest for theft in November 2000.
Christopher Warwick, Princess Margaret's authorised biographer, described him as the "consummate butler".
"He is... everything a butler should be: professional, discreet, dependable. He was always unobtrusively there, and was such a help to Princess Margaret over the years.
"His loyalty to the Royal Family is without question."
Mr Brown was born in Australia in 1952 and, having expressed a childhood ambition to work in royal service, moved to Britain after leaving school.
He took casual work before joining the Royal Household as a page to the Queen.
He later worked for the Prince of Wales - before Charles married Diana - and went on to serve the royal couple after they wed.
In 1986, he was seen by millions of TV viewers in a fly-on-the-wall documentary about the couple, preparing breakfast for them during a tour of Australia.
Replaced by Burrell
When the pair separated, Diana took Mr Brown with her. But gradually she handed over more of his duties to Paul Burrell.
In 1995, when the princess scaled back her public life, Mr Burrell fully replaced Mr Brown.
But Diana's Kensington Palace neighbour, Princess Margaret, came to Mr Brown's rescue, appointing him to her own household.
He became her most senior domestic servant, in charge of a small team of below-stairs staff.
He continued to be held in such high esteem by other royals that the Prince of Wales invited him to Prince William's confirmation at St George's Chapel, Windsor, in March 1997.
'Kindly'
Mr Brown and Mr Burrell also remained friends. When Mr Burrell received the Royal Victorian Medal from the Queen, two months after Diana's death in 1997, Mr Brown was invited to the celebratory lunch.
Mr Brown's arrest on suspicion of stealing several valuable items from Diana's estate shocked the Royal Household.
He was suspended pending the outcome of the police inquiry.
But Princess Margaret liked her servant of seven years so much that she wanted him to continue working for her.
She said she missed his "kindly" touches, and palace aides eventually agreed to allow him to return to work.
Taken in
When Margaret died in February this year, Mr Brown was among 10 members of staff who faced being made redundant.
However, her son Viscount Linley retained his mother's favourite servant.
Mr Brown helped Lord Linley take an inventory of the possessions Margaret had left at her apartment, helping experts from Sotheby's catalogue antiques, furniture, paintings, china and porcelain.
Last month, a casually-dressed Mr Brown was spotted helping the Linley family move out of their palace apartment into a town house several miles away.
Mr Brown lives in Tunbridge Wells, Kent, and his hobbies include collecting photographs of Queen Victoria and her descendants, and designing table lamps.