Paul Burrell has refused to apologise for his book despite Princes William and Harry accusing him of a "cold and overt betrayal".
Mr Burrell's book is due out on Monday
Princess Diana's former butler said his book was "nothing more than a tribute to their mother".
He said he was "extremely proud" of his book as he believed the princess would have been.
But the princes said if their mother were still alive she would be mortified by its contents, including the claim she believed there was a plot to kill her in a car crash.
Mr Burrell left a hotel in Cheshire on Saturday afternoon with his wife Maria, and their two teenage children and driven to his home without making any comment to waiting reporters.
In a statement on Friday the princes made a plea to Mr Burrell to bring these revelations to an end.
The princes would be prepared to meet the ex-royal servant, said Clarence House although there are said to be no immediate plans for a meeting.
Mr Burrell said he would be prepared to meet the princes.
But Dickie Arbiter, former press secretary to the Queen, told BBC Radio 4's Today programme such a meeting would not be a good idea as Mr Burrell would "dine out" on it for some time.
Buckingham Palace is considering its legal position on the book and has been sent a copy for inspection.
The princes' statement continued: "We cannot believe that Paul who was entrusted with so much could abuse his position in such a cold and overt betrayal.
"It is not only deeply painful for the two of us but also for everyone else affected and it would mortify our mother if she were alive today and, if we might say so, we feel we are more able to speak for our mother than Paul".
A Clarence House spokeswoman said the statement was the princes "telling Paul Burrell that they've had enough".
On Saturday a spokeswoman added that "it was now time for other people to examine their conscience".
Mr Burrell said: "When the princes and everyone else reads this book in its entirety, they will think differently."
"My only intention in writing this book was to defend the princess and stand in her corner."
"I would also like to point out that, following the collapse of my trial at
the Old Bailey last year, no one from the Royal Family contacted me or said sorry for the unnecessary ordeal myself, my wife and my sons were put through," he added.
"I have told you the truth where the British public should know the truth."
Overnight Mr Burrell also defended his actions in a television interview on ABC News.
He said he "had to try and put things right, for the princess".
Earlier this week the Daily Mirror published an extract from Mr Burrell's book relating to the claim Diana feared for her life.
In a letter allegedly written by the princess 10 months before her death in a Paris car crash she apparently told her butler, "this phase in my life is the most dangerous".
Mr Burrell's book also claims the Duke of Edinburgh wrote to Diana telling her he and the Queen "disapproved" of the Prince of Wales's affair with Camilla Parker Bowles.
The Duke is said to have written to Diana: "I cannot imagine anyone in their right mind leaving you for Camilla."