Mr Brown said the reaction to the expenses scandal had been "hurtful"
Gordon Brown has said he wants to get involved in charitable work when he leaves office, rather than go on lecture tours.
In an interview with Piers Morgan for GQ magazine Mr Brown said he was "shy by nature rather than extrovert".
Mr Brown - an X Factor fan - questioned the salary paid to BBC star Jonathan Ross, but said entrepreneurs like Simon Cowell "deserved" their money.
The full interview is due to be published on Thursday.
In it, Mr Brown insists Labour can win the next election - but when Piers Morgan asked how he would make himself "sexy", Mr Brown said: "I can't change in the way you're asking me to."
When Mr Morgan suggested the public perceived him as miserable and dour, Mr Brown said: "I accept I have to do better in the presentation area. I've got my strengths and I've got my weaknesses."
Mr Brown said his strengths were "that I make big decisions and I'm not afraid of breaking new ground".
But he said: "I could present our message a lot better. I'm actually shy by nature rather than extrovert, someone who feels that your actions should speak for themselves, but that's not the way politics works these days."
'Very little money'
On money issues, he said ITV X Factor judge Simon Cowell deserved the millions he earned because entrepreneurs had ideas, then put them into practice.
But he said there was a feeling of "unfairness" over the sums of money being paid to TV stars.
When asked: "Is it right that someone like Jonathan Ross gets £18m from the BBC?", Mr Brown replied: "The BBC has got to seriously consider its salary structure."
Earlier this year, the BBC said stars' salaries would be reduced when their contracts came up for renewal. And last week it announced it was to cut the amount it spends on the salaries of some 640 senior managers by a quarter over the next three-and-a-half years.
Mr Brown went on to say that he himself had "very little money", adding: "It's very expensive being prime minister. I gave up my prime ministerial pension that would be worth around £2 million, but on my first day in office I gave it up.
"And my salary is frozen. And I don't want our ministers to take any rise in salaries either."
Responding to allegations the expenses scandal "made you all out to be a bunch of corrupt spivs", Mr Brown said: "Yes, and that's very hurtful. I have never tried to make any money out of being a politician. The timing could not have been worse.
Gordon Brown is not alone in being an avid X Factor follower
"People were doubly angry because they were suffering financially themselves."
While the prime minister said he wanted to be involved in charitable work when he left office, he refused to condemn former prime ministers for doing speaking tours.
He told Mr Morgan he talked regularly to Tony Blair, saying their relationship was "very good", despite what people often claimed.
When asked how he wanted history to judge him, Mr Brown said: "That he stood up for fairness, and tried to ensure that people got a fair deal."
Cameron 'good politician'
On the recent controversy over Andrew Marr asking him if he was taking medication he said he was not bothered by it but added: "I've been honest about it. I really dislike the trivialisation of politics."
On his opposite number, Tory leader David Cameron, he said he was a "very good politician" but added he would never "hang out" with him.
"I have my own friends," he added.
He described Mayor of London Boris Johnson as a "subset of the entertainment business".
On the subject of entertainment, the PM talked of his preference for ITV's X Factor over the BBC's Strictly Come Dancing.
He said: "[Simon] Cowell accused me of wavering in my support for The X Factor, but I haven't. I'm an X Factor fan, and Peter Mandelson looks after Strictly Come Dancing."