As he went to get into his car, Mr Brown told her: "Very nice to meet you, very nice to meet you."
But off camera, and not realising he still had a Sky News microphone pinned to his shirt, he was heard to tell an aide: "That was a disaster - they should never have put me with that woman. Whose idea was that? It's just ridiculous..."
Gillian Duffy: "I want to know why I was called a bigot"
Asked what she had said, he is heard to reply: "Ugh everything! She's just a sort of bigoted woman that said she used to be Labour. I mean it's just ridiculous. I don't know why Sue brought her up towards me."
Mrs Duffy said after hearing of Mr Brown's comments: "I'm very upset. He's an educated person. Why has he come out with words like that?
"He's supposed to be leading the country and he's calling an ordinary woman who's come up and asked questions that most people would ask him... It's going to be tax, tax, tax for another 20 years to get out of this national debt, and he's calling me a bigot."
Mrs Duffy, a widow who has a daughter and two grandchildren, said she used to work with disabled children for Rochdale council before she retired.
She had earlier told reporters she was a lifelong Labour voter and described Mr Brown as being "very nice".
BBC political editor Nick Robinson said it was a disaster for the prime minister because it showed the gap between his public face and private face.
"For those of us who have known Gordon Brown for many years, what we have seen is no huge surprise. He has got better and better at handling himself in public, but quite often he flares up in private, expresses frustration," he said.
Nick Robinson added that the irony was that if his comments had not been picked up, it would have been a lively election exchange which would have been seen to do him credit.
Brown: 'I apologise if I've said anything that has been hurtful'
Speaking on Radio 2's Jeremy Vine show, Mr Brown said: "Of course I apologise if I've said anything that's been offensive and I would never put myself in a position where I would want to say anything like that about a woman I'd met.
"I blame myself for what is done, but you've got to remember that this was me being helpful to the broadcasters, with my microphone on, rushing into the car because I had to get to another appointment and they have chosen to play my private conversation. These things can happen, I apologise profusely to the lady concerned."
Mr Brown later telephoned Mrs Duffy to personally apologise for the comments, telling her he was very sorry and said she "is a good woman".
When asked did this in any way make up for the comments she said "no - absolutely not".
A spokesman for the prime minister said: "Mr Brown has apologised to Mrs Duffy personally by phone. He does not think that she is bigoted. He was letting off steam in the car after a difficult conversation.
"But this is exactly the sort of conversation that is important in an election campaign and which he will continue to have with voters."
The Conservatives said Mr Brown's comments spoke for themselves.
Shadow chancellor George Osborne said: "That's the thing about general elections, they do reveal the truth about people."
Liberal Democrat leader Nick Clegg said: "You should always try to answer the questions as best you can. He has been recorded saying what he has said and will have to answer for that."
Business Secretary Lord Mandelson said Mr Brown was "mortified" over the hurt he had caused Mrs Duffy.
He told the BBC: "It is very unfortunate that this remark that he didn't believe got picked up. But he should be judged on how he responded. And that's why he apologised immediately."
Chancellor Alistair Darling said Mr Brown's apology was profuse and he was well aware he should not have made the comments.
"The election campaign will be decided not just on individuals but what the party stands for. There are big issues at stake. Gordon is a man of considerable strength, considerable resilience and considerable substance," he said.
"I hope people will judge him in the round. The fact we are coming out of this recession is down to him in no small part."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.