Several people in Gordon Brown's office have contacted an anti-bullying charity, its boss has told the BBC.
Christine Pratt said "three or four" calls had been made to the National Bullying Helpline in recent years.
Earlier Lord Mandelson denied claims Mr Brown had been warned about his temper, saying he "doesn't bully people".
Downing Street said it had not been contacted by the helpline, and had "rigorous", well established procedures in place for staff to address concerns.
It follows a separate story in the Observer alleging that the head of the civil service Sir Gus O'Donnell was so concerned he had a word with Gordon Brown about his behaviour towards staff.
A book by the newspaper's chief political commentator Andrew Rawnsley includes details of incidents where it is alleged Mr Brown grabbed staff by the lapels, shoved them aside and shouted at them.
Downing Street says the reports are "malicious allegations" that are "without foundation".
The Cabinet Office said: "It is categorically not the case that the cabinet secretary asked for an investigation of the prime minister's treatment of Number 10 staff."
Brown is not a bully - Mandelson
But Mr Rawnsley said he had claimed only that Sir Gus had made "his own investigations" and had given the PM a verbal warning. He told the BBC his source for the story was "24 carat".
Business Secretary Lord Mandelson, one of Mr Brown's close allies, told BBC One's Andrew Marr Show the story painted a picture of man who "gets angry but chiefly with himself, who doesn't bully people".
He added that Mr Brown "does not like taking no for an answer" and there was "a degree of impatience" about him, but added: "I don't think he so much bullies people as he is very demanding of people."
But Ms Pratt, chief executive of the National Bullying Helpline, said that was sending out "the wrong message" to other chief executives or bosses who are accused of bullying: "Outright denial is just non-credible in today's age".
She told the BBC: "Over recent months we have had several inquiries from staff within Gordon Brown's office.
"Some have downloaded information; some have actually called our helpline directly and I have spoken to staff in his office."
I think the country wants someone that will push things forward, and not allow things to be stagnant and stale
She said "over the last three or four years" the helpline had "probably received three or four calls".
She added she would expect any employer in this situation not to "go into denial, but to look into it, to follow due process".
Outright denial could "compound the stress of those who believe they are being bullied", she said.
"We are not suggesting that Gordon Brown is a bully, what we are saying is staff in his office working directly with him have issues, and have concerns, and have contacted our helpline."
Ms Pratt said she contacted the BBC because she was angered by government denials.
BBC political correspondent Norman Smith said it was understood one of the individuals involved had to take time off because of stress.
Tory MP Ann Widdecombe, who is a patron of the helpline, said it would be "quite a good idea" for Sir Gus to take "an informal look" at workplace conditions at Number 10.
But she also criticised Ms Pratt's decision to go public, saying the helpline was supposed to be confidential.
NICK ROBINSON'S NEWSLOG
The woman who told the BBC her National Bullying Helpline was called by three people who worked with the prime minister is now at the centre of a political storm
Ms Pratt said the organisation would never comment on the merit of a case, and added that it was non-political.
Mr Brown was also supported on Sunday by cabinet colleagues Harriet Harman and Alan Johnson, who said Mr Rawnsley had a book to sell. Mr Johnson said in 17 years he had "never" heard Mr Brown raise his voice.
On Saturday the prime minister told Channel 4 News things were sometimes said "in the heat of the moment" and when he was angry he might "throw the newspapers on the floor or something like that".
But he went on: "Of course you do get angry, mostly with yourself. But I'm very strong-willed, I'm very determined.
"I think the country wants someone that will push things forward, and not allow things to be stagnant and stale, and every morning I get up with a determination to do my best for this country."
A Downing Street spokesperson said: "At no time has the National Bullying Helpline contacted Number 10 about these allegations.
"We have rigorous, well established procedures in place to allow any member of staff to address any concerns over inappropriate treatment or behaviour.
"The civil service will continue to have a no tolerance policy on bullying."
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.