Mr Brown appointed just four women to full cabinet posts
Senior female Labour MPs have hit out at Gordon Brown for a government style that they say excludes women.
Ex-Health Secretary Patricia Hewitt told BBC Radio 4 that there was a "laddish" culture inside No 10.
And Caroline Flint, who resigned saying Mr Brown treated women as "window dressing", said female ministers were "picked out" for hostile briefings.
But Business Minister Baroness Vadera said Mr Brown was committed to getting more female MPs into Parliament.
Former children's minister Beverley Hughes said she was "shocked" at the lack of women in cabinet.
Former minister Jane Kennedy said she agreed with suggestions that Mr Brown dealt with colleagues like a "mafia boss".
Ms Hughes also reignited speculation about a possible leadership contest by suggesting that Mr Brown had three or four months at most to improve Labour's standing in the polls.
Mr Brown has faced criticism for appointing only four women as full members of the cabinet.
Speaking to BBC Radio 4's Gordon's Women programme, several prominent female MPs criticised the lack of women in senior positions as well as the prime minister's style of conducting government business.
Ms Hewitt said she was "surprised as well as disappointed" by the lack of women in senior jobs.
She added that, with the exception of adviser Sue Nye and Baroness Vadera, "Gordon Brown's inner circle has always been small, almost entirely men and as far as I can see really rather laddish in its culture".
Ms Hewitt added that the prime minister "needs to get a move on" with fulfilling his pledge to the Parliamentary Labour Party to change his style of government.
Ms Flint - whose "window dressing" remarks in her resignation letter as Europe minister threw the spotlight on Mr Brown's relationship with women - said the cabinet "power base" was "overwhelmingly male".
She said female ministers had been offered roles which merely entitled them to attend cabinet "to give an impression that women are there and are equal" when they were not.
"When Tony (Blair) left government I think a third of his cabinet were women and I'm talking a third of cabinet - proper cabinet," Ms Flint added.
She said of the present prime minister: "For one reason or another I don't think he trusted me and we never got a chance to really get to develop our relationship.
"I don't think he really knows me and what makes me tick."
Asked about Mr Brown, former Lords leader Lady Jay admitted that "personally I find him quite intimidating" because he "doesn't make it easy to feel that you've established a personal connection".
Ms Kennedy - who resigned saying she could not support Mr Brown - said the prime minister lacked "clarity and willingness to listen to what the voters are telling us about policy", describing this as a "a behavioural trait that you keep being promised is going to change and doesn't".
She added that, as a result, "I fear that with Gordon as leader we don't have much chance" of winning the next general election.
Asked by presenter Martha Kearney whether it was true that the prime minister acted like a "mafia boss" who "might not pull the trigger but he knows who's getting bumped off", Ms Kennedy replied: "Yes, that's how I would characterise it - that's my perception of it."
However, some prominent Labour women defended the prime minister.
Discussing the number of women in front-line politics, Lady Vadera said that "Gordon worries about that - and that's why we work so hard to get women into parliament".
Angela Smith said the number of women ministers compared favourably with "any party's alternative government".
She added: "I have never, ever felt held back by the Labour party because I am a woman."
Pollster Deborah Mattinson said Mr Brown had made clear he did not know that hostile briefings had taken place and had put a stop to them, adding: "That's good enough for me."
Gordon's Women is broadcast at 1330 BST on BBC Radio 4.