Mr Phillips' management style has come under scrutiny
Equalities and Human Rights Commission head Trevor Phillips is facing calls to step down after a fourth resignation over his leadership in eight days.
Ben Summerskill of gay rights group Stonewall is the latest commissioner to go. He said Mr Phillips was damaging the cause of equality by staying on.
Auditors have criticised the commission for spending almost £1m on re-employing staff after giving them redundancy.
Mr Phillips will not be resigning, the commission said.
A spokesman added that Mr Summerskill's decision to leave was "regrettable".
Mr Phillips declined to comment when contacted by the BBC.
He was appointed chairman of the Commission for Racial Equality (CRE) - his current organisation's predecessor - in 2003.
Mr Phillips continued as chairman when the organisation evolved to cover the areas of age, religion and belief, sexual orientation and the promotion of human rights in 2006.
However, his leadership style has come under intense scrutiny in recent months and the three commissioners who resigned last week criticised his management skills.
A refusal by the National Audit Office to fully sign off the commission's accounts added to the pressure, with opposition MPs accusing it of a "careless waste of taxpayers' money".
The body admitted it had not followed the rules when it re-employed seven people who had taken redundancy from the CRE - at a cost of £629,276 - on short-term contracts worth £323,708.
It prompted Mr Summerskill to say that "destructive headlines" about Mr Phillips were interfering with the commission's work and that he was damaging its cause by staying on.
He told the BBC's Today programme: "Trevor is a brilliant communicator, he's a fantastic maker of television programmes, but he has not been successful in running the commission and bringing it together.
"We should be crystal clear - this isn't an issue about policies, this isn't an issue about whether the commission should be a modern 21st century commission, it's an issue about old-fashioned management."
In a statement, the commission said it had a "clear sense of direction" as demonstrated in its three-year strategy which had been unanimously agreed by the board.
The statement continued: "It is regrettable that Ben Summerskill has chosen not to re-apply for a position as commissioner, given his experience and expertise. But we need to focus on what we're delivering for the public.
"The commission, led by Trevor Phillips, continues to work on a number of substantial fronts including delivering a new Equality Bill which will make Britain a fairer and better place to live and making our Human Rights Inquiry recommendations lead to real change in public service delivery."