Mr Phillips has been head of the EHRC since it was formed
Equalities chief Trevor Phillips has told MPs he regrets a "painful" row over his leadership, but shrugged off calls to quit.
MPs are investigating the Equality and Human Rights Commission after a spate of resignations by commissioners.
He told them "arguments" and "passions" were part of the EHRC's role as it was dealing with difficult issues.
His colleague Kay Carberry said it was "demeaning" to suggest those remaining on the board were "cowed individuals".
The joint committee on human rights decided to hold an inquiry into the EHRC in July, following resignations by four commissioners and the chief executive within months.
Mr Phillips was asked about evidence given two weeks ago by EHRC commissioner Ben Summerskill - who stands down in December - and former commissioners Professors Francesca Klug and Kay Hampton.
Lib Dem MP Evan Harris asked about their references to a "culture of intimidation" and a "clique that the chair surrounded himself with".
Mr Phillips replied that in his recollection the commissioners were not "that shy in putting forward their views".
He said the EHRC had achieved a lot in its two-year lifespan, adding: "The picture of a board divided, where people could not speak and so on, really is not one that I personally recognise."
He regretted it "if people felt that way" but said: "It seems to me most members of the board never had any trouble in disagreeing with each other or indeed with me".
Mr Phillips said the row had been "painful", adding: "Of course none of this is pleasant, more importantly not because it reflects on me personally but because it distracts from the very important and much of it successful work that the commission is doing. That is what I think is actually the thing I most regret here."
"If I can now act differently to ensure that doesn't happen you can be sure that would be at the top of my personal agenda."
But when Mr Harris asked "what about resigning?" - Mr Phillips asked for an example of anything he might have done differently to answer "these general assertions".
"Unless they are concrete, it is rather hard to respond to them."
Fellow commissioner Kay Carberry - who is applying for a second term - said the evidence given two weeks ago was "was very far from my own experience".
She said the impression "of a cowed bunch of individuals who were unable to express a view and were unable to challenge the chair" was "demeaning to a group of independent and strong-minded people".
She pointed out "there are ten commissioners who didn't choose to resign" and suggested there had not been many challenges to the chairman because there was not "widespread dissatisfaction with the style of chairing".
But the Earl of Onslow, a committee member, said that was "very odd".
"How many members of a board have to resign before you recognise widespread disagreement?" he asked, adding it reminded him of a "sackful of alleycats not getting on".
Mr Phillips also denied he had "brought over" staff who had received redundancy payoffs from the old Commission for Racial Equality - something that has been criticised by the National Audit Office - saying he had nothing to do with recruitment.