A Lincolnshire councillor has been suspended from the county's Racial Equality Council (REC) over his links to the British National Party.
Mr West said he wanted to talk to all groups
The Conservative Robert West, who is on South Holland District Council, is being investigated by his party.
The REC said allegations he addressed a BNP meeting are wholly incompatible with its aim and objectives.
Mr West, who said he had no knowledge of the suspension, had previously said the public misjudged the BNP.
Vice chairman of the REC, Sidney McFarlene, said: "A motion to suspend Mr West from the council was put before the district council and was not even seconded. But if they won't do it, then we will do it ourselves.
"What is particularly concerning is that he was suspended by the local Conservative association but subsequently reinstated while still being investigated by the national party."
Mr McFarlene said an independent panel was being formed to which Mr West would be invited to put his views.
He confirmed the allegations concerned both Mr West addressing a meeting of the BNP and the publication of "opinions and quotes".
The REC has also written to South Holland District Council to request it also reviews his position.
Mr West was suspended from the local party for a short time before being reinstated.
He said: "I believe it is important we debate and engage with all sections of the political world.
"After I had a chance to explain my views a call for my suspension did not even get seconded in the council chamber.
"I do not subscribe to the idea I should have to get my every move and action passed by those at the top."
A spokesman for the Conservatives has said Mr West's future with the party is to be considered by its national board.