The UK Independence Party has written to David Cameron demanding he apologise for calling its members "fruitcakes, loonies and closet racists mostly".
In his letter, UKIP leader Roger Knapman insisted it was "most definitely not a racist party".
It was sent as UKIP officials spent the day consulting lawyers with a view to taking libel action against Mr Cameron.
Tory MP Bob Spink came out in support of UKIP saying the party was "right to ask for an apology".
The war of words began when Mr Cameron spoke out against UKIP during a phone-in to London radio station LBC.
He defended his "closet racists" suggestion by saying it was the result of some of the things he had heard its members say.
UKIP MEP Nigel Farage, speaking to reporters at an event in Strasbourg where fruitcake was served, said: "We have been called all sorts of things by the Conservative Party .... and we don't mind any of that.
"But we will not accept being called racist by Mr Cameron or anybody else.
"We simply can't walk away from this. Allowing that racist slur would be political death for us.
"We will do everything we have to through the law to make him apologise."
'Error of judgement'
In his letter, Mr Knapman told Mr Cameron he "was disturbed to hear your comments yesterday labelling members of UKIP as 'closet racists'.
"I am sure that on reflection you rather regret the comments you made and were you to issue an apology to our MEPs, staff, members and voters, I would be more than happy to let the matter rest there.
"All of us have made serious errors of judgement when we lack the political and/or life experience to know better."
As the row continued, Mr Spink, Tory MP for Castle Point, issued a statement saying "UKIP supporters are no more racist than Tories were in the last election".
He said he regretted the "personalised attack on UKIP", a party he respected and whose members were trying to protect Britain "from constant attacks from the unelected, unaccountable and profligate EU Commission".
"Nigel Farage was right to ask for an apology - decent Tories will freely give it. People are sick of party political bickering," he said.
Earlier chairman David Campbell Bannerman said UKIP had been advised it had a good case for slander against the Tory leader and UKIP founder Alan Sked.
Dr Sked said since leaving the party UKIP had moved "to the far right", with immigration its main topic.
He told BBC Radio 4's Today programme he agreed with Mr Cameron's assessment of UKIP, adding: "There is a trend of this party to the far right. After I left it became much, much more in evidence and it is in evidence today."
But Mr Campbell Bannerman accused Dr Sked of being "a bitter man" who had been responsible for introducing a British National Party activist into UKIP.
He said the party was "not obsessed with immigration", or racist, and had candidates from ethnic minorities.
He said the Tory leader had been "rattled" by UKIP plans to use the Freedom of Information Act to uncover the names of the Conservatives' secret lenders.