Mr Cameron stepped up the Tories' call for an apology
David Cameron wants Foreign Secretary David Miliband to apologise for comments about a Tory ally in Europe.
The Tory leader said a BBC interview showed Poland's Chief Rabbi did not believe Polish MEP Michal Kaminski was anti-Semitic.
Mr Miliband had quoted the rabbi to attack Mr Kaminski for having "an anti-Semitic neo-Nazi past".
He said that quote stood and accused the Tories of a "divorce from the mainstream" of Europe.
The row dates back to Mr Miliband's speech to the Labour Party conference in September, in which he attacked the Conservatives' new grouping in the European Parliament, headed by Mr Kaminski.
He said the Tories' new allies made him "sick" and said Mr Kaminski had been "denounced by the Chief Rabbi of Poland for an anti-semitic, neo-Nazi past".
Since then there has been a political row about what the chief rabbi had actually said and demands from the Tories that Mr Miliband apologise.
The words of the Polish chief rabbi about the attitudes of a Polish MEP have become crucial to the debate about the future of Britain, to its relationship with Europe and, perhaps, to the future of the foreign secretary
Asked to clarify his comments Poland's Chief Rabbi Michael Schudrich told the BBC he would not defend Mr Kaminski's past membership of the "openly anti-Semitic and neo-Nazi" National Revival of Poland (NOP) organisation, but he should be judged on his record as a whole.
Mr Kaminski left the NOP as a teenager and argues that it was not an extreme group when he joined it.
Rabbi Schudrich told the BBC he found it "problematic" but he preferred to look at what he had done over a longer period.
"Since that time he has become a strong ally of the state of Israel and on other occasions has condemned anti-Semitism," he said.
"So what we have here is a complicated person, and we need to be able to understand him in a fuller context, not taking one thing he said but taking a look at what he has said over the past 20 years."
He said Mr Kaminski's party was considered mainstream adding: He said: "No-one here in Poland would consider the Law and Justice Party as a fringe right party."
He said he could not "check a person's heart" but from what Mr Kaminski had said publicly and privately "I certainly see him as a man that today, today, is against anti-Semitism".
Poland's chief rabbi's comments entered a British political row
Mr Cameron said the rabbi had been "very clear": "He [the Chief Rabbi] said this man is the leader of a mainstream Polish party and is not anti-Semitic," he said.
Mr Miliband should now withdraw his "very serious accusation" and apologise.
"Bandying round accusations of anti-Semitism when you are the British foreign secretary about a mainstream party in Europe is quite wrong and David Miliband needs to recognise that as I am sure he will."
But speaking at a summit in Brussels, Mr Miliband said he had been right to "highlight exactly what has been said".
"What I said in my speech to the party conference was that Michael Kaminski had been denounced by the Chief Rabbi for past associations with a neo-Nazi group.
"That quotation is clear on the BBC and other websites and that quotation has not been retracted by the Chief Rabbi of Poland and it is a clear point that he has made," he said.
He accused the Conservatives, who left the main centre-right grouping in the European Parliament over differences on the extent of European integration, of a "divorce from the mainstream" in Europe.
Its new alliance, made up of 55 MEPs from eight member states, represented "the sort of isolation that Britain could face if the Conservatives were to come to power", Mr Miliband said.