William Hague: Has demanded apology from David Miliband
The Board of Deputies of British Jews has written to David Cameron "seeking assurances" about the Conservatives' links to two European politicians.
Michal Kaminski and Robert Zile are both attending the Tory conference, but have been accused by Labour of holding far right views.
The Tories said the allegations had "repeatedly been shown to be false".
On Tuesday, the head of gay rights body Stonewall pulled out of a Tory Pride event over the politicians' presence.
Chief executive Ben Summerskill said the party had made "historic" progress under Mr Cameron's leadership towards greater engagement with minorities.
"But the event tonight has been overshadowed by the presence, not just at conference but on the same platform as some senior members of the party, of people of such extreme and offensive views," he told Channel 4 News.
A Conservative spokesman said that the boycott was a "great shame" but insisted that dialogue "with other people and parties, in other countries and different communities" was the best way to fight prejudice.
And David Cameron told Sky News that he did not believe Mr Kaminski's party was homophobic, saying that such allegations were being made for "political" reasons after the Conservatives formed a new grouping in the European Parliament.
Polish MEP Mr Kaminski, who is leader of the European Conservatives and Reformists group in the European Parliament - of which the British Conservatives are members - has been accused by Foreign Secretary David Miliband of having an "anti-Semitic past".
Mr Miliband also accused Latvian Mr Zile's Fatherland and Freedom party of celebrating the Waffen SS - something which the party and the Conservatives have denied.
A spokesman for the Board of Deputies of British Jews said the letter - written by president Vivian Wineman - was "very polite" and did not accuse the two individuals of anything.
It came after comedians Stephen Fry and Eddie Izzard also wrote a letter to Mr Cameron accusing Mr Kaminski of homophobia.
"Your party's decision to host an LGBT [lesbian, gay, bisexual, transgender] event at conference is a good step in the right direction," they wrote.
"But it will seem empty - a two faced gesture - if in the same week you fawn over allies whose homophobia has no place in modern Manchester, in modern Britain, or in Europe."
Both Mr Kaminski and Mr Zile spoke alongside shadow Europe minister Mark Francois at a conference fringe event on Tuesday.
Mr Zile used his address to dismiss the allegations, suggesting they were more typical of the Soviet-era Kremlin than a Western democracy.
Mr Francois also insisted: "We would never do business with anyone who we thought glorified the SS."
Shadow foreign secretary William Hague has repeatedly denied the party has any links to extremism and has demanded Mr Miliband apologise for his remarks.
He said they were "a slur on the reputation of a country that is a good friend to Britain and a close ally in Nato and the EU" and said they could damage relations with Latvia and "other countries who suffered under totalitarian communist rule".
The Tories say Mr Miliband's allegations of anti-Semitism against Mr Kaminski are based on remarks made by Poland's chief rabbi Michael Schudrich which they say he has claimed were misrepresented.
The Tories also say the Latvian foreign minister Maris Riekstins personally called Mr Miliband about his comments.
A statement on the Latvian Foreign Ministry website says a telephone conversation did take place between the two ministers during which "David Miliband acknowledged that his statements were not directed at Latvia or its government".