Culture Secretary Ben Bradshaw has said "a deep strain of homophobia still exists on the Conservative benches".
Mr Bradshaw, one of three gay men currently in the cabinet, made the comments as a new poll suggested more gay people were turning to the Tories.
Chris Bryant, another gay minister, said: "If gays vote Tory they will rue the day very soon."
But Tory frontbencher Alan Duncan said the two men's comments showed Labour was "actually the nasty party".
Being seen to be more "gay friendly" has been a key part of David Cameron's mission to decontaminate the Conservative Party brand and make it more acceptable to young, socially liberal voters.
There are currently two openly gay men on the Conservative front bench - Shadow Environment Secretary Nick Herbert and Shadow Commons Leader Mr Duncan.
In a symbolic gesture aimed at further healing relations with the gay community and burying the Tories' "nasty party" image, Mr Cameron this week apologised for Section 28 - the controversial law brought in by the Conservatives in 1988 banning local authorities from portraying homosexuality in a positive light.
Mr Cameron, the first Tory leader to speak at a gay pride event, said: "I am sorry for Section 28. We got it wrong. It was an emotional issue. I hope you can forgive us."
Labour has also been pushing its gay credentials.
This Saturday in London, Sarah Brown will become the first prime minister's wife to take part in a gay pride march.
She will be accompanied at the event by veteran gay rights campaigner Peter Tatchell.
Mr Brown himself will not be marching, for "security reasons", but he will be hosting a reception for the organisers.
But new research by Jake, a networking organisation for gay professionals, suggests a worrying trend for Labour, with 38% of Jake members who took part saying they would vote Conservative at the next election, 1% higher than the general population, according to an average of recent opinion polls.
Labour came third on 20% behind the Liberal Democrats - even though 86.6% of those surveyed said Labour was the party that had achieved most for gay people and just 4% said the Tories were "gay friendly", compared with 44% for Labour and the Lib Dems.
Jake surveyed 506 of its members anonymously on their voting intentions between 18 and 21 June and carried out a separate survey of 600 of its largely male membership on their general attitude towards the big three parties.
Foreign Office minister Chris Bryant said he was concerned that a lot of gay and lesbian voters would "bank" the reforms brought in by Labour, such as civil partnerships and a lower age of consent, and vote Conservative.
He said he did not think Mr Cameron and his frontbench team were homophobic but he claimed the Tory leader could be pushed into rolling back reforms by his own backbenchers if he won the next election.
"I think if gays vote Tory they will rue the day very soon," he added.
Culture Secretary Ben Bradshaw said he shared Mr Bryant's concerns.
"I hope that people in the lesbian, gay and transgender community will closely examine the Conservatives' record on this and David Cameron's record in particular, which is not good," he said.
He said Mr Cameron "talks the talk but he doesn't walk the walk," and speaking about Tory MPs in general, he said: "A deep strain of homophobia still exists on the Conservative benches".
But the Tories reacted angrily to the accusations of homophobia.
Alan Duncan told the Evening Standard website: "This is the last gasp of Labour's desperation. Bradshaw and Bryant are simply trying to stir up hatred and division from the last century and it's both unwarranted and unworthy.
"It's simply untrue. I believed we had reached the happy point where politics had been taken out of this altogether. But these remarks show that Labour is actually the nasty party."
Baroness Warsi, shadow minister for community cohesion, said: "On the day that the prime minister said that he did not approve of personal attacks in politics it would appear that members of the Labour Party failed to heed his message. It is a shame that Labour politicians cannot engage on the policy issues around equality and instead revert to name calling.
"David Cameron and the Conservative Party have made clear they do not believe that anybody should be disadvantaged on the grounds of gender, race, religion or sexual orientation."
Ben Summerskill, chief executive of gay rights campaign Stonewall, said the Jake research "seems to indicate there is a significant swing to the Conservative Party", adding: "It must be very reassuring for the Conservative Party".
He praised Labour's record on gay rights, but suggested they might not be rewarded for it at the ballot box.
"There is no doubt a huge number of steps forward have been taken in the last 10 years but voters look to the future rather than the past."
He said he believed there were still "homophobes" on the Tory benches but they were becoming fewer in number and he hailed Mr Cameron's apology for Section 28 as a watershed moment that would remove an obstacle to many gay people voting Tory.
"It was historic, for a politician, to be so explicit," he said.
But he said he thought it was "legitimate" for Labour politicians such as Mr Bradshaw and Mr Bryant to point to the Conservative voting record on certain issues.
There are thought to be three million gay people in the UK.