Gail Coleshill, Dan Norris and Jacob Rees-Mogg are standing for North East Somerset
Labour's candidate for Somerset North East has said his Tory opponent's aristocratic upbringing meant he could not relate to ordinary people.
Dan Norris denied making class an issue but suggested Jacob Rees-Mogg would not be able to understand regular people.
Mr Rees-Mogg said it was trivial to discuss his background and that it was more important to discuss issues.
Liberal Democrat candidate Gail Coleshill said she did not understand what the fuss was about.
The comments were made at a general election debate on BBC Somerset held at the Royal British Legion in Keynsham.
Mr Rees-Mogg attended Eton and Oxford University, was an investment banker and his father is former Times editor Baron William Rees-Mogg.
Mr Norris said: "It's not Mr Rees-Mogg's class that is the challenge here, it's his background - how well does he understand ordinary, everyday people?
"He says he has an accent that is typical of Somerset - I think anybody listening to this broadcast will hear that that is not the case."
Mr Norris raised the issue of class when discussing previous claims that Mr Rees-Mogg had called people who had not attended university at Oxford or Cambridge "potted plants".
Mr Norris was referring to an appearance on Newsnight in 2006 when Mr Rees-Mogg was discussing the number of Tory candidates who had attended 'Oxbridge'.
At the time, he said: "The Tory party, when it's elected, has to be able to form a government and it's not going to be able to form a government if it has potted plants as candidates simply to make up quotas."
Mr Norris said he only brought up the accusation because "if he [Mr Rees-Mogg] says things like that, he is setting that agenda; he is raising those issues."
Mr Rees-Mogg said: "I think these personal criticisms are mistakes and I don't think it's the way people want politics to be run when the country is facing major problems.
"The risks in politics of being taken out of context is a very great one... just because you mention it, it doesn't mean you are calling somebody a potted plant."
Liberal Democrat candidate Gail Coleshill said: "I don't understand why such a fuss is being made.
"There must be people in Somerset North East who are rich, and have aristocratic backgrounds - we're a grand mix of people in North East Somerset and I hope that we would all represent them all."
The candidates announced so far in Somerset North East are: Conservative: Jacob Rees-Mogg; Labour: Dan Norris; Lib Dem: Gail Coleshill; UK Independence Party: Peter Sandell; Greens: Michael Jay.