Mr Pickles urged politicians to stop 'looking down their noses' at voters
Tory Party chairman Eric Pickles has said the BNP is set to become a "very serious force in British politics".
He told a Westminster lunch he did not expect the party to gain MPs but the mainstream parties still had to "get their act together" to take them on.
He also said he believed there could be an October general election.
He still thought the most likely date was May next year but he said Labour could force Mr Brown out before then and go to the country early.
Either that, he argued, or Mr Brown could use Labour's autumn conference to announce a poll.
Under normal circumstances, the prime minister would have been forced to step down after disastrous European and local election results, he told the journalists' lunch.
But the results had been "a lot worse than Labour had predicted so he was able to stay on".
Mr Pickles predicted the "numbing effect" of the elections on Labour would wear off and there were still "various circumstances" in which Labour could elect a new leader and be forced to go to the country early.
He also called on politicians of all parties to stop "looking down their noses" at voters and claimed some Labour MPs were ashamed of the places they represented and had an idea of the working classes culled from the pages of George Orwell.
The BNP should be defeated on the doorstep, Mr Pickles said
He also accused Labour of using "dog whistle" tactics against the BNP - giving their activists one message about the importance of defeating the "fascist" party and giving voters another - that Labour was the party to be tough on immigration.
He said Labour should stop its "student politics" approach to fighting the BNP and accept that they had been elected.
Seeing BNP leader Nick Griffin walking the corridors of the Houses of Parliament - as he is entitled to do as an MEP - would not be a "comfortable" sight but it might make MPs on all sides wake up to the fact that "we have created the conditions for him to be here".
The Conservatives had collaborated with Labour to take on the BNP at the recent elections - by making sure they stood candidates in areas where they were expected to do well and tipping Labour off when it was felt they were gaining support.
But he said he was "not terrifically in favour of all joining hands and marching together against the BNP".
Instead he said he would like to see the BNP defeated on the doorstep and by exposing them as "lousy councillors and lousy attenders" which he said is what they are.
Mr Pickles, speaking at a Westminster Press Gallery lunch, said of the BNP: "They are going to be a very serious force in British politics and the mainstream political parties have got to get their act together and start confronting them.
"We have got to start working in those areas where they have got contact, harder than we have ever worked before.
"That's the way you are going to defeat the BNP - you have got to take the BNP on.
"They have filled that vacuum which Labour retreated from so long ago."
The Tory chairman, who is a former Bradford council leader, said there was "an enormous disconnect between politicians and the electorate" and he had grown up on estates where that disconnect typically occurred.
"It is on those Labour estates where this kind of disconnect occurs," he said, adding that Labour MPs sometimes "didn't feel part of it" but lived instead in a "posh house" on the other side of town.
"If you feel ashamed of these communities, you shouldn't represent them," he said.