There could be "very serious" problems if people's concerns on crime and immigration are not tackled before the next election, says the home secretary.
Charles Clarke stressed the need to tackle public concerns
Charles Clarke warned of the risks of voters becoming further alienated from mainstream politics.
But he said minister Margaret Hodge had overstated the British National Party threat at May's elections, although she had been right to raise the issue.
She said white working class voters were being tempted by the BNP.
The Joseph Rowntree Charitable Trust also said up to 25% of voters said they "might vote" for the far-right party.
'Wake up call'
On BBC Question Time, Mr Clarke said he did not believe there had been a rise in racist sentiment.
But there was a significant group of people who did not believe the main parties were engaging with their concerns.
Mr Clarke said the problem was a "wake up call", especially for Labour as the party in power.
"The only answer is a combination of a clearly set out policy on immigration and asylum - which I think we have published - a clear set of issues about how we deal with violent crime, a clear set of policies on anti-social behaviour," he argued.
"Unless we actually make a difference in constituencies and communities up and down the country by the time of the next election, then it could indeed become a very serious issue."
Mr Clarke said Mrs Hodge, MP in Barking, was right to point to anger about issues such as immigration and housing but the BNP threat was only "localised".
"Personally, I think she overstated the BNP issue in this election," he said. "However, I don't think it's wrong to face up to these issues."
Conservative shadow chancellor George Osborne said Mrs Hodge had vastly overstated the case.
There needed to be a good sense of proportion, he argued, urging the media to think very carefully before putting the BNP story at the top of the news agenda.
BNP had only 21 councillors out of a total or 22,000 nationwide, he added.
Liberal Democrat Treasury spokesman Vince Cable said Mrs Hodge had been "extremely stupid in giving the BNP the kind of publicity which they would have paid good money for".