Anger with the main parties has led more people to consider voting for the British National Party, a report for a social policy research group says.
The British National Party is led by Nick Griffin
The Joseph Rowntree Charitable Trust said up to 25% of voters said they "might vote" for the far-right party.
The BNP said the report reflected voter "tension" about multi-cultural Britain.
But Home Office minister Andy Burnham said he believed support for the BNP was very localised - and in many cases represented a "protest vote".
The report echoes fears by Employment Minister Margaret Hodge that voters may be tempted by the BNP in May's local elections in England.
The authors asked focus groups about their voting views and looked at a series of opinion polls that asked people which party they might consider voting for.
It revealed "underlying support" for the BNP rather than voting intentions, said one of the authors, Professor Peter John of Manchester University.
"This is a very hypothetical question," he told BBC Radio 4's Today programme.
"It is not what party you will vote for, but who you might vote for."
Mrs Hodge said many white working-class voters in her east London constituency of Barking said they would consider voting for the party.
Prof John also said the far-right party tended to have more support in predominantly white, working class areas.
East London voters interviewed for the study said they felt "let down" by the main parties.
"They feel their voices have not been heard," he said.
Labour MP Margaret Hodge is worried at BNP gains
However, Home Office minister Andy Burnham dismissed the likelihood of the BNP becoming a stronger electoral force.
"I think the report that has been published ... reflects a growing tendency towards protest voting, particularly at local elections," he said
"But I think things have got to be kept in proportion. There's no way the BNP will get anything close to 25%."
In the 2005 general election, the party raised its total number of votes by 0.5% to gain 0.7% - or 192,850 votes.
It gained support in the 2004 European Parliament elections, increasing its votes by 3.9% to gain 4.9% of the vote, but failed to win a seat.
The BNP has courted controversy over its policies, which include a total ban on immigration, and the forced deportation of illegal immigrants from the UK.
BNP spokesman Phil Edwards said the Rowntree report reflected unease among voters about Britain's shift towards a multicultural society.
He said Britain had moved from a "racially homogeneous society ... into one where the cultures are quite alien."
"That does add quite a lot of tensions and stresses," Mr Edwards said.
"What we are trying to do is preserve the traditional culture and identity of Britain," he added.
The BNP has said it is putting up more candidates than ever before - 356 - for May's local elections.
BNP COUNCIL SEATS
First council seat gained in Millwall, south-east London, in 1993
Won three seats in Burnley, Lancashire, in 2002
By 2004 had 17 seats
Currently has 15 seats, including six in Burnley
It currently has 15 councillors across England, and said at its campaign launch on Good Friday that it aimed to add "another 15 or 20" seats.
The Conservative social justice policy spokesman, Iain Duncan Smith, said people were considering voting for the BNP because they mistakenly believed that the party would improve housing and reduce crime.
"I've been horrified and worried by the degree to which people in difficult communities no longer consider Westminster politics to be anything to do with the solutions that they need to have."
He said that was why "they start turning to others who have what maybe simplistic solutions".
Liberal Democrat President Simon Hughes said the main parties had only themselves to blame if people were turning away from them.
He said successive Tory and Labour governments had failed to provide enough affordable housing where families wished to live.
Operation Black Vote, a group which campaigns to make politics more multi-racial, agreed that voters in run-down areas felt the government had let them down.
"In these areas, deprivation and poverty exist," Simon Woolley, national co-ordinator for Operation Black Vote, told BBC Radio Five Live.
"Now that's a genuine debate to be had. ... It's nothing to do with black people."
The report, prepared by the Democratic Audit and funded by the Joseph Rowntree Charitable Trust, is due to be published next week.