Conservative community cohesion spokesman Lady Warsi has described claims she is pandering to the BNP on immigration as "ludicrous".
Lady Warsi stands by her comments
The Muslim peer is under fire after telling the Independent on Sunday immigration had been out of control and was making people "uneasy".
Lady Warsi criticised the BNP's race agenda but said the party's supporters had valid concerns about immigration.
Former race minister Phil Woolas described the remarks as "shocking".
"You have to acknowledge the fears that people have that make them vote for the BNP," said the Labour MP for Oldham East and Saddleworth.
"But effectively to endorse the BNP's views is shocking."
Her comments also drew criticism from anti-racist group Operation Black Vote - for whom Lady Warsi used to work.
Spokesman Simon Woolley told the Independent on Sunday: "Pandering to the racist views peddled by the BNP and bought by the BNP voters is wrong. "The fact of the matter is that this country would collapse if it wasn't for migrant workers."
But Lady Warsi - who was nominated for a peerage earlier this year when party leader David Cameron promoted her to a seat in the shadow cabinet - told the BBC she stood by her comments.
"To suggest I am some sort of apologist for the British National Party is ludicrous.
"You only have to go on their websites and look at their leaflets. I am one of their most hated figures," said Lady Warsi.
"I was talking about people who have been duped into voting for the British National Party because they feel that some of the concerns they are wanting to talk about are not being dealt with by the mainstream political parties."
'Crime and disorder'
She said she was speaking from her own experience as a Conservative candidate in Dewsbury, West Yorkshire, at the 2005 general election, where the BNP polled more than 6,000 votes.
She said voters "had concerns about crime and disorder in their communities, they had concerns about immigration, they had concerns about community cohesion and they felt the mainstream political parties were just not listening to those concerns".
She said the BNP "had their own racist agenda" but added: "People who vote BNP have some quite legitimate concerns."
She said it was up to the Conservatives and other mainstream parties to "go out there and convince them that they are listening to their concerns".
Asked if saying immigration was "out of control" was the right language to be using, Lady Warsi said: "It is not about language. It is about having an honest debate".
She said she wanted an "honest debate" about immigration and as the daughter of an immigrant, she was not averse to "people coming into this country".
But she added: "people do have concerns when they have no idea about who is coming in and who is leaving, no clear border police, no control about numbers".
"What we do need to have is an element of control because where the government is seen to be out of control on immigration, that creates unease and that creates bad community relations," added Lady Warsi.
Mr Cameron defended Lady Warsi, insisting she had made clear that the BNP was "completely unacceptable".
"She has spent most of her political life in West Yorkshire fighting the BNP street by street, standing up for the Conservative party."
Speaking on BBC1's Andrew Marr show, he said there was no doubt that people were "concerned about immigration" but politicians had to keep the issue "in perspective".
"We have to be very cautious and sensible about the language that we use and I always am when I speak about immigration.
"But what I have said before, and I'm happy to say again, if you have very high immigration it does put huge pressure on housing and on health and on education."
Mr Cameron said the BNP was "dreadful" and he wished the party did not exist.
"I think they are despicable people. But we have to recognise that people do have concerns about immigration and I have always said that what the British public want from their politicians is calm, reasonable, moderate language."
"They want the calm language but the firm measures. They want us to make sure we are controlling immigration and that is exactly what a Conservative government would do."
The BNP campaigns for the voluntary repatriation of immigrants but chairman Nick Griffin has repeatedly insisted that neither he nor his party is racist.