Communities and Local Government Minister Phil Woolas has become the second minister to join the debate over Muslim women who veil their faces.
Mr Woolas said non-Muslims feelings should be considered
Wearing veils could provoke "fear and resentment" and play into the hands of far-right activists, he warned.
In an article for the Sunday Mirror, he urged Muslims to be aware of the views of non-Muslims.
His words follow Jack Straw's call for Muslim women to take off veils at his Blackburn constituency surgery.
'Fear and resentment'
"It can be hard to tell whether women wear the veil as an expression of their faith or because they are compelled to do so," Mr Woolas said.
"Most British-born Muslims who wear it, do so as an assertion of their identity and religion. This can create fear and resentment among non-Muslims and lead to discrimination.
"Muslims then become even more determined to assert their identity, and so it becomes a vicious circle where the only beneficiaries are racists like the BNP," he said.
Some Muslim women called Mr Straw's remarks insulting, but other Muslims said they understood his concerns.
Mr Straw has dismissed suggestions that his remarks are designed to raise his profile ahead of Labour's deputy leadership election.