Some Muslim groups have attacked Jack Straw over comments on the veil, although one of the main bodies has expressed understanding.
Debate is going on with the Muslim community
The Islamic Human Rights Commission said he was selectively discriminating. But the Muslim Council of Britain said Mr Straw's views were understandable.
One Muslim human rights campaigner praised Mr Straw's stance, but a local mosque said it was "insensitive".
Islamic Human Rights Commission chairman Massoud Shadjareh said: "It is astonishing that someone as experienced and senior as Jack Straw does not realise that the job of an elected representative is to represent the interests of the constituency, not to selectively discriminate on the basis of religion."
Mr Shadjareh compared Mr Straw's stance to asking an Orthodox Jew to remove their religious clothing.
But Dr Daud Abdullah, of the Muslim Council of Britain, said Mr Straw's views were understandable.
"This veil does cause some discomfort to non-Muslims. One can understand this.
"Even within the Muslim community the scholars have different views on this. There are those who believe it is obligatory for the Muslim woman to cover her face.
"Others say she is not obliged to cover up. It's up to the woman to make the choice.
"Our view is that if it is going to cause discomfort and that can be avoided, then it can be done. The veil over the hair is obligatory."
Muslim Labour peer Baroness Uddin said she defended Mr Straw's right to speak on the issue but said his choice to do so had been a mistake.
"He's walked into the latest onslaught on Muslim communities and by citing Muslim women into this arena without doing all the groundwork to make sure that Muslim women are a full part of society.
"The greater British public will support him and I support his right to speak but I think that Muslim women have become the symbol of oppression and of fear."
Muslim human rights campaigner Ahlam Akram said Mr Straw had "hit the nail on the head".
"I stick to the argument that in this insecure world we are living today, I would rather prefer women coming to the UK - or even living in the UK - to respect the culture as well here.
"At the moment covering the face is a threat because I don't know who is underneath that veil or underneath the whole thing."
The Lancashire Council of Mosques said Mr Straw had "misunderstood" the issue and it was "deeply concerned".
"For such a seasoned and astute politician to make such a comment that has shocked his Muslim constituents seems ill-judged and misconceived.
"We fully support the right of Muslim women to choose to follow this precept of their faith in adopting the full veil, which causes no harm to anyone."
By requesting the removal of veils which cover the face, Mr Straw was denying Muslim constituents proper access to their MP.
Mr Straw has not suggested he would not talk to Muslims who choose to continue wearing the veil through the conversation.