Some Jack Straw's constituents believe he "should know better"
The good news for Jack Straw is that even his fiercest critics in Blackburn are still calling him "Jack".
It is a measure of his stature in the town that most of his constituents - including some of those who despise him - are on first-name terms with him.
"Jack should know better," was a repeated response as I waved my microphone around shoppers in the city centre.
Just as many - if not more - said: "I back Jack."
For almost three decades, he has been the local MP. One in four voters here is a Muslim.
Jack Straw asking me to take off my veil before he talks to me is like me asking him to take off his shirt
Even though Mr Straw was Foreign Secretary at the height of the Iraq war, the expected Muslim backlash at the last election never materialised.
His majority went down by 1,000 votes, but in the context of widespread Labour losses it was hardly significant.
The difference with the latest controversy is that it is much more personal.
One Muslim woman told me: "Jack Straw asking me to take off my veil before he talks to me is like me asking him to take off his shirt.
"Where do you draw the line?"
Some see a worrying trend developing.
A Muslim cleric - who did not want to be identified - told me: "Muslim-bashing is becoming a national sport. I didn't expect Jack Straw to jump on the bandwagon.
"His comments were more like those from a BNP member than a Cabinet Minister."
The Koran tells men and women to dress modestly
According to one young Muslim woman, wearing the full veil is a sign of religious devotion - rather than female oppression.
"My interpretation of Islam is that I should be wearing the scarf in this way. And I see it as an obligation," said Ruqayyah, a member of the Muslim Public Affairs Committee.
The fact that so many people were prepared to speak out shows how raw a nerve Jack Straw has touched.
Little did he know that a short article in his local paper, the Lancashire Telegraph, would be leading the national news bulletins for two days.
He will be pleased to hear that the paper is backing him - an indication, perhaps, of the mood of the majority of readers.
For any MP in a multi-cultural constituency, it is a tricky job balancing the interests of all of the constituents.
As Mr Straw now knows, it is impossible to be a "Jack of all trades".