Commons leader Jack Straw says he wants to avoid a situation where "the Muslim community, or any of our communities, feel isolated and defensive".
Jack Straw said he had carefully considered his remarks
A "stronger sense of shared British identity" was needed among all ethnic groups and religions, the Blackburn MP told an inter-faith conference.
Some groups have criticised him for saying he asked Muslim women if they would remove their veils in meetings.
But Mr Straw said a "frank debate" on the state of society was needed.
In a speech at University College London, he argued that during the last 50 years, people's "sense of class" had dissolved.
This had led to an "erosion of collective sense of community", he said.
Mr Straw claimed people had come to view themselves "more in terms of their cultural, ethnic, national, gender or religious affiliations".
"Britishness" could provide "common values", such as liberty, tolerance and the rule of law, he added.
This was "not about a nation - there are Scottish, English and Welsh nations".
But people should speak a "common language", as this was essential to communication between religions and ethnic groups, Mr Straw said.
'Splinter and divide'
He told the Three Faiths Forum: "Simply breathing the same air as other members of society isn't integration.
"Britishness is thus an identity available to Anglicans, Catholics, Jews, Muslims, Sikhs, Hindus and those of other religions and none, and a central element of that identity is the principle that everyone has the freedom to practise their faith not as a matter of tolerance but of right."
He added: "There is no inevitability that our communities will splinter and divide.
"Nor is there any inevitability that our attempts to heal divides will succeed. Progress depends on our willingness to engage."
Last month, a row erupted after Mr Straw revealed in a newspaper column that he had asked a woman wearing a veil to uncover her nose and mouth during a meeting at his constituency surgery in Blackburn.
Moulana Hanif, from the Blackburn Mosques Forum, said: "Jack Straw has insulted Islam and Muslims and he has no right to do so."
But Prime Minister Tony Blair has since described veils as a "mark of separation" which make "people from outside the community feel uncomfortable".
'Not less British'
In his speech, Mr Straw said he feared some groups in society were "drifting apart", adding: "I would write the same column again."
One veiled woman in the audience said to him afterwards: "I recognise that you feel uncomfortable.
"I sympathise in a way but I don't accept that these women who visited your surgery are less integrated.
"I don't fell any less British or feel any less common value with British society."
Mr Straw replied: "We will both go away from this. If we bumped into each other in the street, you would be able to say hello to me. I would not be able to do the same. The obvious reason is that I cannot see your face. Chance conversations make society stronger."
A group of about a dozen people protested outside the UCL ahead of the minister's arrival.