The Proms attract too narrow a section of society, culture minister Margaret Hodge has suggested in a speech.
Flag waving is now the norm on the traditional Last Night
She praised "icons of a common culture" from Coronation Street to the Angel of the North and said culture could "enhance a sense of shared identity".
But the Proms was one of several major cultural events many people did not feel comfortable attending, she said.
Tory leader David Cameron said she did not "get it" and said the Proms were a "great symbol of our Britishness".
He also stressed the numbers of other Proms during the concert season - such as Proms in the Park and the Electric Proms.
'Feel at ease'
In a speech to the IPPR think tank on Britishness, Heritage and the Arts, Mrs Hodge said a "shared sense of common cultural identity" was a key part of social integration and cohesion.
She said she wanted to "challenge our sectors square on".
"The audiences for some of many of our greatest cultural events - I'm thinking particularly of the Proms - is still a long way from demonstrating that people from different backgrounds feel at ease in being part of this," she added.
"I know this is not about making every audience completely representative, but if we claim great things for our sectors in terms of their power to bring people together, then we have a right to expect they will do that wherever they can."
In her speech, Mrs Hodge praised other institutions for "creating the icons of a common culture that everybody can feel a part of" - such as the Angel of the North, the British Museum and the Eden project as well as TV and radio shows "from Coronation Street to the Archers" and shared public holidays.
But she acknowledged that culture could also be divisive - citing the examples of Jerry Springer: The Opera, which Christians said was blasphemous and Behtzi, a play which depicted sex abuse in a Sikh temple and was cancelled after protests.
"Just as culture pushes the boundaries it can make some people proud to belong, it can make others feel isolated and deeply offended," she said.
She also suggested that British citizenship ceremonies be held in historic British buildings like castles, theatres and museums to help people "associate their new citizenship with key cultural icons".
And Mrs Hodge added that talk of national mottos and shared values had to be backed up with "equal economic and educational life chances" if they were to mean anything to ordinary people.
A key part of the classical calendar for more than 100 years, the Proms aims to make music more accessible, but critics say the size of the programme makes it daunting and the traditional Last Night has been overwhelmed by jingoistic flag-waving.
A Downing Street spokesman has said Mrs Hodge's comments were not intended as an attack and that she supported the Proms as a "wonderful, democratic and a quintessentially British institution".
Mr Cameron said: "I think Margaret Hodge is wrong. I think we want more things where people come together to celebrate Britishness and more occasions when people think the Union Jack is a great symbol of our Britishness, rather than sniping at it.
"It is a classic example of a Labour politician just not getting something."