Downing Street has insisted Gordon Brown was just "doing what he was told" after apparently getting lost at the state banquet for Nicolas Sarkozy.
The prime minister and French President were among 150 guests at the banquet in St George's Hall at Windsor Castle.
In television footage, the Queen can be heard saying to Princess Anne: "The prime minister got lost. He disappeared the wrong way...at the crucial moment."
No 10 said the PM "does what he's told on these state occasions".
A spokesman added: "I don't think this is something that has been troubling him too much."
President Sarkozy sat between the Queen and the Duchess of Cornwall, while his wife Carla sat on the other side of the table.
Mr Brown was due to sit opposite as well, but, judging by the Queen's comments picked up by television microphones, must have taken his seat late.
A Buckingham Palace spokesman said he would not comment on "private conversations".
But he added: "The minor mix-up with the prime minister was not significant and did not impact on the occasion."
Conservative leader David Cameron and Liberal Democrat leader Nick Clegg were also at the banquet, where guests enjoyed a four course meal washed down with vintage wine and champagne.
Madame Sarkozy sat with the Duke of Edinburgh and The Prince of Wales, along with other Royals including the Earl of Wessex and Princess Michael of Kent.