Prince Philip's bizarre question was posed to Aboriginal cultural park owner William Brim during a Royal visit to Cairns in Queensland.
His son Willy Brim, who was next to his father when the incident happened, said the Duke had showed a lack of understanding of the Aboriginal culture.
But he added that no one had been offended by the comment and they were honoured that the Queen and the Duke had visited the park.
'Cackle'
Mr Brim told BBC Radio 4's Today programme that Prince Philip and his father had "a little cackle" after the question was asked.
"Personally, I thought it was a bit naive and showed a lack of understanding of the culture here," he said.
Asked about the media coverage of the incident, he said the media has built it up into something it was not.
"The visit of the Queen and the Duke was a very special occasion for us," he added.
Light-hearted
Buckingham Palace, which is said to be angry that a light-hearted statement has been taken out of context, has a different version of events.
The BBC's royal correspondent Nicholas Witchell said it claims the prince asked two men if the tribes had fought each other in the past.
When they replied yes, he asked "Do they still fight each other?
This was greeted with laughter.
Prince Philip is accompanying the Queen on a Royal tour of Australia to help mark her Golden Jubilee year.