The Barmy Army is threatening to boycott the rest of the Ashes because of what they claim is heavy-handed treatment during the first Test.
Bill Cooper was thrown out after playing his trumpet
The England fans group's trumpeter, Bill Cooper, was thrown out of the Gabba ground in Brisbane on Thursday for playing his instrument.
Barmy Army organiser Paul Burnham described the ejection as "surreal".
"We just want to come over and have a good time... not to be treated like school children," said Burnham.
"It seems to me that Cricket Australia just want to win, full stop.
"They believe the Barmy Army will help England win so it just seems everything is geared to try and make sure there is no fun in the game."
Burnham said many England fans were so disillusioned with their treatment at the Gabba that they were boycotting the ground and watching the game at their hotels.
He said the Barmy Army would consider cancelling a wide range of activities planned around the Ashes series if the heavy-handed treatment continued at other Test venues.
"We are waiting to see how it develops before giving advice to people coming over [from England] for the Sydney and Melbourne Tests," he said.
Cricket Australia spokesman Peter Young denied the series hosts were being killjoys.
He said they were not targeting the Barmy Army but "a small minority of idiots who have been ruining people's day out at the cricket for some time now".
"We make no apology for doing that," said Young.
"By all means come along, have a few beers and be boisterous but don't cross the line."
Young said Cricket Australia would meet Barmy Army chiefs to discuss their concerns.
He said the trumpeter was thrown out because musical instruments had been banned at the Gabba for 12 years and added Cooper was warned not to play before he entered the ground.