Brecon residents are objecting to a concert by Welsh rock group, the Super Furry Animals, at the Brecon Jazz Festival next month.
Gruff Rhys will appear with the rest of the band on 12 August
A petition has been signed by 84 people living near the field, where the band will perform in a marquee on 12 August.
Residents claim the concert has been "sprung" on them, although festival organisers deny it will bring any problems.
Meanwhile, street performances at the festival will have to end earlier, by 2000 BST over the weekend.
Local resident and town mayor Mike Gittins is unhappy about the Super Furries' appearence at Watton Field, blaming "11th hour planning".
Festival director Andy Eagle defended the concert but conceded that another venue may have to be found, because of the opposition.
The concert is subject to a public entertainment licence application to Powys council on 28 July.
Mr Gittins said: "Watton Field is hallowed turf in the town surrounded by homes inhabited by senior citizens.
"They, and others in the town, are concerned about the Super Furry Animals' concert.
"I think people associate rock and roll with sex, drugs and alcohol and they are also concerned that the Super Furry Animals are more of a rock group than a jazz group."
He added that the 1,000-capacity market hall would have been better suited to the concert than a marquee.
"The festival has been allowed to use Watton Field on the Friday, Saturday and Sunday, but it seems the organisers have sprung this extra date on us.
"If they'd planned it properly and consulted with the relevant agencies, then it probably would have gone through.
"But the 11th hour planning has caused problems."
Andy Eagle says SFA will launch the festival on 12 August
Mr Eagle said: "The Super Furries gig is for young people and I think it's an insult to say that there will be individuals intent on causing chaos at the gig.
"The people opposed are in the minority and these people should remember that they were young once too.
"We've had bands and individuals kick off the festival before who've had no jazz links.
"A few years ago Blues legend Buddy Guy and then Van Morrison appeared and they're both not considered jazz acts.
"We should be proud the Super Furries are coming, it's a real coup for the town."
Mr Eagle confirmed that restrictions had been placed on street performers who will only play until 2000 (BST) this year, unlike 2300 (BST) in previous years.
"There are concerns about this issue and that's why there's no bandstand. The times of the performances have been restricted too," he added.
The Super Furries gig was announced last week. Organisers have put 1,000 tickets on sale.