Luton boss Kevin Blackwell feels disagreements over the site of a new ground may threaten the club's future.
The Hatters hope to leave Kenilworth Road for one of two locations but each plan has its opponents.
Blackwell told BBC Three Counties Radio: "I've never seen in-fighting like I've seen at Luton Town.
"I am absolutely astonished at what's going on here. There are fans who want to seem to carp and chip away at the very fabric of the football club."
There is no site within the Luton town centre area for a new stadium but there is a location by junction 10 of the M1 where there is enough land for a ground but without other facilities.
The club is currently planning to submit plans for a site at junction 12 of the M1 which does have room for development, training pitches and other facilities.
However, some fans do not want to move there because the site comes under South Beds District Council which is considered to be outside Luton.
The point has been made that infilling will eventually link it up with Luton Borough but the various arguments continue to rage.
There are also doubts over whether planning permission will be granted at junction 12, as it will have to go the Secretary of State.
Blackwell added: "There's a chairman, who's come on board, for whatever reason's getting hammered left right and centre, and he's just put a load of money in. Now what do people want here? Is there an ulterior motive?
"And looking from afar, we could never understand why there's such vitriol around the club to do with moving to stadiums, to do with moving other people out of their positions. It really is amazing to me, and it cannot help this football club one iota."