The Referees Association (RA) believes match officials do face rising levels of criticism - but that is not stopping more people becoming referees.
Swede Anders Frisk found the pressures of refereeing too much
The positive news came as top level referee Anders Frisk quit the game after receiving death threats.
Chairman of the RA, Len Randall, told BBC Sport: "Officiating at any level of the game is more difficult.
"There is greater disdain for authority with players, managers and spectators more willing to behave badly."
He added: "But there's no real shortage of people wanting to be trained as referees."
Premiership referees are administered by the Professional Game Match Officials Ltd (PGMOL) and all but a handful officiate full-time as professionals.
There are 26,000 referees registered with the English Football Association alone. But for most the realities of combining refereeing duties with work and family commitments means retention, not recruitment, is the real challenge.
"In England, the FA trains about 9,000 new referees each year but about the same number tend to go.
"The drop-out rate in the early years of refereeing is quite high. Most people do it because they want to give something back to the game not because they want to be our next Fifa referee.
"It's like stamp-collecting or any other hobby - people get bored of it.
"If you want to become the next Graham Poll or Mike Riley you almost have to sell your soul to football because of the personal sacrifices you have to make."
The role of referees was thrust back into the spotlight after Swedish official Frisk retired from football last weekend.
The 42-year-old's display during Barcelona's first-leg win over Chelsea in the Champions League was heavily criticised by fans and Blues boss Jose Mourinho.
And Frisk, who was the fourth official at the Euro 2004 final, decided to bow out after he and his family allegedly received death threats.
Steve Powell, development officer for the Football Supporters Federation, says what happened to Frisk draws attention to the sort of pressures match officials are under.
"I don't think any referee is worried when someone stands up in the stands and criticises their decisions - that is part of the culture of the game," Powell told BBC Sport.
"But there is a line that shouldn't be crossed and that line is a long way south of what happened to Frisk. When it gets down to threatening life and limb that is not football supporters' behaviour, that is psychopathic behaviour.
Referee Halsey changed his mind about awarding a penalty after Arsenal complaints
"Frisk must think what happened is quite worrying because he got coins thrown at him in Rome and that is also something that is not acceptable.
"Referees do deserve some respect otherwise there will be none left in the game. I think we all just need to figure out how we can make sure any criticism stays within the ground."
While referees are used to abuse from the terraces, rising dissent on the pitch has also contributed to making their jobs harder.
Earlier this month, Southampton midfielder David Prutton was handed a 10-match ban for pushing referee Alan Wiley.
And referee Mark Halsey attracted double criticism last September when he changed his mind about awarding Fulham a penalty after virulent complaints from the Arsenal players. Arsenal went on to win 3-0.
While those were extreme cases, Randall points out players no longer show the same respect to referees.
"You see referees being mobbed by complaining players often using bad language," said Randall.
"It happens more and more so the referee has to work far harder to manage the match."
Powell agrees the players must also take some responsibility for stirring up antipathy towards referees.
"I don't think it helps when players get up close and personal to the referees screaming abuse at them," he added.
"You have to remember young supporters look up to players as an example and dissent on the pitch is one thing that ought to be clamped down on."