Julian Danskin had been convicted of molesting members of the Boys' Brigade Company of which he was the captain.
At a hearing before appeal judges in Edinburgh, his legal team argued that Danskin had not received a fair trial at Kirkcaldy Sheriff Court in 1999.
But the judges upheld the original conviction and the 18-month prison sentence which had been imposed upon him.
Outside the court, a spokeswoman for the victims' families said the process of rebuilding their lives would be a slow and difficult one.
Defence advocate Herbert Kerrigan QC said doubts existed about the trial.
He argued that evidence about an earlier extortion case in which Danskin was involved had prejudiced the jury against his client and led to a miscarriage of justice.
Danskin - who was chairman of East Fife Football club, and a senior partner in a law firm - was sentenced to 18 months in jail for three offences.
But he was released from prison after a week, pending the outcome of his appeal.
Relatives of Danskin's victims who were in court shouted "beast", as he was led from the dock.
Speaking outside the court a spokeswoman for the victims' families said: "For our sons it will take a long, long time I think before they can really move on with their lives because it is like a life sentence for them."
Ann Ward, one of Danskin's neighbours, said the outcome may provide some of the victims with a sense that justice had been done.
She said: "All the while that he was free there was this great concern that there was no justice and that justice wasn't going to be done.
"And I think what we have seen today will maybe restore some faith in the judicial system."