The ruling said their trial had been "critically flawed".
The court also slashed the sentences of two other Bosnian Croats who had been convicted of involvement in the 1993 massacres in Ahmici, when more than 100 Muslims civilians were slaughtered.
The men's relatives in the public gallery hugged each other and cried out in joy as presiding judge Patricia Wald announced the decision.
'Unreliable' witnesses
The judges ordered the immediate release of brothers Zoran and Mirjan Kupreskic and their cousin Vlatko Kupreskic.
They also cut the 15-year sentence of Drago Josipovic to 12 years, and the 25-year sentence of Vladimir Santic to 18 years.
The court said prosecutors had built a weak case against the three members of the Kupreskic family, based on "unreliable witnesses".
"The cases against them cannot stand," the ruling said.
At the trial last year, police commander Vladimir Santic, 43, was found guilty of co-ordinating the massacre.
The other four men were convicted for planning and participating in the atrocities.
Women shot
Zoran Kupreskic, 43, was sentenced to 10 years, Mirjan, 38, received eight years and Vlatko, 43, was sentenced to six years for persecution.
Drago Josipovic, 46, was convicted of murder and persecution.
All five defendants appealed against their sentences, citing inconsistencies in witness statements.
The Ahmici massacre occurred when Bosnian Croat police attacked the village, burning many victims alive in their own homes.
During the trial witnesses said the attackers shot Muslim women and children trying to flee.
The indictment said that after the attack, there were no Muslims left living in the village.