Exiles from Diego Garcia are not entitled to jobs and housing benefits in the UK, appeal judges have said.
The islanders have campaigned for the right to return to the island
Three councils in the Gatwick area refused benefits to the families when they arrived in 2004, on grounds they were not habitually resident in the UK.
They took the move under a rule made to stop benefits tourism. Families claimed discrimination at the appeal court, but three judges ruled against them.
The UK expelled the Diego Garcians from the Indian Ocean island in the 1970s.
Residents of the island in the Chagos archipelago were moved to Mauritius in 1971, when Britain leased the island to the US to use as a military base.
Many exiles came to the UK and settled near Gatwick Airport in Sussex when they landed.
The Chagosians are UK passport holders entitled to reside in Britain.
The case over the islanders' benefits claims was contested by the Department of Work and Pensions and three local councils - Crawley, West Sussex, and Reigate and Banstead.
Thirty one exiles took a case to the appeal court, claiming discrimination because British citizens arriving from Ireland did meet benefits requirements.
They also said the government should have taken into account its responsibility for their impoverished circumstances.
Diego Garcia became a US airbase in the 1970s
Lord Justice Wall said the British government had acted unlawfully in refusing to allow the Chagosians to return to the Chagos Islands.
But he said that "does not mean the secretary of state is required to take the unlawful actions of the British government into account when declining to alter the habitual residence rules".
He added that Chagosian families were not discriminated against when they moved to the UK, and said concessions given to British citizens in Ireland had their roots in historical and geographical connections.
The Court of Appeal has previously ruled the families have the right to return to the islands, but the Government has applied to take the case to the House of Lords for a final decision.