Four local councils in Sussex and Surrey have united to express their concern over the cost of supporting refugees from Diego Garcia.
More than 60 islanders arrived at Gatwick Airport recently
Representatives from West Sussex County Council met government officials to discuss rising costs.
They were part of a group which also included Surrey County Council, Reigate & Banstead Borough Council and Crawley Borough Council.
They say central government and not local councils should be paying.
Diego Garcia, the biggest of the Chagos island group in the Indian Ocean, was given to the US to build a military base in the 1960s, and its population moved. Many ended up in nearby Mauritius.
But many have moved to England after complaining of discrimination.
Earlier this month the West Sussex County Council was told it had to pay £20 a week to support 10 of the islanders.
It has said the rising cost is worrying them.
"Although the Diego Garcians are British citizens they cannot claim state benefits until they can acquire 'habitual residence'," the council said in a statement.
The islanders have British citizenship
"In practice this may take several months. Their care and accommodation are being funded by the councils following court orders requiring this, or in some cases, because they have real care needs."
The councils met with the Office of the Deputy Prime Minister, Department for Work and Pensions, the Foreign and Commonwealth Office, the Cabinet Office and the Government Office South East.
Senor legal advisers on both sides will meet again soon, the council said.
Thirty of the islanders who arrived at Gatwick earlier this month are being housed in Horley in Surrey.