The prison in Cuba currently has about 270 detainees
The US is "stuck" with the Guantanamo Bay detention centre even though it wants to close it, Defence Secretary Robert Gates has said.
Mr Gates said the US wanted to send up to 70 prisoners home but countries would either not take them or could not be trusted to.
Human rights groups have long argued for its closure, saying it does not meet international legal standards.
The prison in Cuba currently has about 270 detainees.
Mr Gates told a US Senate hearing: "The brutally frank answer is that we're stuck. We have a serious 'not in my backyard' problem.
"Either their home government won't accept them or we're concerned that the home government will let them loose once we return them home," he said.
"What do you do with that irreducible 70 or 80 who you cannot let loose but will not be charged and will not be sent home?" he asked.
The Pentagon has said 36 former inmates who were released are "confirmed or suspected of having returned to terrorism".
Democratic Party Senator Dianne Feinstein told Mr Gates: "Nothing you have said absolves the enormous loss of credibility we have in the eyes of the world.
"We are being called hypocrites, that we have double laws, laws for some, and no laws for others."
Rights groups condemned Mr Gates's attitude.
Vincent Warren, executive director of the Centre for Constitutional Rights, told Reuters news agency: "The secretary's comments really are astounding in light of the money, resources and personnel of the department of defence."