The US has accused China of refusing to take back almost 40,000 Chinese immigrants who have been denied permission to stay in the country.
Mr Chertoff said new detention centres would be opened
The United States Homeland Security Secretary Michael Chertoff described the situation as "intolerable".
He said although China had taken back 800 citizens last year, that only made a small dent in the numbers.
Mr Chertoff said there was a backlog of thousands of illegal Chinese immigrants in the United States.
Speaking to the Associated Press news agency, Mr Chertoff said illegal immigrants were clogging detention centres at federal expense.
"We can't be in the position any longer where we are paying the burden and bearing the burden for countries that won't co-operate with us and take their own citizens back," he said.
According to the Homeland Security Department, the American government has spent an estimated $667m in detention costs on the Chinese, AP reports.
Mr Chertoff said other countries were also refusing to take back their citizens but said the numbers from China far outweighed those from other nations.
He suggested that financial decisions and a lengthy process could be behind China's reluctance to take back its citizens, but also blamed "a low priority" by the Chinese authorities.
He also revealed that the Homeland Security Department would be opening detention facilities in the next few weeks to house entire families, but said "we're not going to let people get away with this".
Under current legislation, illegal immigrants can be detained for 180 days before they have to be released.
In October, new laws come into force which will allow illegal immigrants to be held until their home country is prepared to take them back.
Mr Chertoff is due to visit China, Japan and Singapore at the end of the month.