De La Hoya (l) has decided against fighting Vargas
The fight would have been a boxing showdown of rival junior middleweight world champions
Gary Shaw, Vargas' agent and chief operating officer of Main Events, said he was told the news on Thursday during a telephone conversation with De la Hoya's promotor, Howard Rose.
"I am confused and disappointed in De la Hoya's decision," said Shaw.
"We had agreed to terms and were just waiting for the attorneys to iron out the final details.
"There were no outstanding issues."
The fight would have been dependent on Vargas beating Jose Flores of Mexico for the vacant WBA super welterweight title on 22 September.
There has been speculation that De la Hoya has a chance to line up a better deal with "Sugar" Shane Mosley or unbeaten Puerto Rican star Felix Trinidad, each of whom has beaten De la Hoya.
Trinidad, 40-0 with 33 knockouts, has a middleweight title unification showdown set with American Bernard Hopkins for 15 September. Trinidad is looking to face undisputed light-heavyweight king Roy Jones.
Mosley, the WBC welterweight champion, is 38-0 with 35 knock-outs.