The territorial issue had already strained relations between the two countries, and was to have been discussed during the Madrid meeting.
The island, known to the Spanish as Perejil and to the Moroccans as Leila, became an international flashpoint when Moroccan troops landed on it in July.
The island, only a few hundred metres off the Moroccan coast, belongs to Spain, but has been uninhabited for many years.
US Secretary of State Colin Powell intervened in an effort to resolve the dispute, and Monday's meeting was to have discussed the issue.
'Unacceptable'
Moroccan Foreign Minister Mohamed Benaissa told his Spanish counterpart Ana Palacio that the talks were off.
"A Spanish army helicopter landed today (Sunday) in the early afternoon on the Moroccan island," said a statement by the Moroccan foreign ministry.
This was "an unacceptable act by which Spain has violated, once again, national air and space and territory".
Agenda uncertain
The two sides had been expected to discuss the possible return of their respective ambassadors at Monday's meeting.
Morocco recalled its ambassador last October and Spain recalled its envoy during the Perejil dispute.
But the agenda had been left open, since neither side had been able to agree on exactly what should be discussed.
Spain wanted to talk about the problems of illegal immigration from North Africa into Europe.
But Morocco was determined to bring up the question of sovereignty of Spain's North African territories, which include the enclaves of Ceuta and Melilla, as well as a number of islands, including Perejil.
Morocco regards them as occupied territories and wants them back.
Spain says sovereignty of the territories is not up for discussion.
The BBC correspondent in Rabat says that relations now appear to have sunk to rock bottom.