Instead he was invited to address a meeting at a neighbouring location, which he refused to do.
The exiled Tibetan spiritual leader has not been invited to the UN-sponsored religious conference because of fears that China would protest.
Instead he was invited to address a meeting at a neighbouring location, which he refused to do.
The decision has drawn criticism from a number of countries, including the United States, which has expressed its "deep disappointment".
Mr Annan said he understood the concerns at the Dalai Lama's exclusion, but said it was necessary to maintain consensus at the UN.
"This house is really a house for the member states and their sensitivities matter," he said.
Official delegation
Tashi Wangdi, a minister in the Tibetan Government in exile, said the decision was clearly made at the behest of the Chinese authorities.
"His Holiness is recognised by the world as one of the leading religious leaders," Mr Wangdi said.
"When he is excluded it undermines the credibility of the conference and the image of the United Nations," he added.
China has sent its own, government-approved delegation of church leaders to the summit.
The delegation leaders accused the Dalai Lama of trying to split China and described him as not a peace-maker but a trouble maker.
China regards Tibet as part of its territory and views the Dalai Lama as a separatist leader.
He fled Tibet in 1959 after a failed uprising against Chinese rule.
He continues to campaign for Tibetan rights from India where the government in exile is based.
'Appeasement'
UN officials said although the summit was meeting in the UN General Assembly Hall, it was not being organised by the UN itself, but by a group of religious leaders.
More than 1,000 leaders representing 75 different faiths are expected to attend the meeting, which opens on Monday.
US Senator Jesse Helms - always a virulent critic of the UN - said the body had "chosen to appease the communist dictators" in China.
He said it was "ironic that one of the world's most brutal atheistic dictatorships can dictate to the UN who can and who cannot participate in a so-called world peace summit."
But Mr Annan said: "In any effort of this kind, you try to make progress, take progress as you get it, and not hold out for the absolute best."