About 250 children are taking part in the forum
"To the adults in this room, I would say: let us not make children pay for our failures any more," he told the first UN Special Session on Children in New York.
The three-day conference is being attended by 60 world leaders and numerous celebrities, as well as about 250 child delegates.
Addressing the children directly in his opening speech, Mr Annan told them they had the right to education, health, a clean environment, and to live without the shadow of war, poverty, abuse or exploitation.
"These rights are obvious. Yet we, the grown-ups, have failed you deplorably in upholding many of them," he said.
"We the grown-ups must reverse this list of failures. And we are pledged to do so," he added.
Serious debates
But there are serious divisions at the conference, which aims to produce a final document setting new goals for the international community on issues such as reducing child poverty.
The BBC's Greg Barrow reports that the United States, Arab nations and the Vatican want references to sex education and "health services" - which they see as code for abortion - removed from the draft.
This group of countries also wants a lower priority given to parts of the document focusing on reproductive and sexual health, arguing that abstinence is the best recipe for avoiding unwanted pregnancies and sexual diseases among children.
The US is also deeply opposed to any reference to children's rights, which are viewed by the conservative wing of the Republican Party as a threat to the role of parents as the head of the nuclear family.
The disputes could even drag on past Friday's closing session.
About 3,000 delegates are attending the conference, along with 3,000 representatives of non-governmental organisations.
High-profile delegates
Many leading figures from business and the arts were also expected to attend, including Microsoft's Bill Gates, singer and actor Harry Belafonte and actor Roger Moore.
An estimated 150 million of the more than two billion children in the world are malnourished, and nearly 11 million die before their fifth birthday.
More than 120 million do not attend school, and an estimated 300,000 are believe to be fighting in wars as child soldiers.