The Moroccan, a massive favourite, looked inconsolable after his defeat, which mirrored his failure to win gold in Atlanta four years ago.
The 21-year-old Ngeny, last year's world silver medallist behind El Guerrouj, stuck to him like a shadow throughout the race and then found the power to forge past in the final straight, winning in 3:32.07.
That eclipsed the Olympic record set by Seb Coe in Los Angeles in 1984 (3:32.53).
World record holder El Guerrouj's legs seemed to go in the final straight as he finished in 3:32.32. Bernard Lagat of Kenya took bronze.
After crossing the line, El Guerrouj slumped on the track, looking down at his feet.
Celebrated
The 26-year-old, who celebrated his birthday the day before the Games' opening ceremony, had lost only one 1500m race since his Atlanta defeat four years ago. His victories included world titles in 1997 and 1999.
He was led out by compatriot Youssef Baba in the early stages of the race, Baba sacrificing his own chances in an attempt to break up the field.
But Ngeny and Lagat stuck close throughout before the dramatic finale.
El Guerrouj had lost in Atlanta when well placed at the bell, clipping eventual champion Noureddine Morceli's heel and crashing to the track before finishing last.
The Moroccan, who has been troubled by a leg injury, clutched his thigh after the race and left the track in tears again.
"I'm shocked he lost," said John Mayock of Britain, who finished ninth.
"It's quite sad really - he was the deserving winner."