O'Sullivan was on the Romanian's shoulder as the two athletes rounded the final bend - but Szabo kicked again and held her lead of two metres under intense pressure to finish in 14mins 40.79secs.
O'Sullivan, who returned to top class competition this year after the birth of her daughter, gave everything on the final lap in the attempt to overtake the rival she beat at the Europeans in Budapest two years ago when she completed an historic double of 5,000m and 10,000m.
But Szabo would not be denied as she slashed 19 seconds off the Games record set by Wang Junxia of China in Atlanta in 1996.
Thirty-year-old O'Sullivan clocked 14:41.02 - a new Irish record - to take silver from Ethiopa's Gete Wami, who will console herself with the thought that her favoured event - the 10,000m - is still to come.
Britain's Joe Pavey ran a creditable 14:58.27 to finsh 12th, after a season when her preparations were marred by injury.
Comeback
O'Sullivan revealed she had been lifted by her loyal support in the Olympic Stadium: "I felt half guilty doing a lap of honour but there were so many Irish fans and Irish flags in the stadium I felt I had to show my appreciation so I went round the track.
"I would love to come back and go round again," she said.
"It was nearly a dream come true. In some ways it is because I am going to get a medal round my neck and represent everybody in Ireland.
"I don't know whether to laugh or cry. I am caught in between really.
"I am happy with a silver - I think."
Related to this story:
Sonia's close encounter
(15 Sep 00 | Northern Ireland)
Wami turns heat on O'Sullivan
(21 Sep 00 | Northern Ireland)
O'Sullivan safely through to final
(22 Sep 00 | Northern Ireland)