Gavin Henson scored all the points as a below par Ospreys needed an injury-time penalty to extend their lead at the top of the Celtic League.
Ospreys (13) 22
Pens: Henson (5)
Ulster (11) 21
Tries: Maxwell, Bowe
Pens: Humphreys (3)
Ospreys led 19-18 with five minutes to go, but Ulster looked to have snatched victory with a David Humphreys penalty.
Henson had already scored a try and kicked 14 points before he kept his nerve to land a long distance penalty.
Ulster won the try count with scores in each half from wingers Andrew Maxwell and Tommy Bowe.
Henson's try was embarrassingly simply as he met no resistance as he glided through a gaping hole in the Ulster defence after catching an up and under on the run just inside his own half.
The 22-year-old, starting at full-half instead of centre, added the conversion to give his side a perfect start.
A missed penalty each followed before Humphreys found the target with his second attempt.
The Ireland international then created his side's first try on 32 minutes to get the Ulstermen back into the match largely against the run of play.
As with the Ospreys try, the defensive cover was poor as winger Maxwell burst onto the fly-half's inside pass.
He had enough pace to beat the rather lethargic cover defence.
Henson and Humphreys had two penalty attempts each in the closing stages of the half.
The Ulsterman missed one but the Welshman landed both to ensure the home side took a 13-11 lead into the interval.
The 'Henson show' continued five minutes after the break as he kicked his fourth penalty.
But a rare piece of inspired attacking play from Ulster's Kieran Campbell soon put the visitors back ahead.
The scrum-half broke quickly down the blind side from the base of scrum, brushed off Adrian Durston's attempted tackle and chipped forward perfectly for Bowe to dive on the loose ball for a try.
Humphreys' conversion put Ulster ahead, but the lead last just five minutes as Henson added another three penalty points.
The Ospreys began to control possession and territory; but just when they thought they had done enough to secure the win, Humphreys' third penalty put Ulster 21-19 ahead with six minutes left.
In keeping with the seesaw nature of the game, there was time for another twist as more Ulster indiscipline gave Henson the chance to snatch victory at the death.
Ospreys: G Henson; S Terblanche; D Bishop, D Tiueti, S Williams; M Jones, J Spice; P James, B Williams (capt), A Jones, B Cockbain, J Thomas, A Lloyd, S Tandy, R Jones.
Replacements: R Hibbard, A Millward, A Newman, R Pugh, R Rees, E Seveali'i, A Durston.
Ulster: B Cunningham; T Bowe, P Steinmetz, K Maggs, A Maxwell; D Humphreys, K Campbell; R McCormack, S Best, R Best, G Longwell, M McCullough, C Feather (capt), N Best, R Wilson.
Replacements: P Shields, R Frost, A Ward, R Spee, A Larkin, S Stewart, Shawe.
Referee: Rob Dickson (SRFU)