"It was a brilliant second half. We had our backs to the wall but we got stuck in and came away with the result we wanted," said Ulster coach Brian McLaughlin.
Ulster's Ian Whitten in action against Richard Fussell of Ospreys
Biggar drew first blood for Ospreys with an early penalty which was matched by O'Connor seven minutes later.
Ulster raised their game and a great charge into the Ospreys' 22 by ex-Springbok and new signing Pedrie Wannenburg led to Ulster putting a penalty into the corner.
From the line-out Rory Best's throw missed Johann Muller but when Jerry Collins fumbled the overthrow, Ferris gratefull gathered to run over unopposed. O'Connor slotted the conversion and Ulster were 10-3 up.
That advantage was stretched to double figures when O'Connor landed a long-range penalty.
Ulster coach Brian McLaughlin on his team's 27-26 win over Ospreys at Ravenhill
However, Ospreys showed why they are the current title holders by fighting their way back.
First Biggar slotted a penalty after Ulster strayed offside and then, even with Nikki Walker sent to the bin for taking out the airborne David McIlwaine, Fussell was put away in the left corner after it was Ulster had messed up a line-out.
Biggar's excellent conversion tied the scores, but Ulster crept ahead again after Cave shook off Marty Holah for an unconverted score.
Biggar then kicked his fourth shot at goal from four attempts after Bryan Young was penalised at a scrum.
The home side stretched their lead just after the restart with an O'Connor penalty but another Ulster error allowed the Ospreys back into the game.
Scrum V analysis: Ulster 27-26 Ospreys
A poor pass from O'Connor to McIlwaine was hacked downfield and Biggar got to the ball first for a 42nd-minute try which he converted to put Ospreys 23-21 up.
O'Connor was then wide with a long range penalty before he put Ulster one point ahead with his fifth success from eight attempts after 53 minutes.
But the lead did not last as Biggar kept his 100% kicking record intact with six from six attempts to put Ospreys into a 26-24 lead after Ulster infringed.
But Ulster were not finished and after the Ospreys strayed in front of a kick ahead, O'Connor stepped up to slot what proved to be the winning penalty.
Ospreys fly-half Dan Biggar: "It was very frustrating and certainly a game we should have won. When you go ahead three or four times you should win.
"Our discipline was extremely poor and cost us the game.
"I was in good kicking rhythm, but it was just extremely disappointing."
Ulster: Ulster: D'Arcy, Seymore, Cave, Whitten, McLlwaine, O'Connor, Marshall, Young, Best, Fitzpatrick, Muller, Barker, Ferris, Faloon, Wannenburg. Replacements: Wallace for Whitten (21), Court for Young (40), Brady for Best (48), McAllister for Fitzpatrick (65), Diack for Ferris (65). Not Used: Caldwell, Porter, Shiels.
Ospreys: Owen, Walker, Parker, Bishop, Fussell, Biggar, Nutbrown, Bevington, M. Davies, Griffiths, Gough, Evans, Collins, Holah, Thomas. Replacements: Byrne for Owen (65), Williams for Fussell (57), Webb for Nutbrown (77), D. Jones for Bevington (76), Bennett for M. Davies (60), James for Griffiths (60), Goode for Evans (70), Lewis for Holah (73).
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.