Kenny Miller broke his Celtic scoring duck as the SPL leaders increased the gap between themselves and city rivals Rangers to seven points.
Miller and Gravesen scored their first Celtic goals
Rangers goalkeeper Allan McGregor brilliantly denied Aiden McGeady, but Thomas Gravesen was in the right place to head home the opening goal.
Lee Naylor cleared off the Celtic line, while a Kris Boyd shot struck the bar.
But, by then, Rangers were two behind, McGeady's through ball having set up Miller to poke home for a deserved win.
The former Rangers striker had been preferred to Maciej Zurawski in the Celtic attack and was about to be replaced with the Pole before striking that decisive goal.
Sasa Papac had started ahead of Karl Svensson in the Rangers defence alongside Julien Rodriguez after an impressive midweek display.
Boyd was given a start, being paired with Prso up front.
However, it was Boyd's Scotland team-mate, Miller, who had the first clear-cut chance of the game after racing clear of Papac to a long ball out of defence.
But, with McGregor racing off his line, the attempted lob was high and wide.
Rangers looked less dangerous with an equal amount of possession, although Charlie Adam did threaten the goal with a low, curling free-kick.
McGregor blocked clear a powerful effort by Shunsuke Nakamura and did even better when turning away Jan Vennegoor of Hesselink's dipping 20-yard volley at full stretch.
Boyd powered a header over the bar from 10 yards, but Celtic had started to dominate possession.
Rangers' Barry Ferguson seemed shocked by the defeat
And they took the lead when a Neil Lennon cross found the head of the unmarked McGeady eight yards from goal.
Although McGregor made a superb block with his right hand, Gravesen was following up to head his first Celtic goal from a couple of yards out.
McGeady almost had his name of the scoresheet with a 25-yard drive that skimmed the top of the crossbar.
Rangers looked to have been denied a penalty when Gary Caldwell appeared to prevent a Barry Ferguson cross reaching Adam's head with his hand.
Boyd squandered a chance when he blasted over from 20 yards after a fine chip through by Ferguson.
But, just as Rangers started to apply some pressure, McGeady picked out Miller, who grabbed his first goal since joining Celtic.
Lee Naylor had to hook the ball off the Celtic line from a Prso header and, after an excellent through ball from the Croat, Boyd clipped the top of the bar with a shot on the turn.
But it was Celtic who ran out deserved winners of the first Old Firm derby of the season.
Celtic: Boruc, Telfer, Caldwell, McManus, Naylor, Nakamura (Sno 71), Lennon, Gravesen (Pearson 87), McGeady, Vennegoor of Hesselink, Miller (Zurawski 78).
Subs Not Used: Marshall, Maloney, Balde, O'Dea.
Booked: Vennegoor of Hesselink.
Goals: Gravesen 35, Miller 74.
Rangers: McGregor, Hutton (Sionko 61), Rodriguez (Rae 82), Papac, Smith, Burke (Martin 64), Ferguson, Hemdani, Prso, Adam, Kris Boyd.
Subs Not Used: Robinson, Svensson, Sebo, Buffel.
Booked: Hutton, Martin.
Ref: D McDonald