Barry Robson scored a penalty winner as Scotland's title race pendulum swung back Celtic's way after a dramatic win that extends their lead to five points.
Scott McDonald looked offside as he fired Celtic into an early lead.
Two Steven Davis corners were headed home by David Weir and Daniel Cousin for Rangers, but a McDonald drive looped in via a deflection.
And, when McDonald won a penalty, Robson fired home, after which Rangers' Steven Whittaker was shown a red card.
Rangers still have three games in hand despite back-to-back Old Firm derby defeats.
But the reigning champions could have increased their lead by another three points by the time their Glasgow rivals face Hibernian on Sunday should Gordon Strachan's side defeat Motherwell 24 hours earlier.
Cousin had been preferred to Jean-Claude Darcheville up front for Rangers.
It appeared that the Frenchman was being rested with Thursday's second leg of the Uefa Cup semi-final in mind.
But his replacement immediately looked to justify his selection, with goalkeeper Artur Boruc having to beat away the striker's drive on the turn from 12 yards following the first corner of the match.
Celtic responded to the early scare by taking the lead after four minutes.
With the influential Carlos Cuellar suspended, Christian Dailly had moved into the heart of the Rangers defence.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.