Shota Arveladze's extra time goal took Rangers into the CIS Insurance Cup semi-finals and ended a seven-game Old Firm winning streak for Celtic.
Celtic's Stanislav Varga hit the crossbar just before John Hartson got on the end of Alan Thompson's 65th minute corner to head the opening goal.
Dado Prso took the game into extra time with an 84th minute tap-in after David Marshall had saved from Hamed Namouchi.
And Arveladze shot home the winner in a pulsating extra period.
Rangers started strongly and Nacho Novo's powerful strike was blocked by the legs of David Marshall after an impressive run from the striker.
Peter Lovenkrands then sent a dangerous angled shot wide of Marshall's goal and the same player was denied a shooting chance minutes later by a great Didier Agathe tackle.
Celtic were struggling to get out of their own half and Rangers defender Zura Khizanishvili was next to try his luck when he shot into the side-netting after 18 minutes.
However, the visitors won a free kick in the 21st minute and Hartson met Thompson's delivery just six yards out, but his downward header hit the turf and bounced over the crossbar.
A disjointed ten minutes of play was followed by Agathe's impressive surge into the Rangers penalty area but the full-back's shot was blocked by the out-stretched leg of Jean-Alain Boumsong.
The interval came with no sign of the whirlwind pace abating and Rangers exerting the bulk of attacking pressure.
The second period began with Celtic asserting themselves more in midfield but neither side able to string a telling series of passes together.
Celtic came close to a goal after 64 minutes when Varga beat Stefan Klos to a headed knockdown from Hartson but the defender's toe-poked effort struck the crossbar.
But the opening goal came 60 seconds later when Hartson rose highest at the back post to head home a Thompson corner.
Rangers almost replied immediately when Lovenkrands broke down the left and centred for Namouchi, who could only direct the ball wide of the target from inside the six yard box.
Both sides gave everything in an untidy game
Rangers got on level terms when Novo's stray pass was played into the path of Namouchi by the heel of substitute Arveladze.
The youngster's low shot from 20 yards was well saved by Marshall but Prso was on hand to knock the loose ball into an empty net.
The home side were swarming all over the Celtic penalty area and Bobo Balde blocked a Novo shot before Agathe hacked away Boumsong's header from the goal-line.
In the third minute of extra time Celtic substitute Craig Beattie squeezed between two defenders to prod a shot past Klos but Boumsong recovered to clear the ball just a yard from the goal-line.
Rangers went in front in the 100th minute when Fernando Ricksen sprang a counter-attack from deep in his own half and found Arveladze, who drilled in a low shot from 15 yards.
Henri Camara escaped the Rangers defence early in the second period of extra time but shot wide of the target.
And Rangers held on for their first win over Celtic since the final of this competition in March 2003.
Rangers: Klos, Khizanishvili, Andrews, Boumsong, Vignal, Namouchi, Ricksen, Alex Rae (Hughes 111), Lovenkrands (Arveladze 79), Novo, Thompson (Prso 27). Subs Not Used: Graeme Smith, Hutton.
Booked: Prso, Vignal. Goals: Prso 85, Arveladze 100.
Celtic: Marshall, McNamara, Balde, Varga, Valgaeren, Agathe, Lennon, Petrov (Beattie 90), Thompson, Juninho Paulista (Camara 67), Hartson, Beattie (McGeady 105). Subs Not Used: Hedman, Pearson.
Booked: Varga, Balde, Hartson. Goals: Hartson 66.
Ref: S Dougal