Witteveen spurned a great opportunity to double the hosts' lead with half an hour remaining when he failed to find the unmarked Glen on the edge of the penalty area.
And McCulloch made Hearts pay for their profligacy when he headed Smith's free-kick past the helpless Balogh.
The equaliser galvanised the visitors, with Smith, Davis and Naismith became more influential as Hearts seemed to panic with the ball and regularly surrender possession.
Suso stung McGregor's palms with a thunderous shot four minutes from time, as Hearts were left to rue a host of missed opportunities.
Boyd then broke Hearts when he scored from the spot after Bouzid's reckless challenge on Naismith, as Rangers stretched their unbeaten league run to 12 games.
Hearts manager Csaba Laszlo: "We played some good football and had some good chances. I have my opinion about the penalty, but the referee gave it and I cannot do anything about it.
"This was a better game than what we showed against Zagreb, but if you have so many young players then you lose a lot of the ball.
"But we were unlucky and we lose the three points. We made too many mistakes in midfield.
"I was forced to save Christian Nade after his yellow card. We had discipline and didn't make any big fouls.
"But up front we have a big problem."
Rangers manager Walter Smith: "I don't think it was a sending off, but it was an opportunity for the referee to send him off.
"I thought it It looked far worse than it actually was. "I don't think there was any great contact made and obviously if there had been, the player would never have been able to continue in the manner he did do for the rest of the game.
"In the second half we played better, we imposed ourselves on the game.
"Sometimes when you've got 10 men you can play between the opposition a bit better strangely enough. "At the time we took the lead I felt we deserved it for what we had put in to the game in the second half."
Live text and stats
Sunday, 23 August 2009
David Witteveen 31
Boyd (pen) 90
05 Jose Goncalves
13 Obua yellow card
21 Ismael Bouzid
06 Palazuelos yellow card
07 Suso Santana yellow card
14 Nade yellow card (Glen 32)
15 David Witteveen (Elliot 62) (Novikovas 70)
06 McCulloch yellow card
08 Thomson red card
18 Miller yellow card (Boyd 89)
29 Fleck (Smith 46)
Ref: ThomsonAtt: 16,284
Attempts on target
Attempts off target
90:00+5:47 The referee ends the match.
90:00+4:42 Ian Black fouled by Steven Davis, the ref awards a free kick. David Obua produces a shot on goal direct from the free kick.
90:00+3:33 Volleyed right-footed shot by Suso Santana. Blocked by Steven Whittaker.
90:00+2:49 Ian Black has an effort at goal from 25 yards. Save by Allan McGregor.
90:00+0:57 Unfair challenge on Suso Santana by Steven Whittaker results in a free kick. Suso Santana sends in a cross, Gary Glen has
a headed effort from close in and clears the bar.
89:54 The assist for the goal came from Steven Naismith.
89:54 GOAL - Kris Boyd:Hearts 1 - 2 Rangers Power penalty scored by Kris Boyd. Hearts 1-2 Rangers.
89:09 Foul by Ismael Bouzid on Steven Naismith, Penalty awarded.
88:35 Substitution Kris Boyd is brought on as a substitute for Kenny Miller.
88:35 Left-footed shot by David Obua went over the net.
86:50 Effort on goal by Arvydas Novikovas from just outside the penalty area goes harmlessly over the target.
86:10 Suso Santana takes a shot. Brilliant save by Allan McGregor.
85:19 The offside flag is raised against Suso Santana. Free kick taken by Steven Whittaker.
84:42 Ian Black has an effort at goal from outside the box missing to the wide left of the goal.
82:48 Effort on goal by Lee McCulloch from just inside the box goes harmlessly over the target.
82:18 The official flags Steven Naismith offside. Janos Balogh takes the indirect free kick.
81:46 Jose Goncalves takes the direct free kick.
81:46 Booking The referee shows Lee McCulloch a yellow card.
81:38 Lee McCulloch challenges Suso Santana unfairly and gives away a free kick.
79:25 Free kick awarded for a foul by Ian Black on David Weir. Free kick taken by Allan McGregor.
78:03 Free kick awarded for a foul by Ismael Bouzid on Kenny Miller. Steven Smith has a direct shot on goal from the free kick,
Janos Balogh makes a save.
77:41 David Obua has an effort at goal from outside the penalty box missing to the wide left of the goal.
76:20 Ian Black challenges Lee McCulloch unfairly and gives away a free kick. Direct free kick taken by Lee McCulloch.
75:04 Inswinging corner taken by Steven Smith, save by Janos Balogh. Steven Smith takes a outswinging corner to the near post,
Arvydas Novikovas makes a clearance.
72:04 Suso Santana takes a inswinging corner.
70:51 Steven Davis is ruled offside. Indirect free kick taken by Janos Balogh.
68:39 Sasa Papac takes the direct free kick.
68:39 Substitution Arvydas Novikovas on for Calum Elliot.
68:39 Handball decision goes against Ian Black.
67:02 Ian Black has a shot on goal from outside the area which misses left.
65:15 The assistant referee signals for offside against Suso Santana. David Weir takes the indirect free kick.
65:09 The referee blows for offside against Suso Santana. Indirect free kick taken by Sasa Papac.
62:03 Assist by Steven Smith.
62:03 GOAL - Lee McCulloch:Hearts 1 - 1 Rangers Lee McCulloch scores a headed goal from just inside the edge of the penalty area. Hearts 1-1 Rangers.
61:12 Steven Smith produces a cross,
61:12 Substitution Calum Elliot replaces David Witteveen.
61:12 Booking David Obua receives a yellow card for unsporting behaviour.
61:09 David Obua concedes a free kick for a foul on Kenny Miller.
60:08 Gary Glen has an effort at goal from 25 yards. Save by Allan McGregor.
57:51 Booking Ruben Palazuelos booked for unsporting behaviour.
57:03 Steven Whittaker challenges David Obua unfairly and gives away a free kick. Jose Goncalves takes the free kick.
54:28 Ismael Bouzid concedes a free kick for a foul on Steven Naismith. Free kick crossed by Steven Davis, Janos Balogh makes a
54:01 Shot on goal by Gary Glen from just inside the area clears the crossbar.
53:05 Steven Smith produces a cross, Headed effort from deep inside the penalty area by Steven Naismith misses to the right of the
52:50 Ian Black concedes a free kick for a foul on Steven Smith. Steven Davis takes the direct free kick.
50:03 Free kick awarded for a foul by Eggert Jonsson on Kenny Miller. The ball is crossed by Steven Davis, David Weir has a headed
effort at goal from deep inside the area missing to the right of the target.
47:44 Corner taken by Suso Santana from the right by-line to the near post, Jose Goncalves takes a shot. Brilliant save by Allan
45:01 The second half has started.
45:01 Substitution John Fleck leaves the field to be replaced by Steven Smith.
45:00+2:10 The half-time whistle blows.
45:00+1:42 David Obua handles the ball and concedes a free kick. Direct free kick taken by Madjid Bougherra.
45:00+1:09 Shot from just outside the area by Lee McCulloch goes over the bar.
44:59 Steven Whittaker challenges Ian Black unfairly and gives away a free kick. Direct free kick taken by Ismael Bouzid.
43:42 Outswinging corner taken from the left by-line by John Fleck, Madjid Bougherra takes a shot. Blocked by Eggert Jonsson. Free
kick awarded for an unfair challenge on Gary Glen by John Fleck. Janos Balogh takes the free kick.
40:25 Unfair challenge on Kenny Miller by Ismael Bouzid results in a free kick. Sasa Papac takes the direct free kick.
39:30 Madjid Bougherra challenges Gary Glen unfairly and gives away a free kick. David Obua delivers the ball from the free kick
left-footed from left wing, Sasa Papac makes a clearance.
38:47 Madjid Bougherra concedes a free kick for a foul on Ruben Palazuelos. Direct effort from the free kick comes in from Ian Black.
37:46 Shot from long range by Gary Glen misses to the right of the target.
33:37 Unfair challenge on Steven Davis by Lee Wallace results in a free kick. Allan McGregor takes the free kick.
32:13 Sasa Papac concedes a free kick for a foul on Suso Santana. Strike comes in from David Obua from the free kick, David Witteveen
gets a block in.
30:48 Substitution Christian Nade leaves the field to be replaced by Gary Glen.
30:48 The assist for the goal came from Christian Nade.
30:48 GOAL - David Witteveen:Hearts 1 - 0 Rangers David Witteveen gets on the score sheet with a goal from inside the penalty area to the bottom left corner of the goal. Hearts
29:11 Madjid Bougherra takes the direct free kick.
29:11 Booking Booking for Christian Nade.
29:03 Unfair challenge on Madjid Bougherra by Christian Nade results in a free kick.
28:11 Free kick awarded for a foul by Eggert Jonsson on Madjid Bougherra. Sasa Papac takes the free kick.
26:48 Foul by Christian Nade on Madjid Bougherra, free kick awarded. Allan McGregor takes the direct free kick.
26:32 The offside flag is raised against Kenny Miller. Ian Black takes the free kick.
26:05 The referee blows for offside against Suso Santana. Indirect free kick taken by Allan McGregor.
24:53 Centre by Suso Santana.
24:43 Ian Black challenges Lee McCulloch unfairly and gives away a free kick. Direct free kick taken by Lee McCulloch.
24:24 Unfair challenge on John Fleck by Ian Black results in a free kick. Steven Whittaker takes the free kick.
24:11 Suso Santana takes a shot.
22:52 Lee McCulloch challenges David Obua unfairly and gives away a free kick. Free kick taken by Jose Goncalves.
21:22 Foul by Christian Nade on Madjid Bougherra, free kick awarded. Free kick taken by Allan McGregor.
18:52 The ball is swung over by Suso Santana.
18:26 Short corner taken by Suso Santana.
16:44 John Fleck produces a left-footed shot from long range which goes wide right of the goal.
12:55 Free kick taken by Ismael Bouzid.
12:55 Sent off Red card for Kevin Thomson.
12:48 Free kick awarded for an unfair challenge on Ian Black by Kevin Thomson.
8:50 Ian Black takes a shot. Allan McGregor makes a save.
6:40 John Fleck concedes a free kick for a foul on Lee Wallace. Janos Balogh takes the free kick.
4:05 Corner taken left-footed by John Fleck from the left by-line to the near post, Ian Black fouled by Lee McCulloch, the ref
awards a free kick. Direct free kick taken by Janos Balogh.
3:14 Booking Kenny Miller is shown a yellow card.
3:10 Booking Suso Santana receives a yellow card for unsporting behaviour.
1:21 Steven Naismith concedes a free kick for a foul on Suso Santana. Suso Santana crosses the ball from the free kick right-footed
from right wing.
0:00 The match begins.
Live text and data provided by The Press Association.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.