A last-gasp win over Rotherham could not stop Walsall going down on goal difference, with Gillingham surviving.
Simon Osborn put The Saddlers ahead from close range but he then fouled Scott Minto in the area and Michael Proctor equalised from the penalty.
Osborn was then dismissed for a second booking before Walsall substitute Mark Wright hit their second in a scramble.
Paul Warne equalised from another penalty but Paul Ritchie headed a third which ultimately counted for little.
Walsall made the brighter start and within the opening 90 seconds, Jorge Leitao tested Millers' keeper Mike Polliitt who also had to be alert to tip over Lee Bradbury's stinging drive.
But the Saddlers' early impetus petered out and Rotheham grew in confidence.
Proctor's powerful drive was parried by James Walker on 20 minutes, and the Millers squandered further chances as Walsall's defence went missing.
48 mins: Simon Osborn puts Walsall ahead.
59 mins: Michael Proctor levels from a controversial penalty.
62 mins: Osborn is sent off.
68 mins: Sub Mark Wright puts Walsall ahead.
90 mins: Paul Warne equalises from another penalty.
92 mins: Paul Ritchie heads Walsall's winner.
Proctor ran clear of the square back line, rounded Walker and shot from a narrow angle only for Neil Emblen to make ground and scramble the ball off the line.
Former Walsall striker Martin Butler also found himself with only Walker to beat but as he hesitated he allowed defenders to get back and crowd him out.
Walsall had to win and got their noses in front three minutes after the break.
It was not the tidiest of goals as Bradbury's shot was blocked on the line to spark a frantic scramble which ended when Osborn toe-poked the ball into the bottom corner.
Walsall were unable to hold on to their lead for long as Rotherham levelled in controversial fashion.
Referee Brian Curson pointed to the spot as Osborn tangled with Scott Minto and Proctor grabbed the chance to rifle home to break Walsall hearts.
Osborn was booked for protesting the penalty and Walsall's problems mounted when Osborn was shown a second yellow card for a foul.
But Walsall were not going to go without a fight and the 10 men regained the lead with 20 minutes left.
Their second goal mirrored the first, with another frantic scramble in the Rotherham goalmouth, this time ended by substitute Wright, who prodded the ball home from close range 20 seconds after being thrown into the fray.
Leitao again went close while Darren Wrack's shot was saved by Pollitt as Walsall looked to seal the points.
But referee Curson had a hand to play and five minutes from time he awared Rotherham a second controversial penalty when Richie Barker went down under Emblen's challenge.
This time Warne slotted home the penalty.
But Walsall responded and Ritchie planted in a header for an injury-time winner as the Saddlers were left to reflect on the fact that they went down by a solitary goal after finishing level on points with Gillingham.
Walsall: Walker, Ritchie, Roper, Emblen, Osborn, Andrews,
Bazeley (Aranalde 88), Wrack, Merson, Leitao (Fryatt 82),
Bradbury (Wright 68).
Subs Not Used: Kerr, Corica.
Sent Off: Osborn (62).
Rotherham: Pollitt, Minto, Gilchrist, Swailes, Stockdale,
Mullin, Morris, Shaun Barker (Monkhouse 56), Sedgwick,
Proctor (Warne 68), Butler (Richard Barker 80).
Subs Not Used: Montgomery, Hurst.
Referee: B Curson (Leicestershire).