Robin van Persie rescued a point for Arsenal with a last-minute strike as Southampton were denied a come-from-behind win at Highbury.
Thierry Henry put Arsenal ahead after earlier missing a penalty but Southampton equalised when Delap headed home Neil McCann's precise corner.
The pair combined again, when McCann crossed and Delap headed his second to put Saints ahead.
But substitute Van Persie curled in a left-foot shot for a late leveller.
The Dutchman's first Premiership goal ensured Arsenal avoided a second successive defeat after suffering their first in 50 matches last week at Old Trafford.
And with Chelsea winning convincingly at West Brom, defeat would have seen the Gunners surrender their place at the top of the table.
But their determination to bounce back from the loss to Manchester United last weekend was evident from the kick-off.
Jose Antonio Reyes and Henry's pace and direct running made life difficult for Saints' right-back Darren Kenton.
But it was a clumsy challenge on the less pacey Dennis Bergkamp that saw Kenton concede an early penalty.
Henry stepped up to take the kick, but crashed his effort against the foot of the post.
It was the second time the Highbury woodwork had been blasted, Nilsson having smashed his volley against the upright for the visitors after connecting with Danny Higginbotham's cross.
The Saints, looking for their first win in eight league games, continued to keep Arsenal at bay with their well-organised defence.
And Arsenal looked to have lost their most potent attacking threat when they were forced to replace Reyes, who suffered an injury after Nilsson stepped on his hand.
But moments after the switch, the hosts took the lead.
Bergkamp's long ball over the top found Henry, who collected the ball with one touch and slipped the ball beyond Niemi with his second.
It must have been a relief for Henry, who had been foiled by a superb save from Niemi in the first half.
Henry sparked into life, creating an excellent chance for Freddie Ljungberg to add a second goal soon after.
The Frenchman sprinted down the left and squared for Ljungberg, but Niemi athletically denied the Swedish midfielder.
And Arsenal quickly regretted the missed chance as Delap headed a rare goal from a rare Saints attack to put his side back on terms.
The shock was still reverberating around Highbury when Delap did it again, once more powering home a superb McCann set-piece - this time a free-kick.
The amazing turnaround looked set to give Steve Wigley his first league win since taking over as Saints boss.
However, in a frantic finish, Arsenal substitute Cesc Fabregas worked the ball to Van Persie, who still had plenty to do in a packed penalty area.
But the Dutchman checked inside his defender and curled a shot into the top right-hand corner.
And there was even time for Van Persie to spurn a golden chance to win the match, glancing a header into the arms of a grateful Niemi despite being unmarked.
Arsenal: Lehmann, Lauren, Cygan, Toure, Cole,
Ljungberg (Van Persie 83), Edu (Fabregas 65), Vieira,
Reyes (Pires 61), Bergkamp, Henry.
Subs Not Used: Senderos, Almunia.
Goals: Henry 67, Van Persie 90.
Southampton: Niemi, Lundekvam, Jakobsson, Higginbotham, Telfer,
Anders Svensson (Prutton 85), Delap, Nilsson, Kenton,
Ormerod (McCann 75), Blackstock (Best 90).
Subs Not Used: Blayney, Baird.
Booked: Telfer, Kenton, Ormerod, McCann.
Goals: Delap 80, 85.
Ref: M Messias (S Yorkshire).