Newcastle turned on the power late on to pile pressure on Gary Megson, whose side also had Darren Purse sent off.
After a niggly first 45 minutes, Lee Bowyer hit the woodwork before Purse saw red for a second bookable offence.
Kluivert made it 1-0 after Russell Hoult parried a cross from Laurent Robert, who then struck the bar.
Substitute James Milner got on the end of a Jermaine Jenas cross and Alan Shearer grabbed number three, before Geoff Horsfield netted a consolation.
The Baggies were looking to make amends for their Carling Cup defeat by Colchester - but they could have had more problems after just a minute.
Kluivert headed over after a slip from Riccardo Scimeca allowed Craig Bellamy to get a cross in from the left.
Newcastle had a lucky escape when Rob Earnshaw beat the offside trap, only to stand on the ball when a run in on goal looked likely.
But the Magpies were making all the running and Hoult saved comfortably from a long-range Bowyer effort before the midfielder headed narrowly wide from Bellamy's cross.
Purse appeared to handle a Shearer shot and Bowyer was impeded by Thomas Gaardsoe after finding himself in space in the area, but on both occasions referee Mike Riley waved play on.
Although Jason Koumas was allowed to fire in a 25-yarder that did not trouble Shay Given, and Purse's free header from a Koumas corner went straight at the keeper, the home side continued to look stronger.
Shearer went close with a header from Stephen Carr's cross and two tremendous blocks from Scimeca denied Kluivert.
But Bowyer should have made it 1-0 when sloppy defending saw him one-on-one with Hoult - but he could only find the side-netting.
With the ball coming back incessantly at his defence Megson brought on Horsfield at the break for Earnshaw, who had been ineffectual since his early forays.
But Bowyer continued to pose a threat and his 35-yarder whistled past the post straight from the restart.
Clement used his renowned left-foot to good effect at the other end when his free-kick from even further out flashed wide.
Bowyer then went desperately close to breaking the deadlock when his curling effort hit the angle of post and crossbar after some improvisation on the edge of the area.
It needed a timely intervention from Purse to deny Shearer and substitute Robert fashioning an opening.
But it all went horribly wrong for the centre-back, who had been Albion's best player, when he brought down Kluivert and was sent off for his second bookable offence.
Nicky Butt had a claim for a penalty after being shoved off the ball by Martin Albrechtsen, but again the home side's cries were ignored.
Newcastle's frustration was lifted when Kluivert smashed home after Hoult could only palm out a Robert cross.
And, after Robert had also hit the woodwork, Milner - who had only come on moments earlier - made it 2-0 when he just about managed to turn in Jenas' cross.
Chances were coming thick and fast now and Shearer got in on the act, thanks to more good work from Jenas.
There was still time for Horsfield to head in Andy Johnson's cross and Robert to force Hoult into a fine reaction save.
It was Newcastle's second successive Premiership win under new boss Souness, but Megson will now be hoping for some patience from the Hawthorns hierarchy.
Newcastle: Given, Carr, Elliott, O'Brien, Bernard, Bellamy (Robert 55), Bowyer, Butt, Jenas, Shearer, Kluivert (Milner 77).
Subs Not Used: Harper, Ambrose, Hughes.
Booked: Bowyer, Elliott, Kluivert.
Goals: Kluivert 70, Milner 78, Shearer 86.
West Brom: Hoult, Scimeca, Purse, Gaardsoe, Albrechtsen, Greening, Clement, Johnson, Koumas (Gera 62), Kanu, Earnshaw (Horsfield 45).
Subs Not Used: Moore, O'Connor, Kuszczak.
Sent Off: Purse (63).
Booked: Purse, Greening.
Goals: Horsfield 87.
Ref: M Riley (W Yorkshire).