A scramble in the box after 43 minutes saw Paul Butler's shot skim off the head of Keith and into the net.
The Scottish forward then had a chance to atone but lost out in a one-on-one with United keeper Neil Sullivan.
Matthew Spring nearly doubled the lead for Leeds in the 77th minute but his rising drive was palmed over the bar.
Leeds manager Kevin Blackwell:
"To come to Plymouth, with the record they have had for the last year, and get a 1-0 victory is what any manager would like to do.
"After we weathered the early storm, we matched the power of Plymouth and then created some very good chances.
"I'm glad to have got the first away win but feel we have been unlucky not to get it before."
Plymouth: Larrieu, Connolly, Wotton, Coughlan, Gilbert, Norris (Milne 79), Friio, Makel, Capaldi (Hodges 24), Evans (Crawford 71), Keith.
Subs Not Used: Lasley, Adams.
Leeds: Sullivan, Kelly, Carlisle, Butler, Crainey, Richardson, Wright, Walton (Spring 66), Pugh, Deane (Ricketts 82), Joachim (McMaster 82).
Subs Not Used: Carson, Duberry.
Goals: Keith 43 og.
Ref: S Dunn (Gloucestershire).