The fossils were unearthed seven kilometres (four miles) from the renowned Sterkfontein caves north of Johannesburg, which have yielded many hominid finds, including the recent discovery of a complete 3.3 million-year-old arm and head.
Two fossils were discovered, a skull and a jaw, which are between 1.5 and 2.0 million years old.
"They are not direct ancestors of modern humans but are more like 'kissing cousins' of our ancestors," said Lee Berger, director of the Palaeoanthropological Unit for Research and Exploration at Johannesburg's University of Witwatersrand.
The scientists say the differences and similarities of the hominids from human ancestors, and the reasons for their extinction, may provide clues to humanity's own evolution.
Lost lovers?
The pair of fossils were christened Orpheus and Eurydice after the lovers of Greek mythology and have been identified as Paranthropus robustus, a hominid line that became extinct about one million years ago.
Scientists say the skull belongs to a female of the species while the jawbone belongs to a male.
The fossils were unearthed seven kilometres (four miles) from the renowned Sterkfontein caves north of Johannesburg, which have yielded many hominid finds, including the recent discovery of a complete 3.3 million-year-old arm and head.
Describing the day he and his team made the discovery, palaeontologist Andre Keyser said: "I knew immediately what I was dealing with and was extremely excited and absolutely delighted to have found it. It was certainly the highlight of my career as a palaeontologist."
Family tree
The genetic ancestors of humans are thought to have split from the apes between six million and four million years ago.
Robustus is one of several variations of the genus Australopithecus, which lived between 4.2 million and 1.5 million years ago and whose remains have been found in eastern and southern Africa.
"There has never been a better discovery in this little-known branch of the human family tree. For the first time, we can directly compare unequivocally associated male and female robust Australopithecines," Dr Berger said.
The reasons for the extinction of some hominid lines are uncertain. Some scientists suggest that the ancestors of modern humans showed an early ruthlessness by eliminating potential rivals.