So now we know: Many, if not most, people alive today have some Neanderthal ancestry.
This finding, which comes from analysis of the Neanderthal genome, has taken many experts by surprise.
It tells us there was some mating between modern humans - our own lineage - and the Neanderthals before the latter went extinct some 24,000 years ago.
Neanderthals were a different type of human: very like us, though not exactly like us.
Recent research has done much to debunk the 19th Century view of these ancient people as brutish and dumb - even if that image lives on in popular culture.
Their stone tools were just as efficient as those used by early modern humans. They were able to exploit marine resources - such as dolphins and seals - for food.
They apparently used body paint, keeping shells to mix and store pigments.
Living people from Eurasia and Oceania overwhelmingly trace their ancestry to a small population of modern humans which left Africa 50,000-60,000 years ago.
But some 2% of the genomes of non-Africans are Neanderthal.
Added to the mix
"I think we just need to step back a bit now for a while and see what this means for the big picture of modern human evolution," says Professor Chris Stringer, head of human origins at London's Natural History Museum.
"Certainly, I had moved away from the idea that there was one single place in Africa - a 'Garden of Eden' if you like - where modern humans originated at one point in time, and that most of our features appeared at that point.
"I doubt that happened. I think it's a much more gradual process. I think modern humans were assembled from different populations contributing genes and behaviour. Now we may need to add a bit of Neanderthal to that mix as well - outside Africa."
The ancestors of Neanderthals diverged from our direct evolutionary line some 400,000 years ago.
The Neanderthals then mainly evolved north of the Mediterranean, while modern humans mainly evolved south of the Mediterranean.
So where and when did their close encounter occur? Neanderthal populations had probably been in decline for thousands of years by the time the "moderns" migrated out of Africa some 60,000 years ago.
Somewhere along the way - not too far from their homeland - modern humans may have encountered a population of Neanderthals and interbred with them.
Researchers have suggested that modern human males could have drawn into their fold some Neanderthal females, who subsequently bore them children.
Then, as the modern human population grew in number, so did the Neanderthal genes, spreading far beyond Africa.
Alternatively, there may have been an earlier period of mixing. Early modern humans were in the Levant - at the sites of Skhul and Qafzeh in Israel - some 120,000 years ago.
This was an early, failed foray out of Africa, unrelated to the event which would lead to the colonisation of the world 60,000 years later.
There is also evidence that Neanderthals were in the region 70,000-80,000 years ago.
It is unclear whether the two groups overlapped. But if they did, it is possible they also exchanged genes.
The Skhul and Qafzeh modern humans appear to have died out in the Levant. But their descendents might have retreated south, to the Arabian Peninsula or North Africa, perhaps taking a proportion of these Neanderthal genes with them.
"We know there were people in Arabia between 50,000 and 100,000 years ago making stone tools. The stone tools are very similar to the ones made by the Skhul and Qafzeh people," says Professor Stringer.
When modern humans once again left Africa 60,000 years ago, crossing the Bab-el-Mandeb straits to the Arabian Peninsula, they might have encountered this population and swept them up in the migration.
John Hawks, assistant professor of anthropology at the University of Wisconsin-Madison, says the discovery of Neanderthal genes in us has several implications.
"One implication is that Neanderthals are not a different species from us. Species are things that interbreed and have fertile offspring. That's us and Neanderthals," he told BBC News.
Despite the small number of genes which made it through, the Neanderthals did go extinct in Europe, western Asia and Siberia - for reasons which remain unclear.
Dr Hawks says research should now focus on the processes which led our ancestors to win out: "Does this population [modern humans] succeed because of some cultural trick they have? The Neanderthals are surprisingly sophisticated."
He adds: "The advantage the [modern humans] had - maybe it's disease; their resistance to something they carry with them. Like the squirrels in Britain. Or maybe it's social organisation, or maybe it's communication issues. But whatever it was, it was real important."
Researchers will now want to determine what the Neanderthal genes that are in us do and what, if any, significance they have for the biology of present-day people.
"We really don't know yet; these genes might have no effect - they could be neutral. They could have survived just because they were not disadvantageous," Professor Stringer speculated.
But future research is also likely to focus on the 70-plus amino acid changes that distinguish modern humans from Neanderthals.
"The idea is that you're going to identify the things that make us human," says John Hawks.
"The cool thing is that there are so few of them."