Some species have never been seen alive, and are only known from dead whales washed up on shore.
"Beaked whales are among the least known, least understood and, frankly, most bizarre whales in the ocean," said Scott Baker, associate director of the Marine Mammal Institute at Oregon State University in the US.
"They are the only cetacean species with tusks, and scientists have long wondered why, since their diet primarily is squid."
Tree of life
The shape of the teeth, or tusks, varies markedly between different species.
In some, they actually appear to hinder feeding, as they wrap over the upper jaw, preventing it from opening fully.
Females do not show teeth; and this difference between the sexes, or sexual dimorphism, is virtually the only way to tell them apart.
The teeth are very different from the long, slender, spiralling tusks of the narwhal, which are thought to be primarily sensory organs.
Cuiver's beaked whale encounter
The research team, which also included Dr Merel Dalebout from the University of New South Wales in Sydney, took DNA samples from 14 beaked whale species and used it to construct a family tree depicting how the various species had developed.
One of the theories of beaked whale evolution is that different groups emerged in ocean canyons that were more or less isolated from the wider oceans, and that this pattern of evolution was responsible for different shapes of tooth.
But the genetic work suggests this is unlikely.
"It turns out that tusks are largely an ornamental trait that became a driver in species separation," said Dr Baker, whose research is reported in the journal Systematic Biology.
"The tusks help females identify males within their species, which could otherwise be difficult as these species are quite similar to each other in shape and coloration."
So females use the shape of the teeth to select males of the right species to mate with. They may also choose mates based on the size or shape of the individual's teeth or of the scars they bear.
In turn, this also means that the more successful males are the ones with the shape of teeth most characteristic of that particular species, ensuring that the shapes are preserved and perhaps enhanced over evolutionary time - a secondary sexual characteristic.
The researchers believe this is the first time that secondary sexual selection has been shown to have shaped the evolution of any marine mammal.
Well-studied secondary sexual characteristics include the antlers of deer.
The skull of a male Blainville's beaked whale shows the teeth
They are much more prominent on males, indicate strength, and are used for fighting.
Natacha Aguilar, who has been studying beaked whales in the Canary Islands for a decade, agreed that the theory made sense.
"Little is known about the social structure of beaked whales, but at least some species live in harem-like groups where one adult male accompanies a group with females and juveniles," she told BBC News.
"In this context, the male will need to fight with other males for access to a female group, and to be attractive to the females for them to choose him.
"These parameters all favour the hypothesis of sexual selection as a force in shaping the most striking sexual dimorphism characteristic in beaked whales, the tusks."
Dr Aguilar's group, from the University of La Laguna in Tenerife, has pioneered the use of photographic identification for beaked whales, and has shown that individuals can be identified by the scars they bear.
However, much about these elusive animals remains a mystery, including how many there are in the seas, where they live, and exactly how many species exist.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.