Research on rhesus monkeys in the US found the infants of those that were given cocaine had half the number of cells in a key part of the brain than those who were not.
The study's authors described their findings as a "warning sign" of the drug's effects on humans.
But British experts said it was difficult to ascertain what the effects on human babies actually was.
They took eight pregnant monkeys and gave half of them 20 milligrammes of cocaine per kilogram of weight every day during the second trimester.
That period of the pregnancy was chosen because it is the time when the cerebral cortex is formed.
It largely controls brain functions such as thought, reasoning, memory, sensation and voluntary muscle movement.
They then studied the brains of the babies born to the monkeys before putting them to sleep at three years of age so that the scientists could analyse their brains.
Brain analysis
The study found those monkeys whose mothers had been given the drugs had 60% fewer neurons - brain cells - than those who had not.
And their cerebral cortex was about 20% smaller.
Dr Lidow and his team found the cocaine actually stopped the brain cells reaching the cerebral cortex, something that has also been observed in schizophrenic patients.
He said the study showed many of the key cells stayed in the white matter under the cortex.
He said: "This is the first study that clearly shows the possibility that cocaine may affect the brain structure. It shows that it could happen."
The researchers say the fact the study was carried out on monkeys meant they could isolate the effects of the drug.
In humans, they say, cocaine's effect is difficult to study because of other factors such as the use of other drugs and poor diet.
Implications for humans
Dr Lidow said: "Monkeys are the closest you could get to humans."
But he admitted further research was needed to see exactly how humans would be affected.
Writing in the Journal of Comparative Neurology, Dr Barry Kosofsky from Massachusetts General Hospital and Dr Steven Hyman from the National Institute of Mental Health said the "evolutionary closeness" of monkeys and humans meant that the findings of this study were likely to be transferable.
"We certainly have enough information to make it a priority to identify and treat addicted individuals who are pregnant and to make serious ameliorative efforts aimed at children," they said.
Leading drugs charity Drugscope said the issue of drug use in pregnancy was a very emotive issue.
"Panic stories have often exaggerated the damage done to babies whilst ignoring the fact that most of the mothers were living in very poor and deprived circumstances, factors which themselves are implicated in having a difficult pregnancy and complications in childbirth and for newly born babies.
"The danger of being judgmental about drug-using pregnant women is that they will be reluctant to seek out the medical help they and their babies need."