Ensuring babies born weighing very little are given mental stimulation can stave off the danger of them having language and learning problems in later life, they say.
Around two out of three babies born weighing less than a kilogram (2.2lbs) will suffer some degree of disability, usually as a result of lack of oxygen or brain haemorrhage.
But the team at Mater Children's Hospital, in Brisbane, Australia, said brain development may be held back by parents not playing an active role in stimulating their children's minds.
They studied 352 infants born in Queensland and found that very small children were also at greater risk of maltreatment, reports New Scientist magazine.
Babies born to parents who were later notified to the health authorities for neglect had a significantly greater reduction in their physical and mental development than could be explained by medical complications alone.
Explosion
Lane Strathearn, a paediatric researcher at the hospital, who presented his findings to the Perinatal Society of Australia and New Zealand, said: "There is an explosion in the number of neuronal connections in the first two years of life, and that may be reduced if the infants get insufficient stimulation."
Professor Robin Murray, of the Institute of Psychiatry in the UK, told BBC News Online that parental input into children's development could have an effect, but children born very premature had massive obstacles to overcome.
He said: "It depends how severe complications at birth were. There is quite a bit of evidence that if you are born between 32 and 40 weeks, then the interaction with your parents can have a lot of impact on how things develop.
"If you are born very pre-term, the nature of the family does have an effect, but the more severe the insult to the brain, the less it would be affected."
Babies born before the 32nd week of pregnancy were more likely to have structural abnormalities in their brains, which would still be evident in later life, he said.
The worst possible combination was a baby born very small, very early to a mother who suffered from post-natal depression.