Babies may be at risk from pollutants picked up by their mothers
However, scientists believe the harm can be partly offset by ensuring that young children are given a stimulating home environment.
Although they are now banned, these chemicals were once widely used in industry as coolants and lubricants.
They are still being leaked into the environment from old electrical equipment.
Once there, they do not readily break down, and pose a long-term threat.
They are taken up by small organisms and fish, and thus enter the food chain.
Exposure
Dr Gerhard Winneke and colleagues from Heinrich-Heine University, Düsseldorf, Germany, aimed to establish whether exposure to PCBs had an impact on a baby's brain, and if so, whether damage was only inflicted before birth, or continued when the child was breastfed.
The researchers studied 171 healthy babies and their mothers.
They measured their development at the age of seven months, and again at 18, 30 and 42 months.
PCB exposure of newborn babies was measured by analysing samples of umbilical cord blood and breast milk. At 42 months, the children's blood levels of PCBs were checked.
Writing in The Lancet medical journal, the researchers say they found that babies whose mothers had high concentrations of PCBs in their breast milk were more likely to exhibit low levels of development - both physically and mentally.
The difference was particularly marked from the age of 30 months onwards.
However, those children who had a stimulating home environment were less likely to be adversely affected.
Post-natal effect
Dr Winneke told BBC News Online: "Everybody thought that, if anything, PCBs had pre-natal effect, but we have shown that they can also produce post-natal effects.
"Perhaps this is not surprising as the amount of PCB to which a newborn is exposed is much higher than that which acts on a foetus."
Dr Winneke said steps should be taken to reduce the background level of PCBs in the environment.
He also suggested that paediatricians should reassess the advisability of recommending unlimited breastfeeding.
However, most experts believe that breastfeeding is by far the best way to feed a baby.
It is thought that PCBs might slow down development by reducing hormone production by the thyroid - a gland closely associated with regulating the development process.
Mike Childs, senior campaigner at the environmental pressure group Friends of the Earth said the research provided further proof of the need to tackle pollution.
He said: "We must stop factories releasing chemicals which build-up in people's bodies and develop a strategy to eliminate the release of carcinogens and chemicals where no health based air quality standards exist."