The charity is calling for greater awareness of warning signs of neglect
A quarter of adults in the UK have worried that a child they know may be neglected, but over a third did not act on their concerns, a poll suggests.
The charity Action for Children said it commissioned the survey of 1,038 people to show the difficulty of identifying and preventing the neglect of children.
Neglect can include children being unloved, underfed or badly clothed.
The charity said it is the most common abuse, accounting for 45% of those on England's child protection plans.
An Action for Children spokeswoman said neglect can be harder to recognise, and has not been as high profile as other forms of child abuse, even though it is one of the most common ways in which children are mistreated.
She said this is because neglect is often a symptom of other long-term and complex problems in a family rather than an easily recognisable one-off event.
As a result, she said, it can be hard for people around the family to know the right time to do something and feel comfortable and supported in acting on their instincts.
The results of the ICM survey of more than 1,000 adults and parents in the UK included:
- 16% of adults said they did not tell anyone because they were frightened of repercussions
- 15% said they did not tell anyone because it was not any of their business
- 11% would tell a neighbour, relative or friend first rather than social services or the Police
- 15% said that a lack of proof prevented them from doing anything
- 23% said they did not think they had enough information about who to ask for help
The charity reported that in 2008 in England alone neglect was the reason why 45% of children were on the child protection register, compared to 15% for physical abuse, 7% for sexual abuse and 25% for emotional abuse.
The Action for Children spokeswoman told the BBC: "Our advice would always be if they have a serious concern, go direct to police or children's services.
"With neglect, it's never a one-off event, it's always a series."
"We would always say if there is enough concern and there is enough proof, they should probably step in and they are well within their rights to."
She said warning signs might include children coming into school who are hungry or with dirty clothes, or if parents are rarely seen at parents' evenings and other points of contact.
She said the charity was now asking the government to raise awareness of what constitutes neglect so people know it when they see it.
A Department for Children Schools and Families (DCSF) spokesman said keeping children safe was their "top priority".
He told the BBC: "We have put in place a much stronger framework for tackling child abuse and neglect so that children and young people are at the centre of everything we do, and everything local services do.
"Following the recent Baby P case, Children's Secretary Ed Balls has asked Lord Laming to produce a progress report on how child protection arrangements are being implemented systematically around the country and to identify any barriers to that."
The spokesman added that child safety was "everyone's responsibility" and that although government and local agencies have an important role, they "cannot protect children alone".
He said: "If anyone has concerns about a child's welfare they should report their concerns to their local authority children's services... If anyone thinks a child is in immediate danger they should call 999."