Vulnerable adults need legislation to guard them from abuse, in line with children and domestic violence victims, says charity Action on Elder Abuse.
Elderly women were most likely to be abused
Vulnerable adults include the elderly and people with mental health problems and learning disabilities.
The charity found of 639 abuse cases in nine English local authorities over six months, more than half involved the elderly, and two-thirds involved women.
The government said measures were planned to tackle the problem.
In previous research, Action on Elder Abuse had found there was no standard system which local authorities used when dealing with cases of abuse.
In this latest research, nine authorities - Essex, Hertfordshire, Brent, Redbridge, Liverpool, Hull and East Riding of Yorkshire, Poole, Bournemouth and Dorset - were given a framework to work to.
Just over 200 of the 639 cases related to abuse which had occurred in a person's own home, compared to 188 in care homes.
In 116 cases, the person carrying out the abuse - usually physical in nature - was working in an institution, such as a nursing home or hospital.
In the cases of the people who were abused at home, 65 involved a paid care worker, with abuse by family members other than a partner or carer making up the next largest group.
Thirty-four cases involved the main family carer.
However, of all 639 cases referred for investigation by local authorities, just five resulted in a criminal prosecution.
Action on Elder Abuse is now calling on the government to give abuse of vulnerable adults the same status as that of child protection and domestic violence.
It also wants national data collection introduced, and reporting requirements where people are referred for protection measures.
In addition, the charity wants performance measures so that the work agencies carry out can be assessed against standards.
Daniel Blake, the report author, said: "We applaud the measures that the government is introducing.
"But there is still the fact that there is no legislation - and that is an anomaly when you look at the areas of child protection and domestic violence.
"We want to see equality for vulnerable adults."
Mr Blake said that, without legislation, authorities and agencies were not compelled to put measures in place which would protect vulnerable adults.
David Congdon, of learning disability charity Mencap, said: "This report highlights the desperate need for the protection of vulnerable adults from abuse.
"It is sickening to think that in the 21st Century people with a learning disability are still victims of such attacks.
"We support the report's calls for the abuse of vulnerable adults to be given the same status as that of child protection and domestic violence."
Care Services Minister Liam Byrne said: "The results of this project are shocking - too many of our vulnerable and older people are being subjected to attacks or are harmed as a result of neglect, and too few people are being brought to justice for it.
"We are working on a range of measures to tackle abuse of vulnerable adults."
The government plans a committee of "dignity guardians" - agencies and charities to advise on how best to protect the diginity of vulnerable people.
There will also be a centralised vetting and barring system, registration of care workers and increased spot checks of care homes.
But Mr Byrne added: "It is only together that we can tackle this shame in our society and make abuse of our older and vulnerable people a thing of the past."
Are laws needed to protect older people from abuse? How can more protection be given to vulnerable adults? Send us your views and questions using the form below.
The BBC may edit your comments and not all emails will be published. Your comments may be published on any BBC media worldwide.