Most hate crimes motivated by racism and prejudice are not reported because the victims "lack confidence" in the police, a charity study has indicated.
Hate victims can feel isolated within their communities
Victim Support found the 107 hate victims they interviewed also feared going to court and revenge attacks.
Of those who did report the crime, only one in five felt they were well supported by the police, it said.
An Association of Chief Police Officers spokesman said hate crime investigations were given top priority.
Paul Fawcett, of Victim Support, said: "Publishing the report highlighted the invisibility of the problem.
"Hate crime is a very clear indicator of a destabilised society. We have to be mindful of not demonising people."
The report, entitled Crime and Prejudice, focused on the experiences and support needs of people who had suffered attacks because of their ethnic origin or sexual orientation.
It said the research revealed evidence of police taking a "there is nothing we can do" approach towards so-called low-level harassment.
The victims wanted the police to be "more communicative and "more sympathetic and to take hate crime seriously".
People who have suffered race hate attacks need special support
Another common complaint was about lack of action by police, either to help solve problems or to pursue the perpetrator, it said.
Officers were also criticised for failing to keep victims fully informed about the progress of cases.
Mr Fawcett added: "Victims of hate crime can feel isolated in the community they live in, for example, if they were targeted because they are lesbian or gay, or they can feel isolated by language or culture if they have come to England from another country.
"The findings are worrying and disappointing. We are disappointed about the level of resources provided to deal with this hidden problem."
But where police had a specific brief to tackle hate crime there had been some success, he added.
The report noted that, where victims were dealt with by specialist police officers, these were seen as the most helpful source of support.
The researchers called for greater investment in the promotion of services to at-risk communities and greater understanding from the police and Crown Prosecution Service when responding to the needs of victims.
It also recommended that hate crime should be recorded more accurately.
The Home Office said it wanted to "see more progress in investigating and prosecuting hate crimes".
A spokesman said: "We are committed to ending hate crime in all its forms. Failure to tackle hate crimes undermines confidence in the criminal justice system and can bring fear to whole communities.
"We want people to have confidence that when they report crimes, action will be taken."
He said the Home Office was looking at introducing schemes to make reporting of hate crimes easier for victims.
Ideas included third-party reporting schemes and a 24-hour help line dedicated to reporting hate crimes.
Peter Fahy, of the Association of Chief Police Officers, said: "The detection rate for hate crimes is higher than crime in general and there has been a significant rise in the number of successful prosecutions for racially aggravated and religiously aggravated offences.
"Hate crime investigations are given high priority and the police service does not apologise for this."
The study was part of a £100,000 research programme funded by Co-operative Insurance (CIS).