The number of defendants referred to the Crown Prosecution Service for racially motivated crimes rose by 13% last year.
Between April 2003 and April 2004, the CPS dealt with 4,728 defendants, prosecuting 3,616, or 76% - an increase of 2% on the previous year.
The annual racist incidents report included a full year's religiously aggravated crimes for the first time.
Half of the 44 victims' perceived or actual religion was Muslim.
Work to do
The majority of both the racially and religiously aggravated crimes were public order offences dealt with in the magistrates' court.
This was followed by assault, criminal damage and harassment.
Director of Public Prosecutions Ken Macdonald QC said he was reassured the conviction rate for racially aggravated offences remained high but said there was still "work to be done".
"In this report, witness difficulties accounted for 26% of dropped charges," he said.
"We have high hopes that the witness care units, which are being rolled out to all 42 criminal justice areas in England and Wales, will provide more dedicated care to witnesses to help them attend court for all types of case."
Where cases with an allegedly racial motive were dropped, 45% were because of insufficient evidence and 26% were because of "witness difficulties".
About 15% were not pursued on public interest grounds, such as cases where the defendant was already being dealt with for a more serious offence.
A total of 44 religiously aggravated offences were prosecuted, with the perceived or actual religion of the victim in 22 of the cases being Muslim.
In the remaining cases, the victims' religions included Christian - eight - and Jewish - five.
The overall conviction rate in courts was 77% on religiously aggravated charges.
The Anti-Terrorism Crime and Security Act 2001 created new religiously aggravated offences which were introduced at the end of that same year.
In common with sentencing for racially aggravated crimes, the Act also forces courts to deal more harshly with offenders being sentenced for any offence deemed to be religiously aggravated.
CPS director of equality and diversity Seamus Taylor said that while there were comparatively few religiously aggravated cases it was still "early days" for the new offence.
"To build awareness that we will prosecute this type of hate crime rigorously, we will build on our current work and deepen our engagement with all faith communities over the next year," he added.
"This will include engaging with the Muslim community which should contribute to raising awareness and building confidence in this area."