The DUP had 32% of the entire first preference vote in 2004, this time it has fallen to 18.2%.
The DUP votes have gone to Mr Allister, who split from the DUP over its decision to enter government with Sinn Fein.
He was eliminated at the second stage with a total of 70,481 votes but indicated he would stand in North Antrim in the next general election - taking on the DUP in their heartland.
Mr Allister said he had attained 30% of the unionist vote, adding: "That is a remarkable achievement and shows the depth of feeling that there is among many unionists who refuse to be rolled over in the era of Sinn Fein rule."
At the second stage, the redistribution of the Alliance and the Green Party votes went mainly to the Ulster Conservatives and Unionists and the SDLP.
Mr Allister also got more than 4,000 of those transfers.
Veteran MEP Jim Nicholson, who stood on an Ulster Conservatives and Unionists ticket, said he was very pleased with his performance.
"We have done it, the Ulster Conservative and Unionist Party is back and we're back in style.
"Things are changing, they're changing our way. We're on our way back and that's good. And we feel good today."
In the past Mr Nicholson had always been elected behind the DUP.
Bairbre de Brun topped the poll in Northern Ireland
Sinn Fein's vote is down 0.3% on the last European poll in 2004; the Conservatives and Unionists are up 0.5%; the SDLP has increased its vote by 0.3% and the Green Party is up 2.4%.
The Alliance Party said it was their best result in any election since 1998.
BBC Northern Ireland political editor Mark Devenport said there were a number of reasons for the DUP's poor performance.
"The Westminster expenses story and doubts about Diane Dodds' TV performances clearly played into the DUP reversal.
"As the dominant party in the executive they were also always going to be in line to be hit by a protest vote.
"One of the most important factors is that this was the first election since the DUP did its deal with Sinn Fein. They were always going to lose a section of their "never, never, never" support base."
Turnout, confirmed at just below 43%, has fallen dramatically from the last election. It means that 488,891 people in Northern Ireland voted.
There were 4,319 rejected papers and 484,572 valid papers.
Three hundred people counted the papers at the King's Hall in south Belfast.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.