It had concluded the passports used were copied from genuine British passports when handed over for inspection to individuals linked to Israel, either in Israel or in other countries, he said.
"Given that this was a very sophisticated operation, in which high-quality forgeries were made, the government judges it is highly likely that the forgeries were made by a state intelligence service," he said.
"We have concluded that there are compelling reasons to believe Israel was responsible for the misuse of the British passports."
The incident "represents a profound disregard for the sovereignty of the United Kingdom" he said.
He said the fact that Israel was a friend added "insult to injury".
The BBC's Middle East editor Jeremy Bowen said the expulsion sent a "very clear message" of British disapproval.
"It is a very big step for a government like the British to expel one of the diplomats belonging to one of its important allies," he said.
Ron Prosor: "The relationship between Israel and the UK is of mutual importance"
It is believed 12 fake British passports were used in the plot to murder Mr Mabhouh - the founder of Hamas's military wing - in his hotel room in Dubai on 19 January.
The names and details on the UK passports used by eight of the 12 suspects belonged to British-Israeli citizens living in Israel - all of whom have denied involvement in Mr Mabhouh's murder.
Their passports had been copied and new photographs inserted.
Tim Franks, BBC News, Jerusalem
"Regret" is the official Israeli emotion, not anger, not surprise, and certainly not retribution.
There is a clear Israeli desire to dampen this argument down from one where it could damage the wider relationship.
"It doesn't look good," was the terse verdict of one former senior diplomat. Other sources suggested this was a "standard dance" Britain had to go through.
But this is not being said with a tremendous swagger. After all, this is not the first time Israel has been caught with its hand in a sweet jar full of British passports. On that occasion, more than 20 years ago, Israel promised not to repeat the offence. That promise appears to have expired.
Israel is already feeling the heat from the US for its continued building on occupied territory. The Israeli government will not, right now, want to fight on too many fronts.
Mr Miliband said there was no evidence to suggest that any of the 12 passport holders were anything other than "wholly innocent victims of identity theft".
He said he was amending the official travel advice to Israel to make British nationals aware of the risks of their passports being misused.
He said he had met Israeli foreign minister Avigdor Lieberman in Brussels on Monday and had made clear the UK's "determination to see that this affair should never be repeated".
He handed him a letter "seeking a formal assurance... that in the future the State of Israel would never be party to the misuse of British passports in such a way".
It is not the first time British passports have been misused by Israel. In 1987 the country was caught forging UK passports for an intelligence operation, and Israel promised it would not do it again.
On that occasion, eight British passports reckoned to be for Mossad agents were found in a bag in a West German telephone booth.
A spokesman for the Palestinian Hamas group said it welcomed the decision to expel the diplomat, but wanted international efforts to track down the killers stepped up.
Dubai police have used CCTV footage to identify 27 alleged members of the team that tracked and killed Mr Mabhouh.
Passports from France, Ireland, Germany and Australia were also forged in the operation, and they too are investigating Israel's role.
Dubai officials said they were "99% certain" that agents from Mossad were behind the killing.
Following his death, Mr Mabhouh's family said doctors who had examined him determined he had died after receiving a massive electric shock to the head. They also found evidence that he had been strangled.
Blood samples sent to a French laboratory confirmed he was killed by electric shock, after which the body was sent to Syria.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.