Zeevi's wife found him lying in front of their hotel room
The Popular Front for the Liberation of Palestine (PFLP) has said it carried out the assassination in revenge for the killing of its leader Abu Ali Mustafa, by Israeli forces in August.
Following the shooting of Mr Zeevi, Israeli Prime Minister Ariel Sharon warned that a "new era" had begun, vowing things could never be the same, Israeli public radio said.
'No pulse'
Mr Zeevi was rushed to the city's Hadassah Hospital, with Israel army radio quoting emergency services members as saying he had no pulse on arrival.
Mr Zeevi died at 1000 (0800 GMT) about three hours after being shot, said Dr Shmuel Shapira, deputy director of Hadassah Hospital.
"After prolonged efforts to revive him... the doctors saw that all hope had totally ended," Dr Shapira said.
Mr Zeevi was the head of the right-wing National Union Party, formerly the Moledet Party, and resigned from Prime Minister Ariel Sharon's government on Monday in protest at Israel's military pull-out from Hebron.
His resignation had been due to take effect at 1300 (1100 GMT) on Wednesday.
No bodyguards
Mr Zeevi had been in the dining room of the hotel with his wife when he made his way up to their room on the eighth floor, police said.
Once there he was approached by two gunmen who shot him several times at close range, including twice in the head.
According to the BBC's James Reynolds in Jerusalem Mr Zeevi had no bodyguards with him at the time and the gunmen were able to escape.
When his wife followed him up to their room ten minutes later she found him lying in a critical condition in front of the door, police said.
An American tourist, David Hocking, who was also staying in the hotel, ran into the corridor after hearing Mr Zeevi's wife screaming.
"I saw her kneeling over him and he had obviously been shot, blood was everywhere," he said.
Mr Hocking said that he had not heard any shots fired and the police are investigating whether a silencer was used.
Condemnation
The hotel is in an area of east Jerusalem surrounded by Palestinian villages where a number of attacks have occurred in the past.
Mr Sharon is holding an emergency meeting with his top security and cabinet officials to discuss the assassination.
Security for the Israeli Cabinet has been beefed up in recent months, with some ministers saying that they had received extra body guards, but several of them, including Mr Zeevi, were reported to have refused the new security measures.
The Palestinian Authority has condemned the killing, but urged Israel to halt its policy of killing Palestinians.
"We feel sorry about this assassination. We reject all forms of political assassinations. We want to put an end to this vicious cycle of killing," Cabinet Minister Yasser Abed Rabbo said.
Revenge
The radical PFLP group has claimed responsibility for trying to kill Mr Zeevi saying, "we vowed to avenge the killing of leader Abu Ali Mustafa and we fulfilled our promise."
Abu Ali Mustafa was killed by Israeli forces in August.
As news broke of the shooting broke, dozens of Palestinians in the Ain El-Hilweh refugee camp in south Lebanon rushed into the street carrying pictures of Mustafa and dancing.
A senior PFLP official, Rabah Muhana, told the BBC's Kylie Morris in Gaza that Mr Zeevi was also targeted for his right-wing views.
"The Israelis killed one prominent leader and Mr Zeevi is one of those who have very, very right-wing points of view on discrimination - he wants to deport Palestinians and he is with the most severe terrorism against the Palestinians," he said.
Mr Zeevi was one of the most controversial politicians in Israel. He had repeatedly called for Arabs to be transferred out of the state and is notorious for using the line: "Let the Arabs go back to Mecca".
Mr Muhana also warned that that the matter will not end with the end of Mr Zeevi, saying that Israelis would continue to pay the price of Palestinian anger unless they ensured a real and stable peace.