Mr Barak has demanded that the Palestinian Authority rearrest all militants it has released since the violence erupted at the end of September.
The extremist group Islamic Jihad said it had carried out the attack, which came only hours after the Israelis and Palestinians said they had reached an understanding to end weeks of bloodshed.
Palestinian leader Yasser Arafat said the Palestinian Authority was completely against the bombing.
The BBC's Hilary Andersson in Jerusalem says the blast could not have come at a worse time, and it is now unclear whether Israeli Prime Minister Ehud Barak and Mr Arafat will make the truce announcement as planned.
Israeli police have launched a massive search for the bombers.
Black plumes of smoke rose above the city shortly after the attack, as medical teams rushed to the area. At least 10 people were injured.
Call for rearrest of militants
Israeli TV has reported that one of the two killed was the daughter of Yitzhak Levy, the leader of Israel's right-wing National Religious Party, which is closely associated with Jewish settlers living in the West Bank and Gaza strip.
The 34-year-old man was named as Hanan Levy from Jerusalem, but the television station added that the two were not related.
Mr Barak has demanded that the Palestinian Authority rearrest all militants it has released since the violence erupted at the end of September.
"The attack in the Mahane Yehuda market is another grave incident in the violence that has been forced upon us and which stems from a loosening of the reins by the Palestinian Authority and the release of Hamas and Islamic Jihad prisoners," Mr Barak said in a statement.
A senior Palestinian negotiator, Marwan Kanafani, rejected accusations that the releases meant the Palestinian Authority was responsible for the bombing.
There have been confrontations throughout Thursday between Israeli troops and Palestinians in the West Bank and Gaza.
In the worst incidents two teenage Palestinians were shot dead in the West Bank - one in the village of Himza, north-east of Jerusalem, and the other in Al-Khader near Bethlehem, where at least six people died on Wednesday.
More gunfire was also reported around Beit Jala, where Palestinian gunmen were said to have opened fire again on the Jewish settlement of Gilo.
The current violence began on 28 September when Palestinians frustrated with the direction of the peace process rioted, after Israeli hardliner Ariel Sharon visited the al-Aqsa mosque compound in east Jerusalem, a site which is also holy to Jews.
Since then, more than 170 people have been killed - most of them Arabs.
Israelis had been bracing themselves for suicide bombings, a tactic used to devastating effect in the past.