Palestinian Prime Minister Ahmed Qurei has announced his resignation, saying Hamas must form the next government following the parliamentary elections.
Hamas' strong showing has caused consternation in Israel and the US
It comes as the militant Islamic group appeared to be heading for a shock win.
With counting still under way, officials from the ruling Fatah party said Hamas had won a majority. Official results are due at 1900 (1700 GMT).
Israel, the US and the EU consider Hamas a terrorist group and have said they do not want to deal with it.
Mr Qurei has gone to see Palestinian leader Mahmoud Abbas to hand in his formal resignation.
Hamas says it will ask Fatah to join a coalition, but Fatah officials say they will not sit in government with Hamas.
Hours before official results were due to be released, Fatah officials privately admitted that Hamas had won.
Hamas claimed it had won at least 70 seats in the 132-member parliament, while EU election observer Richard Howitt told the BBC he had been informed that Hamas could have won up to 80 seats.
The BBC's Jon Leyne in Jerusalem says there is no doubt that the Hamas showing has transformed the Palestinian political arena.
For decades, Fatah - the party founded by the late Yasser Arafat - has totally dominated electoral politics, but that time is over, he says.
Hamas is also now a major power and it will enter parliament still committed to its armed confrontation with Israel, our correspondent adds.
With victory looming, senior Hamas official Ismail Haniya said the group would discuss political partnership with Fatah.
"This issue is going to be one of our priorities in the near future," the Associated Press news agency quoted him as saying.
But senior Fatah official Jibril Rajoub told Reuters: "Fatah rejects participating in a government formed by Hamas.
"Hamas has to take up its responsibilities. Fatah will act as a responsible opposition."
Another Hamas official, Mushir al-Masri, warned that Hamas would not hold peace talks with Israel.
"Negotiations with Israel is not on our agenda," he said.
"Recognising Israel is not on the agenda either now."
The likelihood of a resounding victory for Hamas - which is committed to the destruction of Israel - sent shockwaves though the Jewish state.
Speaking on election night, acting Israeli Prime Minister Ehud Olmert said Israel could not deal with a Palestinian Authority which included Hamas.
"Israel can't accept a situation in which Hamas, in its present form as a terror group calling for the destruction of Israel, will be part of the Palestinian Authority without disarming," Mr Olmert's office reported him as saying.
"I won't hold negotiations with a government that does not stick to its most basic obligation of fighting terror."
The outcome of the Palestinian election is the biggest challenge facing Mr Olmert since he took over from Israeli Prime Minister Ariel Sharon, who remains in a coma following a massive stroke on 4 January.
Call to disarm
Mr Haniya called for the US to "respect ... the will of the Palestinian people and the result of the ballot," AFP news agency reported.
Washington has not yet commented on the emerging results, but US President George W Bush warned on Wednesday he could not sanction a government led by Hamas in its present form.
"A political party, in order to be viable, is one that professes peace, in my judgement, in order that it will keep the peace," Mr Bush told the Wall Street Journal.
UN Secretary General Kofi Annan said the Palestinian elections were "an important step toward the achievement of a Palestinian state," UN spokeswoman Marie Okabe said.
But the UN chief warned that "any group that wishes to participate in the democratic process should ultimately disarm because to carry weapons and participate in a democratic process and sit in parliament - there is a fundamental contradiction".
The European Union - the biggest provider of aid to the Palestinian Authority - said it would work with any Palestinian government which renounced violence.
"We are happy to work with any government if that government is prepared to work by peaceful means," said European External Relations Commissioner Benita Ferrero-Waldner.