Exit polls suggest that the ruling Fatah party has won a narrow victory in the first Palestinian parliamentary elections for a decade.
Israel is worried by Hamas' participation in the elections
The first official exit poll suggests Fatah took more than 46%, compared to 39.5% for Islamic militant group Hamas.
The official results may not be announced for several more days.
The BBC's James Reynolds in Ramallah says whatever the final outcome, it is clear Fatah is no longer the single dominant force in Palestinian politics.
That is because Hamas has now managed to convert its own long-standing popular support into a formal political voice, our correspondent says.
Correspondents say Fatah may find it hard to gain the support of smaller parties to form a majority, and may be forced to invite Hamas into a governing coalition.
Both parties said they would consider a coalition if there was no clear winner.
But Israeli acting Prime Minister Ehud Olmert, quoted by AFP news agency, said before exit polls were released that Israel could not allow Hamas in its current form to become part of the Palestinian Authority.
US President George W Bush also said Washington would not deal with Hamas unless it renounced a desire to destroy Israel.
"A political party, in order to be viable, is one that professes peace, in my judgment, in order that it will keep the peace," he said in an interview for the Wall Street Journal.
The BBC's Jonathan Beale in Washington says Hamas' electoral progress will cause serious problems for the US as it tries to persuade Israel to pursue a peaceful path with the Palestinians.
Hamas is banned as a terrorist organisation in Israel, the US and Europe.
It does not recognise Israel and has launched hundreds of attacks against its citizens.
Mr Abbas, however, said he was ready to start peace talks with Israel even if Hamas joined the government.
"We are partners with the Israelis. They don't have the right to choose their partner. But if they are seeking a Palestinian partner, this partner exists," he said.
Voting was brisk, with turnout at 73%, the election commission said.
In the West Bank, 70.6% voted, while in Gaza the figure was 76.8%.
Observers say the elections were peaceful, and one EU official said the vote was an example to the Arab world.
Palestinian police were out in force to guard the ballot boxes, while militants pledged not to disrupt the poll.
The atmosphere has been described as festive, with campaign flags, posters and hooting vehicles competing for attention around polling stations.
Some party activists in Gaza decorated their cars with red carnations as if for a wedding.
"We are so happy with this election festival," Mr Abbas, the Palestinian leader, said after voting in the West Bank city of Ramallah.
Corruption and incompetence
Nearly 1.5m Palestinians were eligible to vote at about 1,000 centres in the West Bank and Gaza Strip.
Hamas has accused Fatah of corruption and incompetence.
Samer Lulu, 29, said he voted for Hamas because he was tired of corruption.
"With religious people at least we will have our public money in clean hands," AP news agency quoted him as saying.
In East Jerusalem, where 100,000 Palestinians were eligible to vote, just 6,300 residents were allowed to cast their ballot inside the city, while the remainder had to travel outside Jerusalem's boundaries.
Voting took place at the central post office.
There, an unnamed 34-year-old woman said she voted for Fatah "because I remain faithful to Yasser Arafat," a badge of the late Palestinian leader pinned to her lapel, AFP news agency reported.
At one point, a group of right-wing Israeli MPs and their supporters tried to force their way into the building, but were blocked by Israeli police.