Acting Israeli Prime Minister Ehud Olmert has declared victory in the election, vowing to push ahead with plans to define Israel's final borders.
Ariel Sharon remains a powerful presence within Kadima
His Kadima party, founded barely four months ago by now coma-stricken Ariel Sharon, won but by less than predicted.
It looks set to take 28 of the 120 seats in parliament. Likud, winner of the last election, was beaten into fifth place with just 11 seats.
Palestinians urged Mr Olmert not to set Israel's borders unilaterally.
The Kadima leader told his party he was prepared to hold peace talks with the Palestinians but would act alone if he had to, to establish permanent borders for Israel by 2010.
But with his margin of victory much less decisive than the party had hoped, he could have trouble forming and maintaining a stable coalition, say correspondents.
Kadima is expected to enter into coalition talks with second-placed Labour and other smaller parties.
Israel's President Moshe Katsav said he would start talking to party leaders next week. A Kadima official told the Haaretz newspaper he expected a coalition to emerge after the Passover holiday, in about three weeks' time.
RESULTS - 99% COUNTED
1. Kadima: 28 seats, centrist
2. Labour: 20 seats, centre-left
3. Shas: 13 seats, ultra-orthodox
4. Pensioners: 7 seats, single-issue
5. Torah Judaism: 6 seats, ultra-orthodox
6. Meretz: 4 seats, left-wing
7. Israel Beitenu: 12 seats, Russian emigres, far-right
8. Likud: 11 seats, right-wing
9. Arab parties: 10 seats
10. National Union/Religious: 9 seats, far-right, settlers
Standing in front of a massive picture of Ariel Sharon, Mr Olmert paid tribute to his stricken predecessor before laying out his plans for his four-year term in office.
"In the coming period we will move to set the final borders of the state of Israel, a Jewish state with a Jewish majority," Mr Olmert told Kadima members in what was effectively his victory speech.
"We will try to achieve this in an agreement with the Palestinians. This is our hope and prayer."
He told the Palestinian leadership: "We are ready to compromise, to give up parts of the beloved Land of Israel... and evacuate, under great pain, Jews living there, in order to create the conditions that will enable you to fulfil your dream and live alongside us."
But he said it was time for the Palestinians to "relate to the existence of the state of Israel, to accept only part of their dream, to stop terror, to accept democracy and accept compromise and peace with us".
The BBC's Jeremy Bowen in Jerusalem says Mr Olmert's task will not be easy.
He will face stiff opposition from Israel's settler movement, will have to convince the US to back his plan, and will not be able to ignore a Hamas-led Palestinian government, our correspondent says.
Palestinian leader Mahmoud Abbas said the "result was expected", but that Mr Olmert should return to the internationally-backed roadmap peace plan and "abandon his unilateral plans to fix the borders".
The new Palestinian cabinet, led by Islamic militant group Hamas, will be sworn in on Wednesday afternoon. It may not be long after that that the Palestinians' relationship with Israel sinks to new lows, says the BBC's Alan Johnston in Gaza.
Voter turnout in the Israeli election was 62.3%, a record low.
With 99% of ballots counted, Kadima has won 21.8% or 28 seats, with the centre-left Labour party coming second with 20 seats, a 15.1% share.
Under Israel's complex proportional representation, the exact number of seats may change as the final votes are redistributed.
The right-wing former ruling party, Likud, is trailing with just 11 seats - behind the ultra-Orthodox Shas, with 13, and the far-right Yisrael Beitenu party, which proposes forcibly transferring Arab populations inside Israel to Palestinian territory, with 12.
Likud leader Binyamin Netanyahu admitted they had "suffered a tough blow" but vowed to rebuild the party.
Since the creation of Israel in 1948, the country has been governed either by the Labour or Likud parties, so a Kadima victory is historic.
Kadima, which means "forward" in Hebrew, was founded by Mr Sharon last year after he left Likud amid bitter rows over his withdrawal of settlers and troops from the Gaza Strip.
Mr Sharon suffered a stroke and fell into a coma in January.
Israel occupied Gaza and the West Bank, including east Jerusalem in the 1967 war. Its settlements are considered illegal under international law, although Israel rejects that.