Mr Netanyahu claimed victory, saying a Likud-led coalition would lead Israel.
"With God's help I will lead the next government," he told cheering crowds at Likud's Tel Aviv headquarters.
On the basis of the exit polls, analysts predict that Likud and various nationalist parties will between them control 65 of 120 seats in the Knesset.
BBC diplomatic correspondent Jonathan Marcus says the closeness of the forecasted result is in many ways the worst outcome for Israeli democracy, as it sets the scene for days and probably weeks of fractious political horse-trading.
Likud looks to have already sewn up a deal with the strongest of the Orthodox Jewish religious parties - Shas - ahead of the election, our correspondent adds.
Once the final results are in, President Shimon Peres will consult with party leaders to determine who among them stands the best chance of forming a coalition government, but he does not have to choose the leader of the largest party.
The chosen party leader has then up to 42 days to form a coalition. If the attempt fails, Mr Peres can ask another leader to assume the task.
Much will depend upon what the Yisrael Beitenu leader Avigdor Lieberman tells the president, and which way he leans may be decisive in determining who is the first to be asked to try to form a coalition government, our correspondent says.
Elections were called early after Ms Livni failed to form a new government following Prime Minister Ehud Olmert's decision to step down last year amid a corruption probe against him.
Mr Olmert will stay on as caretaker prime minister until a new government is formed.
The election has been dominated by security issues following Israel's offensive in the Gaza Strip controlled by the Palestinian militant group Hamas.
Although its rival in the West Bank, the Palestinian Authority, has not expressed preference for any candidate, senior negotiator Saeb Erakat expressed dismay right-wing parties that oppose the traditional land-for-peace formula had performed so well.
"It is obvious the Israelis have voted to paralyse the peace process," he said.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.