According to official preliminary results, the List Pim Fortuyn (LPF) has captured 26 of the 150 seats in parliament and may win a place in government.
Coalition talks are expected to begin later on Thursday.
Jan Peter Balkenende of the Christian Democrats is expected to become the next prime minister after his centre-right party won the largest number of seats, 43.
Click here for a graphic comparing seats in parliament 1998 and 2002
As expected, the governing centre-left coalition of interim Prime Minister Wim Kok - which has been in power for eight years - suffered a resounding defeat.
Mr Kok's Labour Party won 23 seats, as did its main coalition partner the liberal VVD - well down on their performance at the last general election four years ago.
Mr Kok acknowledged the scale of his party's defeat.
"The voters gave us a huge thrashing," he said. "The people of the Netherlands have made a different choice."
Labour Party parliamentary leader Ad Melkert announced his intention to step down following his party's dismal showing.
Long negotiations expected
Jan Peter Balkenende, the 46-year-old leader of the Christian Democrats, said he was "ready to take on the responsibility" of forming a coalition government.
He has not ruled out a right-wing coalition with LPF.
But the BBC's Geraldine Coughlan, in The Hague, says that a broader coalition involving Labour and the VVD - and excluding LPF - is also possible.
Because of the Dutch system of proportional representation, governments are always coalitions and often take many weeks to form.
Voter turnout in the election was high, with many apparently galvanised by Mr Fortuyn's killing by a gunman last week.
Mr Fortuyn's party has done particularly well in its heartland of Rotterdam.
But LPF spokesman Mat Herben told Reuters news agency that the death of Mr Fortuyn had cast a shadow over his party's results.
"It's a wonderful result but there is no real joy," he said. "Today we feel like orphans. We've lost our teacher."
Unprecedented
Mr Fortuyn had won widespread support with his controversial views which challenged the consensus politics of the Netherlands.
He called Islam a "backward" religion and demanded that the Dutch borders be closed to new immigrants. He also said foreigners living in the country should do more to integrate.
About one in 10 of the Dutch population of 16 million belongs to an ethnic minority community.
Election campaigning stopped after Mr Fortuyn's killing last week by a lone gunman outside a radio station.
Thousands turned out for his funeral in an outpouring of grief and anger unprecedented in Dutch society.
Police have charged animal rights activist, Volkert van der Graaf, 32, with the murder.
He has yet to make a statement to police and is expected to appear in court again on Thursday.