The EU has called on pro-reform parties in Serbia to join together to keep extreme nationalists out of power.
Vojislav Kostunica is being courted by the ultranationalist Radicals
Foreign policy chief Javier Solana said Brussels would back a government based on a strong European reform agenda.
The ultra-nationalist Serbian Radical Party (SRS) won more seats than any other in Sunday's election.
Correspondents say it has little hope of entering government now, but its chance could come if reformers fail to create an effective coalition.
"I appeal to all democratic forces to work together in order to ensure that a new government based on a clear and strong European agenda can be formed rapidly," Mr Solana said.
With about 95% of votes counted, the SRS party and the Socialist Party of Serbia, both headed by men facing war crimes charges at the UN tribunal in The Hague, were on track to get 103 seats - of a total of 250.
Four pro-reform parties were expected to get the other 147 seats.
However, correspondents said that the failure of three of these parties to work together in the outgoing government meant the omens are not good for a new coalition.
"If they do tear themselves apart again the Radicals will come back even stronger," the BBC's Matthew Price in Belgrade says.
A senior member of the Democratic Party, which dominated the outgoing government, also said that the fourth group now in parliament - headed by the veteran conservative Vuk Draskovic - had proven difficult to get along with in the past.
"The question hovering above all of us is how effective the government can be," outgoing deputy prime minister Zarko Korac told BBC World Service radio's Newshour programme.
Diplomats and correspondents have speculated that there could be new elections within the year.
On Monday the SRS offered to form a coalition with the party which came second in the polls, the Democratic Party of Serbia, headed by former president Vojislav Kostunica.
Both parties opposed the Dayton peace accords in Bosnia and the Nato intervention in Kosovo, and have criticised Western pressure on Serbia to co-operate with the war crimes tribunal in The Hague.
However, Mr Kostunica's party has previously said it would not work with the SRS leader, Vojislav Seselj, who is awaiting trial in The Hague.
Like Slobodan Milosevic, the head of the Socialist Party, Mr Seselj will not be able to represent his party in parliament, and it is unclear whether they will be formally allocated seats.
Our correspondent says the SRS will form a "formidable" opposition.
Together with the Socialist Party it will have more than enough seats to prevent reformers making changes to the constitution, which requires a two-thirds majority.
The Organisation for Security and Co-operation in Europe's representative in Serbia, Maurizio Massari, said he was "disappointed" by the result but doubted whether the success of Mr Seselj and Mr Milosevic would have "real impact" on Serbian political life.
Chief UN war crimes prosecutor Carla Del Ponte said she was not surprised by the SRS's success but that she was "used to having problems with Belgrade".
"I hope that the new Serbian authorities will not hinder our work and they will know how to show themselves to be reasonable," she said on Switzerland's Italian-language radio.
But Mr Solana said he was confident that the necessary basis existed for Serbia to re-launch its reforms and to make further progress towards EU membership.
The Democratic Party, the Democratic Party of Serbia and the forerunners of the G17 Plus party were members of the umbrella group which ousted Mr Milosevic from power three years ago, but they fell out with one another once in power.
Only Zoran Djindjic's Democratic Party remained in the outgoing government.
His assassination in March helped to precipitate Sunday's election, one year ahead of schedule.
The parties are expected to spend the next few days deciding on their own strategy, and to start coalition talks on 5 January.