Under Serbia's electoral law at least 50% have to take part to make the poll valid.
A spokesman for the Centre for Free Elections and Democracy said the current Yugoslav President, Vojislav Kostunica, had won about two-thirds of the vote.
His only opponent was one of Yugoslavia's deputy prime ministers, Miroljub Labus, an ally of Serbian Prime Minister Zoran Djindjic.
The BBC's correspondent in Belgrade, Matthew Price, said that the low turnout shows that the people of Serbia are disillusioned by politics in their country.
Tired electorate
Our correspondent said that two years ago the people of Serbia voted in what they thought was a reform minded government, but that two years on they feel that the government has made little progress and are dissatisfied.
He said that the young people of Serbia are especially disillusioned, feeling that they did their campaigning for change when they demonstrated for former President Slobodan Milosevic to be toppled and now they are tired of politics.
Mr Kostunica has blamed the election failure on calls for a an "open boycott" by nationalist leader Vojislav Seselj and "subdued boycott" by Serbian Prime Minister Zoran Djindjic.
"No matter whether it was loud or subdued, the boycott is extremely anti-European," Mr Kostunica said.
Yugoslav Foreign Minister Goran Svilanovic said the failure of the vote would damage his country's international reputation.
Reforms under threat
Western diplomats say they are concerned that Serbia's politicians could spend months occupied with a fresh election, which will not take place for three or four months, instead of focussing on economic and political reforms.
There was a last-minute surge of voters at some polling stations, but apparently it was not enough to push the final turnout over 50%.
Correspondents say quarrelling among the pro-democracy leaders who ousted Slobodan Milosevic is also to blame.
"The constant squabbling and arguments among parties must affect voter willingness," said Zoran Nikolic, 47,as he cast his vote in Belgrade.
The post of Serbian president is not a powerful one on paper - the prime minister wields the real power in the land.
But the eventual winner of this election will have major influence over the issue of co-operation with the war crimes tribunal in The Hague, for example.
The new president will replace Milan Milutinovic, who has been indicted by the United Nations war crimes tribunal where Mr Milosevic is on trial.
Mr Milutinovic was deemed immune from prosecution until his term expires at the end of this year.