The conservatives were out of office for more than a decade
The leader of Greece's socialists has accepted defeat in Sunday's elections.
Exit polls gave conservative leader Costas Karamanlis' New Democracy party a lead of about 5% over the socialists led by George Papandreou.
The results - to be confirmed later on Sunday - mean Mr Papandreou's Pasok party has lost power after 11 years.
Mr Karamanlis will face the immediate task of ensuring Athens is ready for this August's Olympics, with many projects still to be completed.
"It is a big honour and also a big obligation," he said after news of his victory broke.
"The Olympic Games will be the best and safest... and Greece will show its modern face.
New Democracy supporters have been driving through the streets of Athens with the party flags waving from the windows, the BBC's Malcolm Brabant reports from the Greek capital.
He says crowds also gathered outside the conservatives' headquarters waiting to anoint Mr Karamanlis as the country's new prime minister.
'Vote for change'
With half of the actual ballots counted, New Democracy had won 46.7% compared with 40.4% for Pasok.
Voting in Greece is compulsory
Pasok in office since 1981 apart from 1990-1993 when New Democracy was in power
Mr Papandreou and Mr Karamanlis are members of the two dynasties which have dominated Greece since the 1950s
"New Democracy has won the elections. I wish Karamanlis success in his work for the good of Greece," said Mr Papandreou when the result became clear from the exit polls.
Provisional estimates say New Democracy will end up with 165 of parliament's 300 seats and the socialists will have only 117.
Greece has voted for change in a big way, our correspondent says.
He adds that the country is due to elect a new president next year, and if the conservatives fail to persuade 180 MPs to accept their candidate, then there will have to be new elections.
New Democracy must immediately tackle two tasks: the Olympics and negotiations aimed at reuniting Cyprus before it joins the European Union in May.
Much remains to be done to ensure everything will be ready for the Olympics in August.
Preparations are well behind schedule, and there are even rumours - vehemently denied by the Olympic Committee - that the Games might have to be cancelled.
A competitive Greece
Revellers celebrating the conservatives' victory on the streets of Athens said a change at the top had been long overdue.
A return to power for the Karamanlis dynasty looks imminent
"We had that regime for 20 years," one told the BBC.
"No matter if they were good or bad, they had to change... and I think that the attitude towards the people will change."
The man tipped to become the new finance minister, George Alogoskoufis, has said that New Democracy has been given a clear mandate for economic reform "to make Greece more competitive".
Mr Alogoskoufis has previously called for tax cuts, investment incentives and market deregulation.
He recently listed his priorities for development as
agriculture, tourism, shipping, energy production, transportation and telecoms.