Some demonstrators threw stones at the police but the trouble was quickly defused.
The unions regard the austerity programme as a declaration of war against the working and middle classes, the BBC's Malcolm Brabant reports from the capital.
He says their resolve is strengthened by their belief that this crisis has been engineered by external forces, such as international speculators and European central bankers.
"It's a war against workers and we will answer with war, with constant struggles until this policy is overturned," said Christos Katsiotis, a union member affiliated to the Communist Party, at the Athens rally.
Others in the capital either see the cuts as necessary or argue that the strike is politically motivated.
"We have to implement the austerity measures, or the country will not be able to get out of this crisis," said Katerina, a private sector employee. "We have to pay for the mistakes of the past."
On Tuesday, Prime Minister George Papandreou's socialist government announced that it intends to raise the average retirement age from 61 to 63 by 2015 in a bid to save the cash-strapped pensions system.
The move comes on top of other planned austerity measures, including a public sector salary freeze and a hike in petrol prices, announced last week.
Deficit and debt
Further government measures include the non-replacement of departing civil servants, and tax collectors recovering billions of euros lost to tax evasion.
Mr Papandreou, who was in Paris on Wednesday for talks with President Nicolas Sarkozy, pledged to "take any necessary measures" to reduce Greece's deficit.
"The stability programme will be implemented in every measure," he said.
Mr Papandreou has already faced down a three-week protest by farmers demanding higher government subsidies.
These protesters referred to "The Pigs" - Portugal, Italy, Greece and Spain
Public sector workers will not be hit as hard as they have been in the Irish Republic, but they complain that some of the lowest paid will suffer while the rich dodge tax with impunity, says BBC Europe correspondent Jonny Dymond.
Financial markets around the world and politicians from across Europe will be watching the situation carefully, he reports from Athens.
Greece's deficit is, at 12.7%, more than four times higher than eurozone rules allow. Its debt is about 300bn euros ($419bn; £259bn).
The markets remain sceptical that Greece will be able to pay its debts and many investors believe the country will have to be bailed out.
The uncertainty has recently buffeted the euro and the problems have extended to Spain and Portugal, which are also struggling with their deficits.
The possibility of Greece or one of the other stricken countries being unable to pay its debts - and either needing an EU bailout or having to abandon the euro - has been called the biggest threat yet to the single currency.
Ahead of the talks between EU leaders in Brussels on Thursday, some business media reported that Germany is preparing to lead a possible bail-out, supported by France and other eurozone members.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.