Shia Muslims traditionally mark 40 days after a death with a ceremony called the "arbayeen".
Mr Mousavi and another opposition leader, Mehdi Karroubi, had asked the interior ministry for permission to hold a memorial service in the Grand Mossala, according to an aide to Mr Mousavi, but permission was denied.
So the opposition leaders said they would join Neda's family at her graveside at the Behesht-e Zahra cemetery.
"Hundreds have gathered around Neda Agha Soltan's grave to mourn her death and other victims' deaths... police arrested some of them ... dozens of riot police also arrived and are trying to disperse the crowd," a witness told Reuters.
Mr Mousavi was surrounded by police shortly after he arrived, witnesses said.
Jon Leyne BBC Tehran correspondent
It's an ominous moment for the government. Those who run the Islamic Republic know only too well the cycle of protests, killings, then Arbayeen ceremonies from 1979, a cycle that helped bring them to power. They must fear history repeating itself, as similar anniversaries approach 40 days after protesters killed in the recent protests.
For the opposition, it's an opportunity to take to the streets despite the ban on protests. They could argue that there is no ban on religious ceremonies, though the police and government militia members attacking them with clubs and teargas obviously disagree.
The protests now are not remotely on the scale of the hundreds of thousands, perhaps millions, of demonstrators who came onto the streets immediately after the election. But they are happening despite repeated threats and intimidation, and they are keeping up the pressure on the government.
"Mousavi was not allowed to recite the Koran verses said on such occasions and he was immediately surrounded by anti-riot police who led him to his car," one person told AFP.
Some people in the crowd threw stones and chanted anti-Ahmadinejad slogans, reports said, as security personnel with batons charged at them.
One man told the BBC there were about 3,000 people there. Seven or eight men used professional cameras to film the protesters, he said.
Shortly afterwards hundreds more demonstrators were said to have gathered at the Grand Mossala.
Police again used tear gas and batons to break up the crowds, some of whom set rubbish bins on fire, witnesses said.
Iranian authorities banned all opposition protests following post-election violence.
And, reports the BBC's Jon Leyne, the authorities are particularly sensitive about these "arbayeen'' turning into political demonstrations.
That is exactly what happened during the Islamic Revolution 30 years ago in a cycle that helped lead to the downfall of the Shah, our correspondent says.
The US criticised the use of violence against the protesters.
"I think it's particularly disturbing to see security forces use force to break up a graveside demonstration," State Department spokesman Ian Kelly said.
In a separate development, a lawmaker said prominent opposition campaigner Saeed Hajjarian had been moved from jail to a government-owned residential complex.
Mr Hajjarian, an advisor to former reformist President Mohammad Khatami, was detained shortly after the polls.
The judiciary had said he would be released on Wednesday, but lawmaker Kazem Jalali told Mehr news agency that he would instead be kept at a government site.
Campaigners had called for his release, arguing his health had deteriorated badly because of harsh treatment in prison.
On Tuesday, officials said about 140 people detained during the protests had been released from Evin prison.
But about 200 others, accused of more serious crimes, remain in jail.
Tehran's public prosecutor's office has announced that the first trials of "rioters" will begin on Saturday, the official Iranian news agency Irna reported.
On Wednesday, US Secretary of State Hillary Clinton said she deplored the way the Iranian government was treating those it had imprisoned after the violence and urged authorities to release political detainees.
Mr Ahmadinejad is to be officially approved as Iranian president on 3 August.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.