A professor in the US is thought to have become one of the first Muslim women to lead mixed Friday prayers.
The congregation was evenly split between men and women
More than 100 men and women attended the service and sermon given by Amina Wadud, professor of Islamic studies at Virginia Commonwealth University.
The location was moved to an Anglican Church building in New York after mosques refused to host the event.
The service has been criticised by a number of Muslim leaders, who say it goes against Islamic doctrine.
"The issue of gender equality is a very important one in Islam, and Muslims have unfortunately used highly restrictive interpretations of history to move backward," Ms Wadud said before the service started.
"With this prayer service we are moving forward. This single act is symbolic of the possibilities within Islam."
Those who attended were said to be evenly divided between men and women. Most women wore the traditional Muslim headscarf and robes.
Some 15 protesters gathered outside the Synod House of the Cathedral of St John the Divine, where the prayers took place. One carried a placard calling for Allah's curse to be upon one of the event's organisers.
"She is tarnishing the whole Islamic faith. If this was an Islamic state, this woman would be hanged," one man, Nussrah, told the Associated Press.
The BBC Middle East correspondent says the controversy has meant Ms Wadud is getting prominent coverage on Arabic television networks.
The service was organised by a group of activists, journalists and scholars who hoped to encourage discussion about the centuries-old tradition of separating men and women during congregational prayer, and reserving the role of prayer leader, or imam, for men.
One organiser, Asra Q Nomani, said they would challenge the "second-class" status of women in Muslim spiritual life.
Mosques reportedly refused to host Amina Wadud
"We are taking actions that no-one else would have dared to think about before," she told The New York Times. "Nobody cared that we didn't have a place in the faith."
However, the sheik of Cairo's Al-Azhar mosque, one of the world's top Islamic institutions, said Islam did not allow for women to preach to men.
"When she leads men in prayer... it's not proper for them to look at the woman whose body is in front of them," Sayed Tantawi wrote in a column for the Egyptian newspaper Al-Ahram.
In New York, Aisha al-Adawiya, head of Women in Islam, said she feared a "backlash".
The prayer service was moved after it was rejected by three mosques and an art gallery venue received a bomb threat, the AP reported.