The men had allegedly agreed to buy explosives from FBI agents posing as Islamic militants.
The four are charged with conspiracy to use weapons of mass destruction within the US and conspiracy to acquire and use anti-aircraft missiles, officials said.
The charges carry jail terms of between 25 years and life imprisonment.
A senior FBI official in New York said three were US citizens and one was from Haiti.
BBC defence and security correspondent Rob Watson says the case appears to be a classic sting operation against suspected home-grown militants rather than a plot with any links to known international terrorism.
Speaking outside the Riverdale Temple, one of the intended targets, New York City Mayor Michael Bloomberg praised the work of New York's police and the FBI.
The four have all made anti-Semitic statements, the police chief says
The alleged plot served as a reminder to New Yorkers to remain vigilant "at all times", the mayor said.
"The bottom line is that we have to be constantly vigilant and we have to constantly be sure that we have the best police department in the world, that they are well led and well trained."
Mr Kelly, the police commissioner, stressed that the arrests were the result of a lengthy operation and that despite the serious nature of the charges, no-one was ever actually put at risk.
According to prosecutors, the men planned to detonate cars packed with C-4 plastic explosives outside the Riverdale Temple and the Riverdale Jewish Center in the Bronx district of the city.
They also intended to target military planes at the New York Air National Guard base at Stewart Airport, 60 miles (85 km) north of New York City.
In their efforts to obtain weapons for the attack, the men dealt with an informant from the FBI, who is said to have provided the group "with an inactive missile and inert explosives."
"This was a very tightly-controlled operation but these individuals did place bombs - or what they thought were bombs - right in front of the building in which we are standing and the temple a few blocks away," Mr Kelly said.
Outlining the charges on Wednesday night, law enforcement officials said the group set up what they believed to be 30lbs (14kg) of explosives.
According to prosecutors, Mr Cromitie told an FBI informant in June 2008 that he was angry over the US-led war in Afghanistan.
The suspects allegedly wanted to attack National Guard planes
He "expressed an interest in 'doing something to America"'.
From October 2008, the informant began meeting him regularly along with the four others at a house in which the FBI had concealed video and audio equipment.
The group allegedly "expressed desire" to attack targets in New York and Mr Cromitie "asked the informant to supply surface-to-air guided missiles and explosives", prosecutors say.
In April 2009, the group agreed on the synagogues they intended to attack and proceeded to conduct surveillance, including taking photographs of the warplanes at the military base, prosecutors say.
Mr Cromitie allegedly pointed out Jews in the street, saying "if he had a gun, he would shoot each one in the head", according to the district attorney's statement.
According to the statement, he told the informant that attacking the Jewish community centre would be a "piece of cake".
He also said he would be interested in joining Jaish-e-Mohammed - a Pakistan-based group considered a terrorist organisation by Washington - "to do jihad".
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.