"The Department of Justice will pursue prosecution in federal court of the five individuals accused of conspiring to commit the 9/11 attacks," US Attorney General Eric Holder told a news conference.
"I fully expect to direct prosecutors to seek the death penalty against each of the alleged 9/11 conspirators."
But Republican leaders immediately criticised the move.
Senate Republican leader Mitch McConnell described it as "a step backwards for the security of our country" that "puts Americans unnecessarily at risk".
Former President George W Bush's last attorney general, Michael Mukasey, said: "The Justice Department claims that our courts are well suited to the task. Based on my experience trying such cases and what I saw as attorney general, they aren't."
Sen John McCain, a Republican who lost to Mr Obama in the 2008 presidential race, stated that military tribunals were the best venue for terror suspects.
"They are war criminals, who committed acts of war against our citizens and those of dozens of other nations," he said.
Paul Reynolds, BBC World Affairs correspondent
The decision to prosecute in a federal court is a major step towards President Obama's target of closing Guantanamo by 22 January. However, problems remain.
About 70 prisoners are in limbo. There is not enough useable evidence for a military or a civilian trial yet no other country will so far take them and release is seen as too dangerous.
Guantanamo Bay, as the attorney general indicated, is therefore unlikely to close by the deadline.
The announcement also confirmed military commissions will go ahead in some cases, so the move to civilian trials is not a general one.
It remains to be seen whether the trial will accept any alleged confession by Khalid Sheikh Mohammed as it might have been based on water-boarding, or whether, in a courtroom close to Ground Zero, he will seek to glory in the attacks.
But Democratic Senator Patrick Leahy, who chairs the Judiciary Committee, said the decision demonstrates "to the world that the most powerful nation on earth also trusts its judicial system".
New York City Mayor Michael Bloomberg - an independent - also backed the move: "It is fitting that 9/11 suspects face justice near the World Trade Center site where so many New Yorkers were murdered."
Two Yemenis, a Saudi and a Pakistani-born Kuwaiti will face trial alongside Khalid Sheikh Mohammed.
They are accused of helping finance and plan the attacks of 11 September 2001 in which nearly 3,000 people were killed.
No date was given for a trial, but US media reports say Congress needs at least 45 days' notice before the detainees can be transferred to the US.
The five men had until now been facing prosecution at US military commissions in Guantanamo. The government had faced a 16 November deadline to decide how to proceed in their cases.
Five other detainees have been referred for military commission trials.
Mr Mohammed has been described by US investigators as "one of history's most infamous terrorists". They say he has admitted being responsible "from A to Z" for the 9/11 attacks.
Believed to be the number three al-Qaeda leader, he was captured in Pakistan in March 2003. He told a pre-trial hearing at Guantanamo in December 2008 that he wanted to plead guilty to all charges against him.
Khalid Sheikh Mohammed has been in Guantanamo for more than six years.
However intelligence memos released earlier this year revealed he had been subjected to harsh interrogation techniques including water-boarding on multiple occasions since his capture - potentially rendering some evidence inadmissible.
Some relatives of 9/11 victims expressed anger at the decision.
"We have a president who doesn't know we're at war," said Debra Burlingame, whose brother was the pilot of the hijacked plane that crashed into the Pentagon.
She told AP news agency she was sickened by "the prospect of these barbarians being turned into victims by their attorneys" if the trial focused on torture allegations.
But other 9/11 families have supported the move toward public trials.
"If we claim to be a country that believes in the rule of law, then we need to behave that way ourselves," John Leinung, whose stepson died in the collapse of the Twin Towers, told AP.
The White House is hoping to close Guantanamo by 22 January 2010.
Mr Obama's administration says it will try some detainees in US courts and repatriate or resettle others who are not perceived as a threat.
However, questions remain over the fate of those assessed as dangerous but who for legal reasons could not be prosecuted in a US court - prompting suggestions that the deadline will slip.
Are you in New York? How do you feel these suspects will be received? Do you think New York is the appropriate place to hold the trials? Send us your comments using the form below.
A selection of your comments may be published, displaying your name and location unless you state otherwise in the box below.
The BBC may edit your comments and not all emails will be published. Your comments may be published on any BBC media worldwide.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.