India is under pressure from Washington to sign the accord before the US presidential elections in November.
Under the terms of the accord, India would get access to US civilian nuclear technology and fuel.
In return, Delhi would open its civilian nuclear facilities to inspection - but its nuclear weapons sites would remain off-limits.
The communists have 59 members in India's lower house of parliament.
TIMETABLE FOR NUCLEAR ACCORD
Approval needed from IAEA, expected to meet in late July
Consent also required from 45-member Nuclear Suppliers Group
Congress to approve deal before President Bush signs it into law
All this to happen before Mr Bush's tenure expires on 3 January 2009
Analysts say their pulling out of the governing coalition will leave it with 226 members in the 543-seat parliament, a good 46 seats behind the majority mark.
But they say the communists' decision may not lead to a collapse of the government and early elections.
Mr Singh thinks so too.
Speaking to reporters at the G8 summit in Japan, he said: "I just learned it [about the withdrawal]. But I don't think it will affect the stability of our government."
Reports say the Congress party has struck a deal with a north India-based regional party, the Samajwadi Party, and several other smaller parties to compensate for the communists' loss of support.
The Samajwadi Party has been a traditional political foe of Congress, but has said that its 39 MPs would support the governing coalition on the nuclear deal issue.
"Every Samajwadi Party vote in parliament will be cast in support of the India-US nuclear deal and to save the government," party chief Mulayam Singh Yadav said on Tuesday.
The Congress party is now expected to have to prove its majority in parliament in a formal vote.
'Time has come'
The communists had been insisting that they would withdraw support if the government went ahead with the deal. Mr Singh confirmed it would on Monday, in remarks to reporters en route to Japan.
Mr Singh has described the deal as 'historic'
Communist leader Prakash Karat told reporters that they had sought an appointment with the president "so that we can formally withdraw support tomorrow".
"In view of the prime minister's announcement that time [to withdraw support] has come," said Mr Karat.
Another senior communist leader, AB Bardhan, said: "The prime minister had to make this announcement at 30,000ft in the air."
Congress party leader Sonia Gandhi has convened an urgent party meeting to discuss the withdrawal of support.
The deal now needs to be approved by the UN's nuclear watchdog, the International Atomic Energy Agency (IAEA), as well as by the 45-nation Nuclear Suppliers Group, which regulates global civilian nuclear trade.
Then it has to be presented to the US Congress for final approval.
Mr Singh said he would submit an application to the IAEA as soon as possible.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.