The combined stockpiles - 34 tonnes from each country - are said to be enough to make 17,000 nuclear warheads. US officials said it would be used as fuel in civilian reactors to generate electricity.
The US will provide $400m of the funding for the disposal of Russia's plutonium, which Moscow estimates will cost up to $2.5bn (£1.63bn).
Several other countries - including Mexico, Chile and Ukraine - had earlier agreed to give up their stocks of highly-enriched uranium.
At the end of the unprecedented 47-nation nuclear security summit, Mr Obama announced that the assembled world leaders had agreed a work plan to counter the danger of nuclear material falling into the hands of terrorists.
Jonathan Marcus, BBC News, Washington
One way or another these issues - proliferation, arms control and the security of nuclear materials - are all bound together. They will be discussed again at the forthcoming review of the Nuclear Non-Proliferation Treaty in New York in May.
In many ways all of Mr Obama's recent efforts - the revised US nuclear doctrine, the new Start treaty with Russia, this nuclear security conference - are all preparation for that event. The Non-Proliferation Treaty is the cornerstone of efforts to prevent the spread of nuclear weapons.
It is widely regarded as ailing, badly in need of repair. That is the next item on Mr Obama's nuclear "to do" list.
"Today, we are declaring that nuclear terrorism is one of the most challenging threats to international security," he said.
"We also agreed that the most effective way to prevent terrorists and criminals from acquiring nuclear materials is through strong nuclear security."
"[This] is a testament of what is possible when nations come together in a spirit of partnership to embrace our shared responsibility and confront a shared challenge," he added.
In a joint communique, the leaders agreed to non-binding measures to "secure all vulnerable nuclear material in four years" and to "prevent non-state actors from obtaining the information or technology required to use such material".
They said they would co-operate more deeply with the UN's nuclear watchdog, the International Atomic Energy Agency (IAEA), and share information on nuclear detection and ways to prevent nuclear trafficking.
But increased security should "not infringe upon the rights of states to develop and utilise nuclear energy for peaceful purposes and technology", they added.
Progress is to be reviewed at a summit in South Korea in 2012.
BBC diplomatic correspondent Jonathan Marcus, who is at the summit, says the deal depends upon the will of participating governments to take action.
There can be no guarantees of progress, but it is clear the Obama administration intends to keep this issue at the top of the global agenda, he adds.
Earlier, Mr Obama warned that although the risk of nations using nuclear weapons had lessened, "the risk of a nuclear attack has gone up".
"Terrorist networks such as al-Qaeda have tried to acquire the material for a nuclear weapon and, if they ever succeed, they would surely use it," he said.
Iran's President Ahmadinejad has called a rival nuclear summit
"Were they to do so, it would be a catastrophe for the world, causing extraordinary loss of life and striking a major blow at global peace and stability."
He warned that a quantity of radioactive material just "the size of an apple" would be enough to kill thousands of people.
The two-day summit is the biggest international meeting hosted by the US since 1945.
It is taking place without representatives of Iran and North Korea, neither of which were invited by the US because of disputes over their nuclear programmes.
In a defiant move, Iran has announced that it will hold its own nuclear summit in Tehran this weekend with the foreign ministers of 15 countries.
It is estimated there are about 1,600 tonnes of highly enriched uranium in the world - the type used in nuclear weapons.
Experts agree that virtually all of it is held by the acknowledged nuclear-weapons states, most of it in Russia.
Last week, the US and Russia signed a new Strategic Arms Reduction Treaty (Start), committing them to each reduce the number of deployed strategic warheads to 1,550 - 30% lower than the previous ceiling.
Mr Obama has also approved a new nuclear policy for the US, saying he plans to cut the nuclear arsenal, refrain from nuclear tests and not use nuclear weapons against countries that do not have them, unless they do not comply with the Nuclear Non-Proliferation Treaty (NPT).
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.