A BBC correspondent in Tokyo says many in Japan will feel North Korea has been rewarded too soon.
There has not yet been any official North Korean response to the US move.
Tokyo argues North Korea should remain on the list because issues related to the abduction of Japanese citizens in the 1970s and 1980s have not been resolved.
South Korea, however, welcomed the US decision.
"This government welcomes these moves as an opportunity that would lead to normalisation of the six-party talks and North Korea's eventual abandonment of its nuclear programmes," said Kim Sook, Seoul's nuclear envoy.
On Saturday State Department spokesman Sean McCormack said North Korea would resume its disabling of nuclear facilities.
This disablement was agreed in 2007, but the process has recently reversed with North Korea threatening to restart its Yongbyon reactor.
Under the latest accord, North Korea will allow nuclear experts to take samples and conduct forensic tests at all its declared nuclear facilities and undeclared sites, the State Department said.
The North will also allow inspectors to verify its denials about transfers of nuclear technology and an alleged uranium programme.
Analysts say it is not clear the latest agreement will succeed: Several previous deals have broken down due to different interpretations of what was required.
Washington's announcement came after a visit to Pyongyang last week by US envoy Christopher Hill, and days of talks between the US and its negotiating partners China, South Korea, Russia and Japan.
Together with North Korea, they have been involved in long-running six-party talks over the de-nuclearisation of the Korean peninsula.
Pyongyang began disabling its Yongbyon nuclear reactor in August, but recently it has made moves to reassemble the plant after Washington refused to remove North Korea from the terror blacklist.
It also expelled UN inspectors and test-fired short-range missiles, increasing tensions with the US.
Correspondents say Pyongyang wanted to be removed from the US list in order to receive international aid and loans, and to facilitate its diplomatic rehabilitation.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.