Biological weapons are banned under the 1972 treaty
Washington's representative said the US was unable to support the draft accord - the result of years of debate - because it would not achieve its goals and would hurt American interests.
Members of non-governmental organisations sitting in the public gallery shook their heads and called the announcement a "disaster".
Some official delegations also said they regretted the US decision and called for negotiations to continue without Washington.
The germ warfare agreement aims to introduce measures to enforce the 1972 Biological Weapons Convention, which the US says it still supports.
The draft, which more than 50 other countries have accepted as the basis for further negotiation, would oblige member states to open for inspection sites that could be used for the development of biological weapons.
It also sets out a series of steps for verification, including spot checks.
But the US said the checks would not stop cheating by states wanting to develop biological weapons and did not protect commercially sensitive information.
Speaking after Mr Mahley's statement, delegates from Canada, Japan and South Africa were cautious about criticising the United States, but all three countries called for the talks to continue in a "calm and business-like manner".
Seiichiro Noboru, head of the Japanese delegation to the talks, said the US move had effectively killed off this round of talks.
"US participation is imperative for our task to set up stringent compliance measures," he said.
Scepticism
The US had taken a leading role in the push for an agreement on enforcing the 1972 convention, since Iraqi armaments discovered after the Gulf War appeared to show that the treaty alone had not stopped countries from developing biological weapons.
One NGO, the US- and German-based Sunshine Project, said the other countries should go ahead with the draft, just as climate-change negotiators approved an agreement to fight global warming earlier this week, after Washington pulled out.
This position was supported by the most governments, while Cuba condemned the "unilateralism of the world superpower on key issues of the international agenda".
A US official, speaking on condition of anonymity, said problems in the draft had been identified by the Clinton administration, before President George W Bush took power.
But the BBC's defence correspondent says the announcement is the latest sign that the Bush administration is prepared to go it alone in the teeth of international opposition.