Iran has refused to suspend its uranium enrichment programme after six key countries agreed to discuss possible sanctions against Tehran.
The UN Security Council is expected to discuss sanctions next week
A spokesman for Iran's foreign ministry said suspension was "unacceptable" and the threat of sanctions "inefficient".
The five permanent UN Security Council members and Germany held talks in the UK on Friday to discuss Iran's repeated refusal to halt nuclear activities.
Despite fears it is developing nuclear arms, Iran says its aims are peaceful.
"The suspension is completely unacceptable and we have rejected it," foreign ministry spokesman Mohammad Ali Hosseini told reporters.
"The threat of sanctions is an inefficient means to achieve a solution," he said.
Russia and China favour diplomacy, not sanctions. A UN debate on punitive action could start as early as next week.
Speaking after discussions in London, UK Foreign Secretary Margaret Beckett said the six powers would "consult on measures under article 41 of Chapter 7 of the UN Charter".
Iran's nuclear programme is popular with many in the country
Article 41 authorises the Security Council to apply non-military means, such as economic or diplomatic sanctions, "to give effect to its decisions".
EU foreign policy chief Javier Solana said earlier that "the door to negotiations is and will be always open", despite saying earlier this week that little progress had been made in his "endless hours" of talks with Tehran.
He said the issue could only be solved through negotiations, although talks could not go on forever.
The six nations - the UK, Russia, China, France, the US and Germany - had offered Iran a package of incentives in return for the suspension of its uranium enrichment programme.
Iran missed a 31 August UN Security Council deadline to stop work, after which the US agreed to allow Mr Solana to continue talks with Tehran.
But BBC diplomatic correspondent James Robbins says the major powers now seem to accept that Iran is not prepared to meet their demand for a halt.
There are sharp differences between the six countries attending the meeting on what kind of sanctions to possibly impose on Iran.
The US and the UK favour immediate punitive sanctions, but in order to win over Russia and China, they are believed to be considering a series of low-level sanctions as a first step.
Tehran says it will not give up its nuclear programme
These would be what British officials describe as incremental, proportional and reversible.
They could include travel restrictions on Iranian officials and embargoes on missile and nuclear technology that could have dual civilian-military use.
But China and Russia have both said diplomacy is still the best way to achieve a solution.
On Thursday, Russian Foreign Minister Sergei Lavrov described sanctions as "extreme measures".
He said he would like to "see what extra possibilities exist to pursue multiparty diplomatic efforts".
Russia has built a nuclear power station in Iran and China relies on oil imports from the country.