Mr Miliband defended his Ukraine trip
The foreign secretary has said Russia's invasion of Georgia marks "a clear end to the relative... calm" in Europe since the Soviet Union's collapse.
David Miliband told the BBC there was no western "plot" to "encircle" Russia and said there was "no question of launching an all-out war".
But he said Russia needed to consider the "isolation, the loss of respect and the loss of trust" from recent events.
He defended his Ukraine trip, denying that it had increased tensions.
Mr Miliband told BBC Radio 4's Today: "Ukraine is a very important country in thinking about the map of Europe after the collapse of the Soviet Union.
"I thought it was very important, at this stage, to say to friendly countries like the Ukraine that we were determined to support their democratic choices.
"The point I want to underline, above all others, is that what's happened since the collapse of the Soviet Union is not a Western plot.
"It's a series of decisions by independent sovereign democracies about the course that they want to take: not a course of confrontation with Russia, but a course of engagement with the West, which I think is wholly within their rights, and is something we've been right to support."
Mr Miliband reiterated UK support for Ukraine joining Nato if it wanted to.
But he added: "We don't accept that... the choice for Ukraine is that either you are an enemy of Russia or you are a vassal of Russia. You can be a partner of the West. You can be a partner of Russia."
Asked if this meant Nato would be willing to engage in armed conflict with Russia to support one of its members, Mr Miliband said Nato was a defensive alliance.
He said while "it's right to talk about an international crisis... there's no question of launching an all-out war against Russia".
He said: "We are in a situation which marks a clear end to the relative and growing calm in and around Europe since the collapse of the Soviet Union.
"The last 15 or so years have seen a period in which Europe's borders have been clearly demarcated, new nations have come out of the old Soviet Union - they've joined the European Union.
"And the European countries have enjoyed a period of unprecedented stability. I think that the danger that arises from Russian actions is that that period comes to an end."
On Wednesday in Ukraine, Mr Miliband called on the EU and Nato to initiate "hard-headed engagement" with Russia in response to its actions in Georgia.
In a speech in Ukraine's capital, Kiev, he urged them to bolster their allies, rebalance the energy relationship with Russia and defend international law.
Russia recognised the independence of Georgia's two breakaway regions, South Ossetia and Abkhazia, on Tuesday.
Moscow's fellow G8 members have condemned its actions in Georgia.
"We, the foreign ministers of Canada, France, Germany, Italy, Japan, the United States and the United Kingdom, condemn the action of our fellow G8 member," the group of seven of the world's leading industrialised nations said in a joint statement.
Ukraine's President Victor Yushchenko has said his country is a hostage in a war waged by Russia against countries in the old Soviet bloc.
Fighting between Russia and Georgia began on 7 August after the Georgian military tried to retake its Russian-backed breakaway province of South Ossetia by force.
Russian forces subsequently launched a counter-attack and the conflict ended with the ejection of Georgian troops from both South Ossetia and Abkhazia and an EU-brokered ceasefire.
Mr Miliband said Russian President Dimitry Medvedev had a "big responsibility not to start" a new Cold War.
The Russian government responded by saying Moscow saw no such threat.
A spokesman for Prime Minister Vladimir Putin said Russia had been taking "measures of precaution" against Nato warships in the Black Sea, but hoped to avoid confrontation.
"I wouldn't agree that we really have a threat of a new Cold War. Russia was, is and will continue to be the last country in the world that would want a repeat of the Cold War," Dmitry Peskov said.
On Tuesday, Mr Medvedev said Moscow had been obliged to recognise the independence of South Ossetia and Abkhazia following the "genocide" started by Georgian President Mikhail Saakashvili in South Ossetia in August.
"The most important thing was to prevent a humanitarian catastrophe," he told the BBC in an interview in the Russian city of Sochi.
Georgia said Russia was seeking to "change Europe's borders by force".
Most of Russia's forces pulled out of the rest of Georgia last Friday but it maintains a presence both within the two rebel regions and in buffer zones imposed round their boundaries.
Mr Medvedev has blamed Georgia for failing to negotiate a peaceful settlement to the crisis.