The US is insisting that American soldiers be guaranteed immunity from prosecution by a new international war crimes court (ICC), which has now come into being.
The Security Council later agreed to extend the mission's mandate for 72 hours to give diplomats time to try to resolve the issue.
A BBC correspondent in the Balkans says 1,500 UN police officers and hundreds of logistical staff will have to start packing their bags on Thursday if no solution is found, but that there is no sense of panic.
After using its veto the US did continue negotiations, which could indicate a willingness to break the deadlock.
UK Foreign Secretary Jack Straw on Monday played down the threat.
Speaking on the BBC's Today programme, he said he had been in touch with the US Secretary of State, Colin Powell, over the weekend.
He said: "We are trying to allay their fears. The ICC statute makes crystal clear that the ICC would only be involved if there was a failure by national criminal jurisdiction to deal with a transgression by peacekeepers.
The US stands alone against the 14 other council members.
The BBC's Mike Fox, at the UN, says there is little sign that more talking will change anything except to buy more time for arranging a Bosnian pullout.
Policing threatened
Correspondents say the first effect of the American veto would be the end of the UN mission to train a new Bosnian police force.
UN staff training the civilian multi-ethnic force include 46 US police officers.
Following the American decision on Sunday, UN Secretary General Kofi Annan - who was present for the vote - appealed to council members to intensify efforts to resolve the dispute.
The Bosnian mission's mandate officially expired at 0400 GMT on Monday.
"The world cannot afford a situation in which the Security Council is deeply divided on such an important issue, which may have implications for all UN peace operations," he said.
Mr Annan said the premature ending of the UN mission would be seen throughout the Balkans as a lessening of international commitment to the region.
Nato, for its part, has insisted that its peacekeeping mission in Bosnia will not be jeopardised by the US veto as its mandate stems from the 1995 Dayton peace agreement.
Nato leads the 19,000-strong Stabilisation Force in Bosnia, or S-For. However, the force includes 3,100 Americans and it is unclear whether they could be pulled out.
Withdrawal threat
Although S-For does not legally require a Security Council mandate, some of the 19 countries contributing to it have indicated they will withdraw their troops without one.
Nato's top decision-making body, the North Atlantic Council, is due to meet on Monday in Brussels to consider the implications of Nato's move.
The ICC
Comes into being on 1 July and begins work early next year
Will be based in The Hague
68 nations have ratified the treaty
100-plus nations have signed up and may ratify the treaty in the future
China, India, Pakistan, Indonesia, Iraq and Turkey have failed to sign up to the treaty
The US, Egypt, Iran, Israel and Russia have failed to ratify the treaty
US ambassador to the UN, John Negroponte, said Washington vetoed the resolution "with great regret".
The US has long opposed the creation of the International Criminal Court (ICC) because it is worried its peacekeepers and officials could face politically-motivated prosecutions.
"We will not ask them [US personnel] to accept the additional risk of politicised prosecutions before a court whose jurisdiction over our people the government of the United States does not accept," Mr Negroponte said.
"With our global responsibilities, we are and will remain a special target, and cannot have our decisions second-guessed by a court whose jurisdiction we do not recognise."