The UK plans to withdraw its 4,100 troops by the end of July 2009
Iraq's parliament has rejected a draft law that would have permitted forces from the UK, Australia and a number of other countries to remain after 2008.
The bill, rejected by 80 votes to 68, would have given the 6,000 non-US troops a legal basis for staying once the UN mandate expires on 31 December.
It will now be sent back to the cabinet for amendment. A vote is due next week.
The 140,000 US troops are allowed to remain until the end of 2011 under a separate status of forces agreement.
The UK Ministry of Defence said in a statement that it would now discuss with the Iraqi government how the bill's rejection could affect the status of non-US coalition forces.
Fariad Rawndouzi, a Kurdish MP, told the BBC many of his colleagues were not happy with the formulation of the bill, and wanted it to more closely resemble the deal between the US and Iraq.
A member of the Sunni Iraqi Accord Front in the Council of Representatives, Hussein al-Falluji, echoed the sentiment, saying that relations between Iraq and other states ought to be "arranged, according to international law, through treaties or agreements".
"For this reason parliament rejected this law. It was a big mistake by the government," he told the Reuters news agency.
Nasser al-Issawi, an MP loyal to the radical Shia cleric Moqtada Sadr, hailed the rejection of the draft as a "great national achievement", and said he hoped the foreign troops would be forced to leave when the UN mandate ends.
The government has so far not commented on the vote. A draft had already been rejected earlier this week, the Associated Press reports.
Earlier in the week, UK Prime Minister Gordon Brown announced that the UK planned to withdraw its troops from southern Iraq by the end of July 2009, as envisaged by the draft law.
Military operations will end by 31 May and the remaining 4,100 service personnel will leave within two months. Several hundred trainers will remain, some working with the Iraqi navy.
"As the prime minister announced to parliament last Thursday, Prime Minister [Nouri] Maliki confirmed to him in Baghdad on Wednesday that the government of Iraq wants UK forces to remain in Iraq to complete their military tasks, by 31 May 2009 at the latest," the UK Ministry of Defence said in a statement after Saturday's vote.
NON-US FORCES IN IRAQ
UK - 4,100
Australia - 1,000
Romania - 500
El Salvador - 200
Estonia - 40
"We have worked closely with the government of Iraq to ensure that there is a firm legal basis for the presence of our forces in 2009.
"We will now discuss with the government of Iraq what the vote in the Council of Representatives means for the proposed legal basis for the UK and other Coalition forces and look at other options."
The BBC's Caroline Wyatt in Baghdad says she understands that, if the bill is rejected by parliament at a third reading next week, the Iraqi government may sign individual agreements directly with each of the foreign states, giving their troops a legal basis to remain.
However, this is an emergency option which the government had been trying to avoid, as there may not be enough time to negotiate the deals before 31 December, our correspondent says.
Apart from the US and UK, the only countries continuing to provide troops for Multi-National Force - Iraq (MNF-I) are Australia, El Salvador, Estonia and Romania.
South Korean troops ended their mission in Iraq on Friday, joining their Japanese counterparts, who pulled out a day earlier.