The authorities say their border guards encountered an armed group wearing uniforms of the Kosovo Liberation Army, trying to enter Kosovo illegally from Albania.
According to the Serbian Ministry of Information, the shootout happened on Monday near the border outposts of Gorozup and Liken, 70km (45 miles) southwest of Pristina.
It said the clash lasted around five hours and explosions were heard from three villages that had been sealed off by police.
Large quantities of weapons and supplies were also reported to have been seized.
The incident resulted in the highest single number of casualties since a US-brokered informal truce in October halted the fighting in the separatist province in southern Serbia.
The clashes also came as a US envoy to Kosovo resumed talks on Monday with the province's ethnic Albanian negotiators.
Christopher Hill, whose proposals for a political settlement for Kosovo have been rejected by both sides, met with negotiators in Pristina on Monday.
He plans to travel to Belgrade on Tuesday to be joined by another US envoy Richard Holbrooke for more talks with Serbian Government authorities.
Country profiles
KLA offers to delay independence
(28 Nov 98 | Europe)
Serbs criticise US Kosovo plans
(18 Nov 98 | Europe)
Agreement in Belgrade
(13 Oct 98 | Europe)
Federal Republic of Yugoslavia
Serbian Ministry of Information
Kosovo Information Centre
Human Rights Watch on Kosovo
The BBC is not responsible for the content of external internet sites.
Violence greets Clinton visit
Russian forces pound Grozny
EU fraud: a billion dollar bill
Next steps for peace
Cardinal may face loan-shark charges
Vodafone takeover battle heats up
(From Business)
Trans-Turkish pipeline deal signed
French party seeks new leader
Jube tube debut
Athens riots for Clinton visit
UN envoy discusses Chechnya in Moscow
Solana new Western European Union chief
Moldova's PM-designate withdraws
Chechen government welcomes summit
In pictures: Clinton's violent welcome
Georgia protests over Russian 'attack'
UN chief: No Chechen 'catastrophe'
New arms control treaty for Europe
Mannesmann fights back
(From Business)
EU fraud -- a billion-dollar bill
New moves in Spain's terror scandal
EU allows labelling of British beef
UN seeks more security in Chechnya
Athens riots for Clinton visit
Russia's media war over Chechnya
Homeless suffer as quake toll rises
Analysis: East-West relations must shift