The court ruled there was no attempt to destroy the Albanian ethnic group
The controversial ruling by the UN-supervised Supreme Court in the Kosovan capital, Pristina, has angered Albanians, and some UN officials are reported to be preparing to challenge it.
The decision comes as authorities in Serbia begin the excavation of another mass grave believed to contain the bodies of around 50 Kosovar Albanians.
Four graves have already been investigated, revealing the remains of 340 victims.
UN 'unhappy'
The court, Kosovo's highest legal body, said there had been a "systematic campaign of terror, including murders, rapes, arsons and severe maltreatments".
Crimes against humanity and war crimes did take place, it said, but "the exactions committed by Milosevic's regime cannot be qualified as criminal acts of genocide, since their purpose was not the destruction of the Albanian ethnic group... but its forceful departure from Kosovo".
However the BBC's Paul Wood in Belgrade says that some UN legal officials are deeply unhappy and have begun a campaign to have the ruling overturned.
The decision was based on the 1948 Geneva convention which defines genocide as the intent "to destroy, in whole or in part, a national, ethnic, racial or religious group as such".
Milosevic debate
The court, which is comprised of two international judges and one Albanian, was ruling on the case of a Serb, Miroslav Vuckovic, convicted of genocide by a district court in Mitrovica.
International officials have raised concerns about the treatment of Serbs by Kosovo's Albanian dominated judiciary.
Mr Vuckovic's conviction has now been overturned and he will face a retrial in Mitrovica.
The decision is likely to reopen the debate on whether Slobodan Milosevic should face genocide charges at The Hague, where he already stands accused of war crimes and crimes against humanity.
The tribunal is currently preparing indictments of genocide against Mr Milosevic in connection with atrocities carried out in Bosnia and Croatia.
Mass grave
Serbian authorities in the western Serbian town of Bajina Basta began on Thursday to excavate a mass grave thought to contain the bodies of Kosovo Albanians.
They are believed to have been transported out of the province in an attempt by Mr Milosevic to cover up atrocities and possible war crimes carried out during his campaign of terror.
Serbian police believe around 800 victims of the conflict in Kosovo have been buried around Serbian territory.
The gruesome revelations of the bodies are credited with changing public opinion in Serbia and increasing acceptance that war crimes were carried out under the Milosevic regime.