The court's verdict had long been awaited
Bosnian Muslim leaders have voiced disappointment after the top UN court cleared Serbia of direct responsibility for genocide during the Bosnian war.
The International Court of Justice in The Hague said the massacre of 8,000 men in Srebrenica was genocide, but Belgrade was not directly responsible.
But it said Serbia broke international law by failing to stop the killings.
Serbia's president acknowledged the ruling, and urged parliament to condemn the Srebrenica massacre.
The case was the first of a state being charged with genocide. Individuals have been convicted of genocide in Bosnia.
At least 100,000 people died in the 1992-1995 war, triggered by the break-up of the former Yugoslavia. Bosnia's Muslims and Croats wanted to cut ties with Belgrade, a move opposed by Bosnian Serbs.
The case, Bosnia and Herzegovina versus Serbia and Montenegro, began a year ago.
Bosnia argued that Belgrade incited ethnic hatred, armed Bosnian Serbs and was an active participant in the killings.
Belgrade said the conflict was an internal war between Bosnia's ethnic groups and denied any state role in genocide.
In the ruling, the president of the court, Judge Rosalyn Higgins, said: "The court finds that the acts of genocide at Srebrenica cannot be attributed to the respondent's (Serbia) state organs."
However the court added that the leaders of Serbia failed to comply with its international obligation to prevent the killings and punish those responsible.
The court also rejected Bosnia's claim for reparations.
"Financial compensation is not the appropriate form of reparation," the ruling said.
The war crimes tribunal in The Hague has already found individuals guilty of genocide in Bosnia and established the Srebrenica massacre as genocide.
The BBC's Nicholas Walton in Sarajevo says many Bosnian Muslims were hoping for a clear ruling that Serbia as a state was responsible for pursuing a genocide in Bosnia during the 1990s.
The Bosnian Muslim member of the country's tripartite presidency, Haris Silajdzic, told the BBC there was "disappointment" at the outcome.
However he welcomed the fact that the court had "ruled that Serbia and Montenegro had violated the Convention on the Prevention and Punishment of the Crime of Genocide by not preventing or punishing the perpetrators of the genocide".
In Republika Srpska - the Serb entity - Krstan Simic, a senior member of the ruling Union of Independent Social-Democrats, said he was pleased that the judges had taken "real facts " into account.
In Serbia itself, President Boris Tadic urged parliament to pass a declaration "condemning the crime in Srebrenica without any doubt".
The German presidency of the European Union urged Serbia "to use today's judgment as a further opportunity to distance itself from the crimes committed by the Milosevic regime".
The ruling comes with Serbia still facing challenges linked to the break-up of the former Yugoslavia.
Admission talks with the EU have been stalled over Belgrade's failure to hand over war crimes suspects for trial.