Jovanovic had appealed against his nine-month suspension for failing a drug test, but instead received a ban of two years.
The Court of Arbitration for Sport (CAS) quashed the 25-year-old's last chance of making it to the Winter Olympics at Salt Lake City.
Jovanovic, who had been due to compete in two-man and four-man bob, tested positive for steroids in the national trials on 29 December.
He protested his innocence, saying he had unintentionally took a banned steroid in a nutritional supplement.
But, after nearly two days of deliberation, the CAS refused to accept the bobsledder's defence and decided to extend his original penalty.
"It's the right decision. It's good for sports. It's good for the Olympics," said Dick Pound, chairman of the World Anti-Doping Agency.
Jovanovic was to be a key member of the Olympic team that would end an American medal drought in bobsleighing that stretches back to 1956.