Rio's Maracana Stadium hosted the first Pan-American Games in 2007
Chicago, Madrid, Tokyo and Rio de Janeiro are the cities shortlisted by the International Olympic Committee to host the 2016 Olympic Games.
Doha, Prague and Baku were eliminated from the nominal list of potential hosts by the IOC executive board.
IOC president Jacques Rogge said: "All bids were of a high standard. It's a tribute to the health of the Olympic Movement that the field was so strong."
The IOC will select a winning city at a secret ballot on 2 October 2009.
The strongest contender in terms of infrastructure, support and money is Chicago, which is bidding to host the first summer Olympics in America since Atlanta in 1996.
The Americans will however have to improve the technical aspect of their bid after an IOC evaluation report ranked Tokyo and Madrid higher than Chicago, with Rio in fourth place.
Chicago's bid chairman Patrick Ryan said there was plenty more work to be done to improve their bid.
"Four cities is better than five," he said. "Three would have been better than four. But we're very proud to be one of the four. It's exhilarating.
"There are a few things we've learned - don't assume anything, be humble, and work, work, work. And we're going to work right to the end."
Rio de Janeiro, host of the Pan-American Games in 2007, would be the first South American city to host the Olympics.
Madrid, meanwhile, is bidding on the back of finishing a close third behind London and Paris in the race for the 2012 Games, and Tokyo hopes to host the event for the first time since 1964.
"We are very happy with the decision by the IOC," said Tokyo bid chief Ichiro Kono.
"We are also delighted with the evaluation but we have not seen it yet and we must now analyse it very carefully to strengthen our weak points and make our strong points stronger."
Doha, the capital of Qatar, successfully staged the 2006 Asian Games and was regarded as a wild card but the city failed to make the cut despite having hired a high-powered team to run the campaign in the hope of making it to the final vote in Copenhagen.
Some of the IOC's executive board members had argued that a city with a population of around 500,000 was too small, while Doha's plan to hold the Games in October to avoid the searing summer desert heat proved unpopular.
"The weather was the main reason we were left off the shortlist," said Doha bid official Aneesa Al Hitimi.
"Qatar are the leaders in the Middle East and we would have staged 2016 perfectly. God willing, Qatar will make it in the future."