IOC president Jacques Rogge announces the winner - Copyright IOC 2009
Brazil will become the first South American country to host the Olympics after the city of Rio de Janeiro was chosen to stage the 2016 Games.
Rio won a majority of the 95 votes at the meeting in Copenhagen, eliminating Madrid in the final round. Tokyo and Chicago had already been knocked out.
"The world has recognised that the time has come for Brazil," said President Luiz Inacio Lula da Silva.
Chicago's early exit was a surprise, after bookmakers made them favourites.
US President Barack Obama had flown to Denmark on Friday morning to join his wife, Michelle, and make an emotional address to the International Olympic Committee delegates.
But the gesture - the first time a current US president had addressed the IOC in an attempt to win the Games - failed to persuade the voters as Chicago became the first city to see its dream of hosting the biggest sporting event in the world fall by the wayside.
Obama's wish for better 2016 news
Speaking to reporters at the White House on his return, Mr Obama said he wished he had come back with better news, but congratulated Brazil for a "truly historic" win.
"As friends to the Brazilian people, we welcome this extraordinary sign of progress," he said.
The president said he had no doubt that Chicago's bid had been the strongest possible, could not be prouder of the city, and insisted that he had no regrets about travelling to Denmark.
"I believe it is always a worthwhile endeavour to promote and boost the United States of America," he added.
'Heart and soul'
Chicago received only 18 of the 94 votes available in the first round poll of IOC delegates at the Bella Convention Centre on Friday afternoon. Madrid came out top with 28, followed by Rio on 26.
I confess to you if I die right now my life would have been worth it
In the second round, however, Rio almost secured the absolute majority needed to win outright, with 46 of the 95 votes cast. Madrid came a distant second with 29, while Tokyo was eliminated after receiving 20.
The final ballot saw Rio win by a comprehensive margin of 66 votes to 32.
More than an hour later, IOC president Jacques Rogge finally revealed the result to the world: "Like in every competition there can only be one winner.
"Tonight, I have the honour to announce that the Games of the 31st Olympiad are awarded to the city of Rio de Janeiro."
The Brazilian bid team leapt to their feet in celebration, began singing their "Marvellous City" song, waving flags and hugging each other.
A tearful President Lula told reporters afterwards: "The other countries made proposals. We presented a heart and a soul.
"I confess to you if I die right now my life would have been worth it."
The BBC's Gary Duffy in Rio said there was "absolute, unrestrained joy" on the city's famous Copacabana beach after the result was announced, with silver glitter flying through the air and tens of thousands of people singing.
The golden sand disappeared under a sea of green and yellow, the colours of Brazil, our correspondent said.
There have been questions about Rio's bid not the least the high level of violence which so often scars the city's image, but in one glorious moment that was all set to one side for a truly Brazilian party, he added.
A promotional video which aims to boost Rio's bid - copyright IOC 2009
In his speech to the IOC earlier, President Lula had promised: "Rio will deliver an unforgettable Games. You will see for yourselves the passion, the energy and the creativity of the Brazilian people.
"It will not be just Brazil's Games but South America's. It will serve to inspire the 180 million young people on the continent. It is time to redress the balance. It is time to light the Olympic flame in a tropical country."
Rogge said Rio had presented the IOC "with a very strong technical bid, built upon a vision of the Games being a celebration of the athletes and sport, as well as providing the opportunity for the city, region and country to deliver their broader long-term aspirations for the future".
"This call to 'live your passion' clearly struck a chord with my fellow members, and we now look forward to seeing Rio de Janeiro staging the first Olympic Games on the continent of South America," he said.
Rio plans to stage all the competitions inside the city, bringing "dynamics to the Games and facilitating the athletes' interaction", according to the bid website.
There will be seven competition centres in four Olympic regions - Barra, Copacabana, Deodoro, and Maracana - with football matches held in the cities of Belo Horizonte, Brasilia, Salvador and Sao Paulo.
Its bid is divided into a $2.8bn (£1.92bn) budget for operating costs, and $14.4bn (£9.90bn) for construction and security.
Madrid's surprising success in reaching the final round came after former IOC president Juan Antonio Samaranch made an emotional appeal.
Shock in Madrid as it fails in its bid to host the 2016 Olympics
"I know that I am very near the end of my time. I am, as you know 89 years old. May I ask you to consider granting my country the honour and also the duty to organise the games in 2016," he said.
However, a win for Madrid would have made it three European-based Olympics in a row, with London 2012 and Sochi 2014 preceding it.
After the result, Prime Minister Jose Luis Rodriguez Zapatero said Madrid could leave Copenhagen with "dignity" and did not rule out another bid.
"We were nearly there, but Rio won," he told Spanish reporters. "Now is not the moment to talk about 2020. Madrid is always a strong candidate, but it depends on the city, not me."
Adam Brookes, BBC News, Chicago
The shock of Chicago's elimination was greater for the fact that it came in the first round.
And greater for the fact that President Obama had taken valuable hours from his packed and tense political schedule to travel to Copenhagen. His legendary powers of persuasion will be said to have failed him, though in reality it will be Chicago's bid that failed him.
Nonetheless, this is a moment which allows the president's detractors to allege waning prestige on the part of his presidency. And it will raise questions about the political advice that he is receiving.
The BBC's Steve Kingstone in the Spanish capital said Madrilenos were leaving the square in front of the royal palace, where they watched events in Copenhagen, with a sense of pride.
In the lead-up to the decision, King Juan Carlos had called the Olympics "Madrid's unfulfilled dream of hope", and for now at least that is what the games remain.
Tokyo's bid team were said to have been shocked and stunned after their elimination in the second round, as they had felt their focus on the environment and youth would impress IOC delegates.
"It was really disappointing for us and for the children but this is not the end. We want to carry this on to the next chance," said marathon runner Yuko Arimori.
In Chicago, which had also believed it would be a finalist, thousands stood in silence in the city centre after it became the first to be eliminated.
"We fought a good fight and I am very proud of the team and the campaign. I have no idea why we went out so early. The IOC members will have to ask themselves that question," said Chicago bid chief Pat Ryan.
This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.