The IOC was concerned about the Nazis' rise to power but, after the successful Winter Olympics in Germany earlier that year, Berlin hosted the Games.
Jesse Owens was the star of Berlin in 1936
Did you know?
Owens took 70 mins to break three world records and tie a fourth in 1935
Many countries proposed boycotting the Games but the only notable absentees were Spain because of their civil war.
The Games were opened by Adolf Hitler and some teams gave the Nazi salute during the opening ceremony although Great Britain and the US abstained.
But the star of the Games was black American Jesse Owens.
Owens made a mockery of Hitler's Aryan superiority by winning four gold medals - in the 100m, 200m, long jump and as a member of the 4x100m relay team.
The powerful German team, backed fully by the government, did top the medals table after picking up 33 golds, with the US collecting 24 and Hungary ten.
Canoeing, basketball and outdoor 11-a-side handball all made their Olympic debuts.
Because of the Second World War these were to be the last Olympics until 1948.