There is even the risk of being shot down - in 1995 the balloon of two Americans was shot down and they were killed when they entered Belorussian airspace.
So what drives them? Money? Richard Branson and Steve Fossett are already millionaires.
A place in the history books? Or, perhaps, the simple pleasure of floating away and above it all?
Branson described his fascination with this gentle sport: "There were definitely times where I said to myself, if I get out of this alive I will definitely thank someone for getting me out of this fix, and call it a day.
"Then about three hours later you start seeing the magnificence of the Sahara and the beautiful mountains, the dawn coming up, being somewhere which most people in the world have never seen... It's difficult to resist the whole fascination of that." In the last year three men have nearly lost their lives trying to fly round the world in a balloon.
The prize
The winner of this exclusive race could win up to $1.5 million (£919,664).
The US brewery Anheuser-Bush has set aside a one million-dollar prize: $500,000 (£307,000) go to the winners, and a further $500,000 to the charity of their choice.
But most of these men are millionaires already, so it can't be the money that drives them on.
One can only assume that the challenge and the hope of glory keeps the competition running.
Rules of the game
The deadline for the great ballooning challenge is December 31, 1999. The International Aviation Federation, which officiates over record-breaking flights, has set strict rules:
International Aviation Federation - Ballooning Commission
Ballooning News
ICO Global
Cable and Wireless balloon
Team Remax
J Renee
Breitling Orbiter III
The BBC is not responsible for the content of external internet sites.
Balloonists: Next stop Geneva
Stranded balloonists rescued
Desert touchdown for balloon heroes
News Online users cheer record balloonists
Balloonists soar into history
Sky-high hopes
Balloons make history
Ballooning's 'triumph of a dream'