The number of people using the gym has risen in the past decade.
Middle-aged men and women in England are more likely to play sport than younger people, research suggests.
A study of more than 60,000 adults also found those who are comfortably off and white were most likely to do exercise.
But the figures showed a fall in the number of young men taking part in activities such as cycling and running.
Writing in the British Journal of Sports Medicine, the researchers said the findings raise concerns about the "widening gap" between rich and poor.
More than 33,000 women and 27,000 men over the age of 16 years were questioned as part of the Health Survey for England between 1997 and 2006.
By the end of the study period men were around 10% more likely to regularly play sports than in 1997 while women were 20% more likely.
And the proportion attending regular gym or fitness classes rose from 17% to 19.2% among men and from 15.9% to 18.7% among women.
However, the researchers found the increase was not evenly spread over different age and socio-economic groups.
In general, people with higher incomes, car owners, those in generally good health and higher social classes were more likely to take part in sports.
The findings also showed that, particularly among men, white people were more likely to take exercise than other groups.
Those aged over 45 were 25% more likely to have undertaken sport over the period and there were also increases among women aged 30 to 44.
But men aged 16 to 29 were less likely to have taken part in popular sporting activities.
Study leader Dr Emmanuel Stamatakis, from the Department of Epidemiology and Public Health at University College London, said older men and women might have taken up more activities in the exercise boom of the 1990s and then kept it up
"But what's more tricky to explain is why younger people are falling back to doing less sport, and this is more pronounced the lower you go in the age ranges."
If promoted properly, the 2012 Olympics should be a "huge opportunity" to encourage more people from all walks of life to do sport, he added.
"But it has to be a co-ordinated strategy to target those groups most in need, if 2012 is going to leave the lasting sporting legacy this country needs."
A spokesperson for the Department of Health said they were introducing schemes aimed at those least likely to do exercise.
"For example, patients in a small number of GP pilot practices across London are receiving advice and support form their local GP or practice nurse on becoming more active."
John Brewer, performance director at the Lucozade Sport Science Academy in Slough, said he was not surprised by the findings.
"There's so many other things young people are interested in and so many of them are sedentary, things like texting and computer games.
"We need a fresh approach to what constitutes exercise."