Funding for the scheme is guaranteed for at least two years
New figures show the government's offer of free swimming for young people and the elderly in England led to 4.4m pool visits in the first three months.
The £140m scheme was open to people aged 16 and under or 60 and over.
It aims to ensure a lasting legacy from the 2012 London Olympics, by getting more people physically active.
More than 250 councils took part, but about 140 councils turned down central funding because they could could not afford to offer free access.
The scheme is operating at more than 1,000 council pools in England, with 259 councils offering free swimming for people aged 60 and over, and 190 for young people aged 16 and under.
The figures for the first three months of the scheme were welcomed by Health Secretary Andy Burnham.
He said: "I'm thrilled the free swimming initiative is so popular with both young and older people.
"Getting moving in the water makes your whole body work that little bit harder than on dry land - so activity in the pool is a great way to get your active minutes."
Culture Secretary Ben Bradshaw said the figures were proof of the scheme's success and the government's commitment to long-term benefits from the 2012 Olympics and Paralympics.
"We never had any doubt that free swimming was going to make a splash - but it has shown itself to be an even bigger success with people, young and old, than we had ever expected," he said.
Broadcaster and keen swimmer Sir Terry Wogan is championing the initiative. He said free swimming was a "great idea that can only do good".
"I am delighted to endorse it and am very pleased to hear that so many people have dived in and enjoyed the benefits," he said.
Swimming's governing body, the ASA, is working with the government and local authorities.
Chief Executive David Sparkes said: "All the research pointed to a massive latent demand for swimming and this initiative seems to have put a light to the blue touch paper.
"We now need to keep this going by getting more people involved in swimming, having more fun together."
Funding to cover the cost of free pool access, during standard swimming sessions, is guaranteed for at least two years.
The cost is being shared between five government departments and uptake results will be published every three months.