Primary schools may need to consult local mosques and involve families in their efforts to get all children in England swimming, inspectors have said.
Schools in the report had extra funding to improve swimming
An Ofsted report said most 11-year-olds did now reach the national curriculum target of being able to swim 25 metres.
But some groups of pupils were missing out, including ethnic minorities with little experience of swimming.
Even among schools which had received extra money to boost swimming, in one in six attainment was "inadequate".
In these schools, the key factor was lack of time devoted to the subject.
In a quarter of the 12 local authorities visited, the way that swimming facilities had been centralised meant pupils spent more than twice as long travelling to the pool as they spent in the water.
Ofsted's director of education, Miriam Rosen, said pupils recognised that swimming would help them to enjoy life, be healthy and stay safe, particularly in cases of emergency.
"Their views on the swimming programmes have been overwhelmingly positive.
"However, more needs to be done to address the barriers that prevent all pupils participating in swimming activities in order to build self-esteem, keep fit and healthy and to meet swimming standards."
Her inspectors' report focused on visits to 30 schools in 12 areas.
The schools were in 17 "school sport partnerships", given additional funding under a special initiative to boost the quality and effectiveness of swimming provision.
Most teaching was good, with teachers often being qualified swimming instructors.
Good attention was paid in most instances to ensuring that appropriate account was taken of cultural issues, the report said.
But it added: "Evidence from this small sample of schools suggested that pupils from some black minority ethnic groups often started with less experience of swimming than other groups and made the least progress.
"Providers did not always recognise or acknowledge this in their planning.
"Instances of good practice included consultation with a local mosque on how to improve participation of Muslim pupils, and family swimming programmes to involve parents from black minority ethnic groups."
Gifted and talented swimmers were often not well catered for, the report said. And the tracking of the individual progress made by all pupils was generally not well done.
"Teaching was often less effective with pupils with behavioural, emotional and social difficulties.
"The swimming teachers lacked experience and confidence with these pupils and sometimes resorted to excluding them from lessons in the absence of other strategies," said the report.
Children's Minister Kevin Brennan said the government had made "excellent progress" in its drive to make sure every 11-year-old in England could swim 25 metres.
"We introduced the top-up programme specifically to target those pupils who needed extra help, and to date 30,000 more pupils have learnt to swim as a result," he said.
"I am encouraged that well over half of pupils targeted by the national top-up programme are learning to swim and meeting the national curriculum requirement.
"This represents a real achievement since these pupils were poor or non-swimmers before."
Ofsted found however that the effect of the national top-up programme for weak swimmers had been mixed.
"In half the schools visited the impact was inadequate, so that too many pupils entered Key Stage 3 unable to swim the expected 25 metres."