Too many schools in England do not take children on outdoor activities, partly through fear of being sued if there is an accident, Ofsted inspectors said.
Expeditions typically encourage teamwork
Activities helped pupils develop their physical and social skills, they said.
But many youngsters were missing out, with costs and limits on the numbers who can go contributing to the problem.
The chief inspector, David Bell, said teachers had "nothing to fear" if they followed the right guidelines - but teachers' unions disagree.
The Ofsted report was based on inspections of primary and secondary schools and reports on physical education, plus visits to 15 outdoor education centres.
There, the quality of teaching was good or better in 80% of the sessions - very good in almost a third.
More might be done to assess the longer-term impact on students.
But generally they made good progress, developing physical skills and social skills such as teamwork and leadership.
Their behaviour was "often exemplary, with mature responses to challenging situations".
In one case, a secondary school decided its curriculum for 15 and 16 year olds was too academic for the least able, leading to disaffection and absenteeism.
With a local further education college it devised a programme on which students spent five days at an outdoor education centre.
They had to share the domestic chores - a novelty for many - and rely on each other in the outdoor sessions.
"Before they went to the centre, if they were unhappy with any aspect of school life they did not object, but simply opted out and stayed at home," Ofsted said.
"After their residential experience, they talked more, sharing feelings and opinions with each other and with their teachers."
Relationships improved and teachers revised their courses to offer the youngsters a curriculum that met their needs.
But the report said most students did not get such opportunities.
"Often, the extra-curricular nature of the activity, its cost or limits on the numbers that can be taken, lead to a 'first come, first served' basis for selection."
Some schools remained unconvinced because of pressures on curriculum time, lack of specialist expertise, concerns about taking risks and fear of litigation.
The chief inspector, David Bell, said: "The benefits of outdoor education are far too important to forfeit, and by far outweigh the risks of an accident occurring.
"If teachers follow recognised safety procedures and guidance they have nothing to fear from the law."
But teachers' unions say this is not so. One of the biggest, the NASUWT, advises its members to avoid school trips.
Its general secretary, Chris Keates, said Mr Bell had failed to grasp "the reality of what actually happens when accidents occur".
"As NASUWT casework has demonstrated time and time again, following the procedures and guidance is no protection against litigation.
"Fortunately, the government is now taking our concerns seriously, having recognised that the demise of the concept of the genuine accident and the rise of the blame culture has left teachers and schools vulnerable," she added.
Talks were going on to analyse the problems and identify possible solutions to protect staff.
These followed a pledge by the Education Secretary, Charles Clarke, to the union's conference in April, when he said: "I am a firm believer in being able to offer children some form of residential experience."
A spokesperson for his department said: "We believe our guidance on outdoor activities achieves the right balance, by helping schools assess and manage the risks so that the educational objectives of the visit can more easily be achieved."
Accident claims had declined in 2003-04, and there was little evidence to support the idea that compensation claims were rising in the UK.