The vast majority of excluded pupils are boys - often with special needs
Schools can avoid excluding very young children through techniques to manage behaviour that includes running off, biting and swearing, inspectors say.
It is rare for schools to exclude those aged seven or under and those that do are usually in very deprived areas.
Ofsted compared schools in England that had done so with others that had not.
It found good relationships with parents, opportunities for children to talk together and the use of nurture groups all helped prevent exclusions.
The latest figures, for 2006-07, show there were 13,460 fixed term exclusions [suspensions] and about 260 permanent exclusions - with boys 10 times as likely as girls to be excluded.
Inspectors visited 30 schools which had excluded several young children on more than one occasion.
They also went to neighbouring schools which had not used exclusion during the same period and 12 which had excluded only one young child but on several occasions.
In the report, The Exclusion From School of Children Aged Four To Seven, Ofsted said most children "responded well to the school's expectations", but a few found this difficult.
Ofsted chief inspector Christine Gilbert said: "Exclusion of children aged under seven is still very rare.
"Ofsted inspectors found that almost all children in the schools they visited knew how to behave properly.
"Only a small number of children found this difficult but, with proper guidance and support, the need to exclude them can be avoided."
Bad behaviour included biting other children, persistent refusal to follow instructions, swearing, running away from staff or kicking or hitting them, climbing over the school fence and throwing chairs.
Often the children had experienced trauma - in the case of one refugee, seeing his mother killed.
Eight of the 69 schools visited had suspended children for behaviour which they perceived to have an inappropriate sexual element.
Most had also instigated child protection proceedings or contacted social workers.
Six other schools had witnessed sexual elements to children's behaviour but viewed the incidents instead as child protection concerns.
"Two of the schools reported a worrying lack of response from social workers and other local authority support services to their concerns about seemingly serious incidents," the report said.
Ofsted, in its recommendations, said the government should "produce urgently, guidance for schools on identifying and responding to sexually inappropriate behaviour in young children".
Local authorities should also take referrals seriously, it said.
Ofsted also called for guidance for governors about the exclusion of young children so that they could respond in an informed way to the level of exclusion in their own school.
Sometimes schools had experienced instability themselves so management had been inconsistent, and exclusions might be a symptom of a new head "drawing a line in the sand".
Ofsted said a school's philosophy, a supportive and stable environment and strong relationships with parents were important factors in preventing exclusions.
Circle Time - regular meetings in which children talk and listen to each other - was beneficial, as was the national "social and emotional aspects of learning" (Seal) programme.
In particular, so-called nurture groups "were highly effective in improving children's behaviour and preventing exclusion, but many of the schools said they were unable to afford them", the report said.
These are small supportive gatherings of up to a dozen children, usually located in a mainstream primary school.
Begun more than 30 years ago, they focus on social and emotional as well as academic development.
The government supports the idea - but says local authorities should pay for them.
Schools Minister Vernon Coaker welcomed the fact that the report showed that most infant and primary schools do not exclude any children aged four to seven.
But he added: "Where pupils continuously disrupt classes or are violent, my department is committed to backing head teachers' authority to exclude even for a 'first' or 'one off offence', this also applies to primary schools.
"Exclusion must continue to be used as a last resort and the government is investing in wide-ranging measures to improve standards of behaviour and to develop and encourage the use of alternatives to exclusion."