One in eight pupils 'wrong level'

By Gary Eason
BBC News education correspondent

The system depends on consistently correct levels being awarded

At least one in eight pupils in England may have been given the wrong test level in their primary school science "Sats", a reliability study has said.

The study, commissioned by regulator Ofqual, compared the results of actual science tests with test development papers using the same questions.

It found 83%-88% of pupils probably were given the correct level. Another study suggested 84% for English tests.

The NUT teacher union said schools were right not to trust the test results.

Ministers have now scrapped the "Sats" - national curriculum tests - in science.

But they still require all pupils aged 10 and 11 to sit English and maths tests at the end of Key Stage 2 - the latter part of primary school.

The report, which looked at evidence over five years, was commissioned by Ofqual from the National Foundation for Educational Research (NFER).

The problem they were addressing is that ideally pupils would sit the exact same questions each year, so it would be obvious whether standards in the test were rising or falling.

In the real world this is not possible and the tests have to be changed annually, raising questions about reliability.

Consistency

But "pre-test" questions are taken by a representative sample of pupils every year and then form the actual test in the following year - which does allow for a direct comparison.

Also, so-called "anchor" tests taken by another sample do actually use exactly the same questions year-on-year, as a check.

So, because the questions are the same, the proportion of children being awarded the various test levels should also be the same - thus providing a measure of the consistency of the results.

They then went one stage further, because "consistency" does not mean the levels awarded are necessarily correct - they might be consistently wrong.

So the researchers calculated the "correctness" of pupils' results using a probability formula.

"These results show that between 83% and 88% of pupils would be likely to be given the correct level when sitting the tests described in this study," the report said.

'Policy matter'

The figure improved, from 83% in 2004-05 to 88% in 2006-07 and the following two years.

Another study from NFER earlier this year considered just one year's worth of Key Stage 2 English tests and pre-tests in reading, writing and the subject as a whole.

For these tests the classification consistencies were: reading, 73%, writing 67% and English as a whole, 73%.

The consistency of the English tests could also be converted to an overall "correctness" - which was 84%.

Ofqual's programme of reliability studies is ongoing and it has no official view as yet on the findings.

The report concludes that the tests achieve "as high a value of consistency as might be expected from tests of this type".

"Whether this level of consistency is satisfactory is a matter of policy," it adds.

Christine Blower of the National Union of Teachers, which is campaigning for the abolition of the tests, said: "These results are worryingly unreliable".

If such high stake test results were out by so much, schools had every right not have any confidence in them, she said.