Test results: what went wrong?

By Gary Eason
Education editor, BBC News website

Thousands of marks have to be entered online

"Unfortunately, because of technical problems at the company contracted by the Qualifications and Curriculum Authority (QCA) to collect and sort the 7.5 million marks, schools will receive their official computer printout of results later than planned."

An apology for delays in the delivery of the 2008 Sats results to England's schools?

No, that statement was issued in 1998.

That is not to say that this year's failure is not a huge embarrassment to all involved.

But it illustrates what schools know - and complain about - that things go wrong with the massive test marking exercise every year to some extent.

Most spectacularly, in 2004 the head of the National Assessment Agency resigned after a delay of three months in getting out accurate Key Stage 3 English results.

Online systems

It is no coincidence that the English papers are once again the ones most badly delayed.

With maths and science papers, answers are shorter and - essentially - right or wrong.

But in English, markers have to read long, handwritten answers then make subjective judgements about where they fit alongside the mark scheme for the test.

So English marking is inherently slower.

For the past three years exam board Edexcel held the marking contract.

It has invested heavily in electronic marking, although its contract prevented it from using this for marking the tests.

How it works in other exams is that the papers are collected from schools and go to one centre in Yorkshire where they are scanned into a computer system.

Markers elsewhere can then log in to the system, call up the scripts and mark them online.

An incidental part of this process is that the marks are already automatically held in the system.

When the three-year Sats marking contract came up for renewal the Qualifications and Curriculum Authority awarded it to ETS as providing best value for money, following what the authority's board minutes record as having been an exemplary procurement process.

'Ludicrously long time'

ETS does not use electronic marking.

Instead scripts are collected from schools and go to one central depot - and are then sent out by courier firm to the homes of examiners, who mark then return them.

The markers work with the paper scripts - then have to enter each mark for each answer into the ETS computer system online.

This typically means entering many thousands of separate marks.

Experienced English marker Wendy Spicer - who says she will not be doing it again - told the BBC News website:

"Reading test papers had 32 questions. Each mark had to be recorded online after being marked on the student's test paper, rather than an overall mark.

"It took a ludicrously long time to do this. I calculated that I'd made 14,000 recordings."

Quality issues

The National Assessment Agency has said it is happy with the quality of the marking if not the speed, which ultimately is what matters.

But others are not convinced.

Markers were required to go through a quality threshold in the form of a "standardisation" test, taken online.

People were allowed to make a number of errors and still get through.

Worryingly, though, e-mails from some of those involved indicate that there was a high tolerance of marking errors.

A Key Stage 3 maths team leader told us: "One member of my team passed standardisation with 29 errors on the scripts of five students. His attempts were very poor indeed...."

To counter this the regulator points to the ongoing checks built into the system, which means markers must stop after 80 sets of scripts and complete a "benchmark" exercise online.

Under the old system, they had to post samples of actual marked scripts to team leaders who assessed the quality of their marking and would feed back suggestions if necessary.

Some regard the new process as an improvement on that old bureaucracy - the drawback is that no actual marked scripts are being checked by anyone.

Problems with this year's marking process were being flagged up early on, with markers - or potential markers - concerned at apparent confusion at ETS, the company running it this year for the first time.

This was compounded by the slowness and shortcomings of responses on its telephone and e-mail helplines and confusion over training venues, then complaints that scripts were being delivered slowly or not at all.

In some cases markers did not receive the scripts they had been allocated, or received a mixture of subjects.

ETS has promised that it will learn from its mistakes this year - for which it apologises - and they will not happen again.