Page last updated at 12:38 GMT, Tuesday, 16 December 2008

Q&A: Sats results problems

ETS screengrab
The ETS marker system is web based

What are the Sats?

Properly called the national curriculum tests, they are tests all children in England are required by the government to sit at the end of "key stages" in their schooling.

Specifically these are Key Stage 2 - the end of primary school, when they are aged 10 or 11 - and until this year's problems also at Key Stage 3, after three years in secondary school when they are 13 or 14. Those tests have now been scrapped.

The tests are designed to measure their understanding of the national curriculum in English, mathematics and science.

Everyone takes them on specific days in May. Children are awarded a level.

According to the Qualifications and Curriculum Authority (QCA), most seven-year-olds will be at Level 2, most 11-year-olds at Level 4 and most 14-year-olds at Levels 5 or 6.

Who administers them?

The National Assessment Agency (NAA), part of the QCA, is responsible for the tests.

It contracts the process to an external awarding body or exam board.

Formerly it was Edexcel. This year the contract had to be renewed under EU rules.

A lengthy procurement process with initially five bidders resulted in the contract going to a US firm, Educational Testing Services or ETS.

How are the tests marked?

Schools bundle up the pupils' test scripts and send them off for marking.

In previous years they have gone directly to their designated markers.

ETS instead required them to be sent to a central depot - in Dewsbury - where they were split up then taken by courier to the markers it had recruited - who had to be available to sign for them.

It did this to remove any suggestion of a direct relationship between school and marker, and to avoid issues of pupils' scripts being left lying around outside people's homes.

But there was a delay in providing markers with scripts and in some cases scripts were sent to the wrong markers.

Who are the markers?

For the most part they are practising or retired teachers - and largely the same people this year as in previous years.

They are given training on the actual tests that are being given to the children, and assessed themselves on how well they can administer the mark schemes against which they have to gauge children's efforts.

What went wrong?

For months, markers and would-be markers have been signalling that there were problems with ETS.

Initially these related to communication problems in the recruitment and training and assessment processes - such as being given late notice of training venues a long distance from their homes, and not being able to get responses from ETS's telephone or e-mail helplines.

Some experienced markers got fed up and dropped out.

ETS had in fact wanted to do all the training for the maths and science markers online, not at face-to-face meetings.

In a letter to the Children, Schools and Families Secretary, Ed Balls, in March, the chief executive of the QCA, Ken Boston, said: "The marking pilot study showed that some markers were not entirely comfortable with the training materials delivered online and so, after careful consideration, the NAA decided not to proceed with this innovation for the 2008 tests."

One innovation that was adopted was "online mark capture".

This meant that every mark that a marker put on a pupil's script then had to be entered against that pupil's name and the relevant question online via an ETS website.

Markers reported this to be enormously time-consuming, and problematic in cases where pupils were not showing as having been registered to take the tests.

Dr Boston also said in his letter: "We are aware that the introduction of a new IT-based improvements must carry risks, but we have mitigation plans to ensure delivery is secure.

"This includes extensive load testing of the systems and having markers access the system in advance of the marking period."

In reply Mr Balls wrote on 30 March that he was encouraged to hear about the improvements in marking quality that the NAA was planning to introduce.

"Online systems, although they confer many benefits, are also subject to risk when first introduced," he said.

"I welcome your strategy for mitigating the risk to the tests and I would like to be kept informed of its effectiveness over the next few months."

Online mark capture was supposed to speed things up and automate the totalling of marks and allocation of test levels.

Dr Boston later told a committee of MPs: "There have been however, despite the pilot, when it was scaled up, some very significant problems in that marking was going faster than the data was able to be processed."

What was being done about this?

When the NAA realised the process was at risk it initially advised ETS to set up two emergency marking centres and recruit extra markers to staff them - eventually there were seven such centres around the country.

ETS offered an "incentive payment" of £100 to markers completing their allocation of scripts and submitting the marks online by the deadlines of Monday 23 June for Key Stage 3 and Thursday 26 June for Key Stage 2.

NAA staff dealt with a backlog of 10,000 e-mails ETS had failed to answer, and set up an extra call centre to handle queries.

Ken Boston said problems were being dealt with as they emerged.

But it was too late?

On 27 June ETS finally told the QCA it would not manage to deliver all the results to schools by 8 July as it had been contracted to.

On 4 July the NAA announced that the Key Stage 2 results would be delayed until 15 July. But many schools did not get all their results even then.

Marking of Key Stage 3 results would not be complete by then, but available results would be released by the end of that week so as many schools as possible had them before the end of term.

As it turned out, up to 29% of pupils' results were not available.

Other problems emerged. Groups of children were recorded on the ETS system as having been absent when the tests were taken - even though they did sit them.

Some schools had scripts returned unmarked. Some scripts had been sent to the wrong schools.

Some markers had been unable to input marks into the ETS system.

Questions were raised about the quality of some of the marking, though the regulator insisted the built-in checks meant things should have been better this year than before.

Does it all matter?

Yes, at different levels.

For one thing, head teachers have a statutory duty to provide the results to parents along with their children's end-of-year reports.

Some say the tests are an irrelevance to pupils, as teachers make their own assessments anyway - and, incidentally, still have a duty to report those to the authorities.

In other cases, schools use the results as part of the process of deciding which sets pupils will be in for the following year, or which GCSE subjects they will be allowed to take.

What makes them most controversially "high stakes" is that Ofsted inspectors judge schools by their results - and the scores are published in the government's annual performance tables, which the news media use to rank schools.

It is clearly hugely embarrassing for ETS, the NAA and the QCA, and the Department for Children, Schools and Families which insists on children taking the tests - and is quick to "name and shame" schools for any shortcomings on their parts.

The QCA terminated ETS' contract and has appointed Edexcel to run next year's Key Stage 2 tests.



Print Sponsor


SEE ALSO

RELATED INTERNET LINKS
The BBC is not responsible for the content of external internet sites


FEATURES, VIEWS, ANALYSIS
Has China's housing bubble burst?
How the world's oldest clove tree defied an empire
Why Royal Ballet principal Sergei Polunin quit

BBC navigation

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.

Americas Africa Europe Middle East South Asia Asia Pacific