[an error occurred while processing this directive]
Front Page | In depth | Education
INTERIM REPORT FINAL REPORT PART 1 PART 2 PART 3
INTERIM REPORT


Most computers will open this document automatically. You may need to download Adobe Acrobat Reader to open this PDF document.


On 27 September the interim report of the Tomlinson inquiry into this year's A-level results was published. Read the full text of the report below.

INTRODUCTION

1. This initial report into the setting of the A level standards in July 2002 results from the independent inquiry I was asked to undertake by the Secretary of State on the 19th September. This follows concerns expressed publicly by a number of headteachers of independent and maintained schools following the publication of results.

2. The precise Terms of Reference for the inquiry are:
1. To investigate allegations about the setting of standards for A-level grades this year. In particular, to make sure that the conversion from marks to grades was determined according to proper standards and procedures. A first report on this will be provided to the Secretary of State by Friday 27 September.

2. To investigate the arrangements at QCA and the awarding bodies for setting, maintaining and judging A level standards, which are challenging, and ensuring their consistency over time; and to make recommendations by November to the Secretary of State and Ken Boston, Chief Executive of the QCA, for action with the aim of securing the credibility and integrity of these exams.

3. This report is concerned only with the first point. I must stress that throughout the inquiry and in the preparation of this report the paramount consideration has been those students who took GCE A-level examinations this year, their parents and their teachers. Even if the concerns expressed relate to a proportion A-level candidates this year, all do deserve to be awarded a grade commensurate with the standard of their work and one which is consistent with the standard employed in previous years. I have not therefore regarded my inquiry as an administrative exercise. Rather I have regarded it as vital that the anxiety and concern felt by many students were if possible, resolved quickly and conclusively.

4. My inquiry has taken written evidence from a wide variety of sources and has included face to face meetings with the Headmasters and Headmistresses Conference (HMC), Secondary Heads Association (SHA), National Association of Head Teachers (NAHT), the teacher and lecturer associations, the Association of Colleges (AoC), Chief Executives of three examination boards in England, a sample (8 in total) of Chief Examiners, officials from the QCA, including the Chairman and the former Chief Executive Professor Hargreaves, officials from the Department for Education and Skills and Ministers.

5. I must record my sincere thanks to everyone for their openness and their readiness to provide me with data, papers and notes of meetings. I also wish to thank all the headteachers, teachers and others who have submitted written evidence. I intend making evidence available in due course, where no desire for confidentiality was expressed by the contributor.

6. Finally, I would wish to record my thanks for the exceptional hard work of the colleagues who have supported me in this inquiry: Lisa Couchman, Darren Goff, Mark Hayward, Nancy McLean, Peter O'Connor, Kate Taylor, and Matthew White.

^^ Back to Top

RECOMMENDATIONS

7. I recommend to the Secretary of State that:

^^ Back to Top

BACKGROUND TO THE DEVELOPMENT AND STRUCTURE OF THE CURRENT AS/A LEVEL SYSTEM

8. The current "Curriculum 2000" reforms of the A level system were introduced in September 2000 with the first AS and A level awards being made in summer 2001 and summer 2002 respectively. The principles behind "Curriculum 2000" were wholeheartedly endorsed during my inquiry.

9. Because the structure of the new A level and the manner of it's implementation have emerged as significant issues in the course of my inquiry, paragraphs 10 to 12 below set out the essential characteristics of the new system.

10. The current A level is divided into two parts: three units at AS level which, together, equate to the first year of a traditional A level course and three A2 units awarded during the second year of study. Taken together these six units comprise a full GCE A level and form the basis for an A level award. The three units studied in the first year at AS level can, if the student wishes, be "cashed in" to provide a certificated qualification in its own right. Each unit of the award is equally weighted, with the AS and A2 programmes each accounting for 50 per cent of the overall grade.

11. This system was established with the intention that students would take a broad range of AS level courses during the first year of study – up to four or five. They are then able to narrow their studies in the second year by selecting the subjects which they will pursue to the full GCE A level standard, whilst receiving a qualification for subjects they pursue no further. Students may also retake units to seek to improve their grade.

12. These design features might reasonably have been expected to lead to an increase, compared to the former "legacy" A levels, in the proportion of full A-level candidates who achieved the GCE A level standard without any change in the overall level of demand of the qualification.

13. Some aspects of the system were, from early in the implementation phase, causing concern to some people involved with the system, including at least one QCA board member (who later acknowledged that note had been taken by QCA of his major concerns). For instance, I have collected evidence that concerns were expressed, including at QCA seminars, from 1998/99 about:

a. the difficulty of, but vital need for, setting appropriate standards for the AS and A2 units as a consequence of giving equal weighting to all units, even though they require differing levels of demand;

b. the complexity of the aggregation processes needed to achieve the established A level standard, including the statistical challenges.

14. The evidence suggests overwhelmingly that there is no clear, consistent view among awarding body officials and many examiners and teachers about the standard required at AS and A2 unit levels in order to ensure that the overall GCE A level standard is maintained. This concern had been identified specifically in relation to AS in the review of Curriculum 2000 carried out by Professor David Hargreaves, then Chief Executive of the QCA.

15. This clearly has created a risk that differing interpretations of the two standards exist. Such differing interpretations were strongly evident during my hearings and in written evidence.

16. Crucially, in order to reflect the different maturity levels of the candidates during the first and second years of their study, AS and A2 could be expected to have different levels of demand. My hearings and the written evidence revealed some common understanding that AS should be set at a level of demand less than the overall A level, to reflect candidates' educational development and maturity during the first year of an A level programme; and that at A2 the level should be higher - possibly higher than the A level Standard - to reflect candidates' developing capacity during the second. The formal position is set out in QCA's publication, Managing the Curriculum 2000 for 16-19 Students, that A2 should be more demanding than the overall A level standard. In spite of this, there would appear to be no common understanding on how much greater the demands of A2 units should be compared to AS.

17. QCA's subject specific grade criteria setting out the level of attainment which justifies the award of a particular grade in each subject reflect only the level expected across the award as a whole, and do not relate to the assessment standard nor the actual attainment expected of students in any one single unit. This absence of a common standard for AS or for A2 contributes significantly to a lack of common understanding among those involved in teaching and examining GCE A levels.

18. AS units were piloted on a limited basis. A2 units were not, for reasons I have not had time to ascertain. Therefore, before this summer, there was no practical experience or relevant scripts to aid the grading process, or to illuminate the challenges of the new grading and aggregation process across the GCE A level as a whole. This resulted, in part, from the speed of implementation of the policy as determined by Ministers.

19. Evidence has been offered to me that not all the practical and statistical issues implied by the new system had been fully and comprehensively understood and worked through at critical stages in the development and implementation process by the DfES, the QCA and Examining Boards. For instance, I would have expected much more comprehensive statistical modelling than actually took place, particularly of the impact of differing assumptions about the patterns of student choice and attainment before introduction and after AS awards in 2001.

^^ Back to Top

THE GRADING PROCESS IN 2002

20. The A level, GCSE and GNVQ Code of Practice governs the process under which GCE A level awards should be made. A distinction must be made between the marking of work, in which examiners award marks to individual candidates for each unit, and the subsequent grading, during which those marks are translated into grades for each unit and ultimately for the subject as a whole.

21. Under the QCA's Code of Practice, for each unit within a syllabus, an Awarding Meeting, chaired by the Chair of Examiners, is responsible for establishing preliminary grade boundary mark ranges for A and E for each unit and for recommending the grade boundaries for A/B and E/U for each unit. These recommendations need to be approved by the "accountable officer", usually the awarding body Chief Executive. Through that mechanism consistency in the standard of the award is maintained. Scrutineers under contract to the QCA attend some awarding meetings, but play no active part, in order to monitor application of the code of practice and report back to QCA. The meeting:

22. The meeting considers candidates' work and compares it to that from previous examinations, where possible, to ensure comparability, and reviews statistics provided by the Awarding Body which are designed to assist the Chair of Examiners in making a formal recommendation. This year more data were available than in previous years, including for the first time analyses of predicted A level grades based on candidates' prior GCSE results. These additional data were welcomed by Chairs of Examiners who found them a helpful additional source of information.

23. The meeting recommends A and E grade boundaries; other boundaries are then arithmetically determined.

24. The awarding body Accountable Officer (AO) is responsible for reviewing the boundaries recommended by Chairs of Examiners with a view to ensuring consistency of standards between subjects and awarding bodies and over time.

25. Different boards have different processes to support the AO in making these judgements, but under the Code, the responsibility rests in all cases with the AO. In making their decisions, AOs do not generally consider students' actual work, but they take account of all other relevant information.

26. The evidence put to me indicates variations in the ways these tasks were approached within the three English awarding bodies this year.

^^ Back to Top

AWARDING MEETINGS

Oxford, Cambridge and the RSA Examinations (OCR):

27. I received evidence from discussions with three OCR Chairs of Examiners that they believed that in making their recommendations they were expected to have very strong regard to the grade distributions that emerged from the previous year's A levels, and that these recommendations should result in a distribution close to the previous year's profile. I have also received evidence from one QCA scrutineer who recorded both in his report of the awarding meeting and in evidence to me that the weight given to statistical evidence compared with that given to professional judgements based on the actual work of candidates was different this year. This may not be wholly unexpected given that there were no archive scripts for A2 units. It is a question as to whether or not the balance adopted was appropriate. I am satisfied that there is evidence that there was a perceived pressure to deliver outcomes in line with those of the legacy A level in 2001. This in at least one subject attended by a QCA scrutineer resulted in the grade boundary decisions being "very much a statistically driven operation". The extent to which this occurred in other subjects is not known precisely to me at this time.

28. The Code of Practice does not specify the balance expected between the various sources of information available to examination boards. However, I am persuaded that consideration was given in all cases to candidate's work. I do not therefore conclude that the committees and Chairs acted outside the boundaries of the Code of Practice. I am concerned however about the evidence that the process leading to some grade boundary recommendations within OCR, while seemingly permissible under the Code of Practice, may have given undue weight to historical and other statistical data.

Edexcel

29. Evidence from Edexcel Chairs of Examiners indicated a significant and proper emphasis on the place of assessment of students' work in forming their grade boundary recommendations, whilst acknowledging the relevance of statistical data in their judgements. I have received no verifiable contrary indications from other sources.

Assessment and Qualification Alliance (AQA)

30. AQA Chairs of Examiners gave broadly similar accounts of their role and the expectations upon them, and were clear that their deliberations had given due weight to professional judgements based on candidates' actual work.

Accountable Officer decisions

31. Within Edexcel, consideration of the Chair of Examiner's recommended grade boundaries is a two stage process. First, a Grade Evaluation meeting consisting of senior awarding body officers considers the recommendations and advises the AO on any changes. The AO then makes a decision based on the Examiners' recommendations and this further advice. For the 2002 GCE A levels, this process resulted in little change to recommended grade boundaries; and most changes were of the scale of one or two marks up or down and within the range recommended by the Chairs of Examiners with the exception of Arabic and Russian, which were subsequently agreed with the Chairs of Examiners after the GEM.

32. A similar pattern is evident from AQA. In this awarding body the AO receives and considers recommendations from the Chairs of Examiners without an intervening advisory stage. It is usual practice for proposed changes to be referred back to the relevant chief examiner, which have in the first instance been considered in detail my a senior officer of the board. There is no further intervention at the advisory stage. Data provided by the awarding body suggest a similar pattern of grade boundary changes, with a total of 53, of which 26 reduced the recommended mark. In only three units were the changes to recommended mark grade boundaries as much as 6 marks.

33. In OCR, like Edexcel, there is an intervening advisory stage – Grade Evaluation meetings (GEM) – which are held before the AO confirms the grade boundaries. I have received clear statistical evidence from the awarding body indicating that in a significant number of units the GEM raised some of the relevant grade boundaries, and the AO raised some of them yet again.

[ Footnote: In total, 423 changes to unit mark grade boundaries were approved by the AO, of which 25 saw the boundary marks reduced. In total, some 86 changes were of 6 marks or more, with the maximum being 13 marks in one case. ]

Evidence from chief examiners in geography, French, German, Spanish, history, psychology and government and politics expressed their concerns with these changes. The same evidence also indicates that in many cases the chief examiners were not made aware of these changes. Their main impact has likely been on the proportion of students passing at grade E, but this in itself has a knock on effect on the higher grades.

34. From the evidence I am clear that all three accountable officers acted within their powers under the Code of Practice. I am, however, concerned that the OCR AO judged that his duty to maintain the standard made it necessary to lift grade boundaries in a significant number of units so substantially in the light of statistical evidence. He is quite clear that his actions were entirely based on a view that the requirement to meet the A2 standard had been underestimated by some examiners.

35. At this stage I should point out that I have also received unsolicited endorsements of this year's processes from a number of schools and examiners. This is an important counterweight to some unjustified perceptions that the whole system is failing to function.

36. The evidence put to me suggests there may be a lack of consistency in practices across the three English awarding bodies in the grading process. In the time available it has not been possible to investigate more fully this matter. I intend to do so in part two of my inquiry.

^^ Back to Top

WAS PRESSURE BROUGHT TO BEAR ON THE AWARDING BODIES?

37. In considering allegations of external pressure I have been mindful of the context in which the A level system operates and in particular the emphasis on year-on-year comparisons of results which looms large in public debate about standards. That undoubtedly creates an ongoing pressure within the system as a whole and raises sensitivities among all concerned in relation to the maintenance of standards. I have been careful to distinguish this type of ongoing background noise from the possibility of distinct and identifiable actions which might have added to this pressure this year.

Meeting between the awarding bodies and the QCA, 12 March 2002

38. This meeting was one in a series of regular and legitimate meetings to discuss ongoing business between the English, Welsh and Northern Irish awarding bodies and their regulators, QCA in England, ACCAC in Wales and CCEA in Northern Ireland.

39. Arising from the 12 March meeting, all three chief executives of the English awarding bodies reported to me that the Chairman of QCA had made clear to them what he expected of the awarding bodies this summer in relation to the grading of GCE A levels. Their perception was that they were being asked to give more emphasis than perhaps was proper to statistical data and the need to have an overall outcome similar to the 2001 legacy A levels.

40. Both the awarding bodies and the QCA Chairman reported specifically the latter's comment that there should be no "grade inflation" or "grade drift". Both of these terms are used loosely to describe more students passing and achieving higher grades year on year, either as a result of changing standards in the qualification itself, or in the level of performance of students. In his evidence to me, the QCA Chairman was clear that he was referring to the former interpretation. I am equally clear that guarding against better grades based on lower examining standards is a wholly proper and necessary concern of the regulator.

41. Following this meeting, the awarding bodies were sufficiently concerned to write jointly to the QCA Chairman setting out their view of their responsibilities with regard to the awarding process and seeking his endorsement that this view was in line with the Code of Practice. This he provided in a letter dated 19 April which indicated clearly and properly that "in this summer's A level awards, the change to new specifications means that awarders have less evidence to assist them than in normal circumstances. In this situation, I do expect last year's A level results to provide a very strong guide to this year's outcomes" and concluded that "I am clear that grades for this summer's A level candidates can only be determined using a combination of professional judgements and statistical evidence."

42. In at least one board, however, the earlier perception remained, as is revealed in an internal note dated 29 April to OCR officers from the OCR Chief Executive, which said of the 19 April letter: ‘This latest letter is very carefully worded – but the underlying message has not changed. "Very strong guide" means something rather less than the most recent annual increases in percentage achieving A or E!'

43. There is evidence that in more than one board there was a perception articulated by Chairs of Examiners during awarding meetings that there was pressure being brought to bear for the candidates' results in their subjects to be close to those for the 2001 legacy A levels. In one case this perception led the QCA scrutineer present to contact QCA headquarters to check the position. He understood the perception was confirmed but I have been unable to check elsewhere the substance of that conversation. The scrutineer (one of 19) subsequently resigned in order to be able to make his concerns known.

26 July meeting between QCA and the awarding bodies

44. This meeting was requested by the chief executives of the awarding bodies in order to brief the QCA on provisional and emerging A level results in a limited range of subjects. These were revealing an increase in the pass rate and in the proportion of candidates achieving an E grade in 2002 compared to previous years.

45. As explained earlier in this report, wholly legitimate improvements in candidates' performance might have been expected as a result of the new AS/A2 system, in part because the A2 cohort in a subject might change because those who had achieved a weak AS grade might not proceed to A2, and in part because of the ability to re-sit units. Papers I have seen also indicate that the subject grade profile for the new A levels was more affected by the statistical phenomenon known as regression to the mean arising out of the modular nature of the course, is taken into account in the grading process.

46. The Chairman of QCA is clear that at the meeting, on the basis of these data, he sought reassurance from the awarding bodies that their awarding processes had maintained the standard. He also indicated that it might be necessary to inquire into the standard which had been applied. I am clear this is a wholly proper position for the Chairman to take in his role as regulator.

47. The chief executives, however, had different perceptions of the implications of this approach. In varying degrees they all believed they were being put under further pressure. For instance, the internal note of the meeting from Edexcel representatives reported to their chief executive that in their view "Sir William Stubbs was not pleased with these outcomes…", and that he "indicated that if we went ahead with these predicted outcomes he would announce an independent enquiry into the awarding bodies and the Curriculum 2002 results." Statements to me from the relevant Chief Executive in the course of my inquiry confirm that this was interpreted within Edexcel as a threat.

48. Following the meeting an internal OCR e-mail gave another chief executive's view that "following his meeting with Joint Council colleagues and the QCA": "For any remaining AS/A2 awards, his [the OCR Chief Executive's] advice is to try to get as close as possible to the putatives grades predicted on the basis of candidates' GCSE results at ‘E' – the phrase he used was to push the boundary ‘until it squeaks' – but not at the expense of alienating awarders."

49. I am unable fully to resolve the clear differences in perception of what was said at the meeting.

50. The QCA explained that no official record of this meeting was made immediately. The record which I have been given has been constructed this week from notes taken at the time by individual QCA officers present.

51. I have taken written and oral evidence from Estelle Morris, David Miliband and relevant DfES officials. My inquiry has heard no evidence that Ministers or officials offered any pressure or guidance on the grading process or the final outcomes of this years examinations; nor was any present in the notes of meetings between Ministers and QCA and officials and QCA. I therefore conclude that there was none.

^^ Back to Top

CONCLUSIONS

^^ Back to Top

SCHEDULE OF WITNESSES

During the course of my inquiry I took oral evidence from:

Head Masters Conference
Girls' Schools Association
Secondary Heads Association
National Association of Head Teachers
National Union of Teachers
NASUWT
Professional Association of Teachers
Association of Teachers and Lecturers
National Association of Teachers in Further and Higher Education
Association of Colleges
The Chief Executives of the three English Awarding Bodies: AQA, Edexcel and OCR
OCR Chairs of Examiners for English, Psychology and History
Edexcel Chair of Examiners for Geography and Chief Examiner for Psychology
AQA Chairs of Examiners for mathematics, psychology and physics
Estelle Morris, Secretary of State for Education and Skills
David Miliband, Minister for School Standards
Sir William Stubbs and Ken Boston, Chairman and Chief Executive of the QCA, accompanied by QCA officers
David Hargreaves, ex-QCA Chief Executive
John Adnit, QCA scrutineer
Celia Johnson, DfES
Rob Hull, DfES

I have also received written evidence and submissions from:

Head Masters Conference
Girls' Schools Association
Secondary Heads Association
National Association of Head Teachers
National Union of Teachers
Association of Teachers and Lecturers
National Association of Teachers in Further and Higher Education
Association of Colleges
AQA
Edexcel
OCR
Department for Education and Skills

In addition I have received and am continuing to receive helpful submissions from a very wide range of individuals, schools, teachers, examiners and others. In the time available it has not been possible to catalogue all of that. I will in due course make available a supplementary list.

^^ Back to Top
 © MMV | News Sources | Privacy