BBCi NEWS   SPORT   WEATHER   WORLD SERVICE   A-Z INDEX   SEARCH 

Front Page | In depth | Education
INTERIM REPORT FINAL REPORT PART 1 PART 2 PART 3
FINAL REPORT - PART 3


Most computers will open this document automatically. You may need to download Adobe Acrobat Reader to open this PDF document.


On 3 December the full report of the Tomlinson inquiry into this year's A-level results was published. Read the full text of the report below.

PROFESSIONALISATION OF THE EXAMINATIONS SYSTEM

120. Any system is only as good as those operating it. During my inquiry I have looked closely at the supply and status of two key groups whose skills and professionalism are essential for the examination system to operate effectively.

The supply and training of examiners

121. Examiner shortages are not a new problem. But are a growing cause for concern, not least to the English awarding bodies. With the introduction of Curriculum 2000, the number of examination entries more than doubled, compounding an 8-fold increase in subject entries between 1951 and 2001. There was a 41% increase in GCE examiners (across the UK) between 1999 and 2002 and even that is thought to be insufficient to meet the demands of Curriculum 2000 with its modular structure and expectation that students will take more subjects.

122. Moreover, it is unclear what impact this year's problems with A levels will have on the supply of examiners in 2003. There are some concerns that the result will be a shortage owing to demoralisation and examiners' unwillingness to associate themselves with a system in which they have lost faith.

123. In the short term, it is necessary to guarantee examiner capacity for January and June 2003. The QCA-led Examinations Taskforce is currently considering a variety of measures to secure the supply of examiners for next year. These include extensive use of marking centres, release of teachers from their regular school or college duties (something some institutions already allow voluntarily, supported by the existing teacher release scheme), engaging the teacher associations and reviewing examiner pay.

124. The Taskforce's work should also build on the findings of pilots conducted by Edexcel and OCR involving the use of graduates - particularly new PGCE graduates - as examiners. The Edexcel pilot used PGCE graduates to address a shortage in GCSE history examiners, while OCR relied on graduates with relevant subject expertise to redress a shortfall in ICT examiners.

125. Contrary to received wisdom that examiners should have lengthy classroom experience, the PGCE and graduate examiners were found to mark consistently and effectively. In the Edexcel pilot, the performance profile of the PGCE graduates was the same as for all examiners in 2002 - with 83% of A- or B-grade performance - and better than that for other new examiners. Factors contributing to this success were the extra training and regular monitoring the PGCE graduates were subject to; their deployment to specific papers; and the support they received from the senior examiner team. The initial allocation of papers they received was however about 25% lower than usual for new examiners to minimise pressure and risk, and the benefits need to be balanced against the cost and administrative implications.

126. At least one awarding body is already planning to make extensive use of PGCE graduates to address the examiner shortage for 2003.

127. In the OCR pilot, GCSE foundation scripts were marked by newly-graduated students in a supervised environment. The pilot met with great success, the level of accuracy being higher than for the teams of assistant examiners who marked at home in the normal manner. The reasons for this were considered to be:

  • the ability to confer or seek guidance, which added significantly to the consistency of the marking;

  • the controlled and supervised environment in which marking took place. This was seen to improve the quality of checking and administrative compliance.

128. On the basis of the evidence presented to me, and subject to rigorous supervision and quality assurance procedures, I am confident in fully endorsing further development of this approach to the marking of public examinations.

The supply and training of examiners - longer-term action

129. In the longer term, I believe that the key to securing the supply of high quality examiners is to raise the professional status of what they do and to ensure that they, and the schools and colleges which provide them, are recognised for participation in the examinations system.

130. Professionalisation of marking and examining, including the marking of coursework, would have clear system-wide benefits. In my inquiry, I have found wide support for professionalisation, including from the awarding bodies and teachers' associations.

131. Professionalisation promises benefits for both institutions and individuals in the potentially greater recognition and rewards for both examiners and the institutions supplying them. Over the longer term, this could enable some centres or federations of centres to earn the right to take more control over assessment, enhancing perceptions of the reliability of internal assessment models, which would at the same time reduce the pressures on the public examination system.

132. The availability and take-up of good quality training is essential to professionalising the examining workforce. I am concerned by anecdotal reports received in conducting my inquiry about the quality of training and the attendance of staff from centres. I believe that additional resources are needed to tackle these issues, and that take-up of available training should be a necessary part of any schemes to enhance teachers' career prospects and status as a result of their participation in the examination system.

133. I am equally concerned by anecdotal reports about the variable quality of the training available. The QCA, working with the Joint Council and other relevant bodies, like the NCSL, should have a role in ensuring training provided is fit for purpose, timely and of a good standard. This should include making use of the exemplar material and other materials currently being gathered by QCA to define AS and A2 standards. Further efforts should also be made to offer common provision as far as possible, with bolt-on sessions offering training specific to awarding bodies as necessary.

134. I recommend that the use of graduates, and especially PGCE graduates, as markers in public examinations should be extended, subject to effective training, supervision, evaluation and public reporting.

135. I recommend a thorough professionalisation of the role of markers and examiners, including coursework markers. Implementation should be taken forward swiftly by the Secretary of State in collaboration with the regulatory and awarding bodies and the teaching profession, and should examine:

  • linking service and training as an examiner to professional recognition and career and reward structures, including through provision made by the National College for School Leadership (NCSL) and the proposed Post-16 Leadership College;

  • improving the take-up of training, including by making available central government funding, distributed through the schools and FE Standards Funds or other appropriate funding mechanisms. Provision should also be made for examiners supplied by the independent sector;

  • improving the quality of training, through rigorous accreditation procedures, drawing on work already underway within the awarding bodies to develop a qualification for examiners;

  • working towards implementation of proposals, like those put forward by the Secondary Heads Associations for 'Chartered Examiners', which would establish a high status professional role for experienced examiners, and would give the schools and colleges in which they are based the right to take more control over examination assessment.

136. In addition, I also recommend that schools and colleges should be rewarded with "beacon" or similar status for significant contributions to the examination system, including examiner recruitment and administrative efficiency. As well as kudos derived from achieving a publicly recognised symbol of excellence, this could bring with it a lower level of scrutiny in relation to the administration and conduct of examinations, and possibly, financial or other rewards.

Examination officers

137. Examination officers have a critical role to play in the effective administration of the system. Perceptions of this role vary widely and it may be the responsibility of teaching or administrative staff. What has been clear in the evidence submitted to my inquiry is that the role of examination officer is time-consuming and cannot be done without training and mastery of the variety of systems used.

138. There is a need for common training across the awarding bodies where commonalities in systems and procedures already exist, and these should increase over time. All training should be to common standards and criteria specified by QCA, with required attendance supported by the availability of money from the Standards Fund. It should be accredited and recognised in career and development and reward structures. The difficulties most schools and colleges face if they lose their examination officer only underlines the invaluable nature of their contribution.

139. Raising standards of examination administration is significant to guaranteeing standards of awards, as well as to the smooth running of the system, and in particular to combating problems arising from the huge number of late entries and entries made on the day of the examination ("pirate" entries).

140. This year AQA received 73,241 late, 48,674 very late and about 11,000 "pirate" (entries on the day using photocopied question papers and answer booklets) GCE and VCE entries. These amounted to nearly 4% of all entries. The corresponding figures for Edexcel GCE entries were 57,432 very late and 6,784 pirate entries, nearly 5% of the total, and OCR had 24,235 late entries, nearly 2% of the total. The awarding bodies are clear that they should not damage candidates' life chances by refusing to accept these entries. Nevertheless, late and pirate entries place a significant additional strain on awarding bodies, which are already working in an extremely tight time window, including by hindering the planning of the numbers of markers and examiners.

141. More than this, however, late entries can hinder efforts to judge and maintain standards. This is particularly true of the late submission of coursework. Ideally an awarding body should have received 75% of coursework by the first deadline to inform standardisation meetings. It has been reported to me that this year by the first deadline one board had received only about 20% of coursework, and only about 40% by a second deadline 2-3 weeks later. This has impacts on the judgement of standards and on whether candidates receive the grade they deserve. Action must be taken to rectify the situation.

142. I recommend that by January 2004, the awarding bodies, in consultation with schools and colleges, the regulatory bodies and the Secretary of State, should develop and publish a strategy to:

  • ensure good quality training for examination officers, who should gain professional recognition for their role;

  • promote and reward the efficiency of schools' and colleges' administration as part of the inspection process;

  • encourage good practice. In my view that might include using the fee structure to provide incentives for timely examination entries.

143. I further recommend that OFSTED and ALI should be asked to look at and report on the quality of schools' and colleges' examinations administration as part of the institutional inspection frameworks.

Implications for Wales and Northern Ireland

144. Examiner shortages are less acute in Northern Ireland and Wales. Indeed CCEA has successfully addressed an examiner supply problem in recent years. Nevertheless, the measures in this chapter would be of benefit more widely than just England, and the principles which underpin them could usefully be applied on a three country basis.

^^Back to Top

THE USE OF ICT IN QUALIFICATIONS

145. Consideration of how to improve the reliability, efficiency and quality of the qualifications system inevitably raised questions about the potential impact of ICT. I have looked at the potential for ICT in three aspects of the system:

a. administration and data processing;

b. to improve the quality and efficiency of the marking processes;

c. as a medium for examining - ie ICT based examinations and assessment tasks.

Administration

146. Each of the awarding bodies uses computer systems to support their administration and data processing. Indeed, the current scale of the examinations task would not be manageable in any other way. From candidate entries at one end of the system through to the issuing of final results at the other, computing power is central to the vast majority of data processes needed to collect information about the performance of individual students, convert their raw marks into grades and issue their results and collate data across subjects, awarding bodies and the system as a whole.

147. All three of the English awarding bodies have recently made, or are just about to embark upon, major investment to update their administrative ICT systems. Nothing in the evidence put to me suggests that in relation to these administrative tasks the computer systems already operated or planned by the awarding bodies are fundamentally inadequate; nor that significant improvements in the quality or efficiency of qualifications administration could be achieved simply by additional investment in ICT beyond that already planned. It is certainly possible that the current position could be improved by continuing investment to extend the capacity of the systems to take on new tasks or manage existing tasks more efficiently. However, further work would be needed to establish the scope and business case for further investment of this kind.

148. Nevertheless, there is some evidence that additional investment elsewhere is needed to make best use of the current ICT capacity in the awarding bodies. This is particularly the case in relation to the commercially- operated Electronic Data Interchange (EDI) systems, which enable schools and colleges to make their examination entries on line and to exchange information with the awarding bodies about their candidates' entries and results.

149. Used to full potential, the EDI system would do much to eliminate differences in administrative procedures between awarding bodies. I gather that a significant obstacle to this is that some schools and colleges lack the necessary hardware and software packages to make full use of the EDI system. I have not had time to investigate this situation in detail, but if this understanding is correct it should be rectified as a matter of urgency.

ICT in marking

150. Based on the evidence presented to this inquiry, I am persuaded that ICT has the potential to bring about a step change in the efficiency and quality of the marking process. Students' scripts are currently distributed via the postal service to markers and the awarding bodies head offices. Typically they spend about five days in transit during the marking stage. The marking process is almost wholly manual. Markers work directly onto the written papers and collate their records manually.

151. The technology already exists to translate students' written performance into electronic form. Once digitised, scripts can be distributed and marked on-line, often in ways that would be difficult or impossible now. For instance, each script can be broken down into component answers and different answers can be distributed in different ways, for example by giving relatively straightforward short-answer questions to clerical or inexperienced markers. Questions which require more complex judgements can be sent to more experienced examiners. Marks awarded during the process can be collated and monitored continuously whilst marking is in progress. So, for instance, the performance of individual markers can be monitored during the marking process and inconsistencies between them can be identified and corrected much more quickly and accurately than is possible currently. Estimates given to me suggest that a 10-20% increase in the efficiency of marking could be achieved, accompanied by a significant increase in the consistency and reliability of marking.

152. All of this would be of significant benefit. But it can not be a quick fix. The awarding bodies have piloted systems along these lines. Whilst some of the potential benefits were apparent in these projects, there are some problems still to be overcome. Significant among these is managing the natural human response of markers and examiners to what would amount to full-scale industrialisation of the marking process and the need to adapt their working practices to reflect this. I am confident that this and other issues, such as the adaptation of the currently available systems to the specific needs of the awarding bodies and their qualifications, can be overcome given time and investment.

153. I am not, however, persuaded of the benefits of using ICT to replace human markers. Whilst there are systems which can produce reasonable marking of multiple choice and short answer questions, I do not have evidence of the capability to machine mark answers which require complex judgement. There would be some efficiency gains to be had from more extensive use of machine marking of the simpler answers in existing examination scripts. But such systems would embed reliance on these question and answer formats, and could significantly constrain longer term innovation in the examination system. On this basis, significant investment in such systems would be misguided.

ICT as a medium for examining and assessment

154. The systems described above would not change significantly the nature of examinations as experienced by students themselves. If anything, the need for a common format for script and students' responses in order to ensure that they can be digitised accurately, for instance by scanning, could impose an additional constraint on the scope for innovation in the examinations process.

155. A more radical longer term solution would be the use of ICT as the means by which students take examinations and submit work for assessment. For instance, video, photographs, other images, and sound, could supplement traditional written work. Work could be submitted electronically and assessed over time to build up a portfolio of assessed work by the end of the course, rather than relying largely on one-off examinations. More flexible ways of assessing students' work could change radically the nature of the assessment process and the extent to which it promotes key attributes such as creativity and communication skills.

156. A further development of this approach would be to bring students' performance data - for instance, test and examination results, and higher education performance - and their work together in a single national database to provide a permanent and comprehensive record of their progress and achievements through school, college, training and higher education. This would be information which they could access themselves, for instance to support applications for university or employment - to supplement or replace paper certification of their results.

Investment

157. Any new ICT systems would carry a significant cost. It is not clear that any of the awarding bodies have the capacity for investment on the scale necessary. Indeed, even some of the limited piloting of new ICT systems has been curtailed simply on the grounds of costs.

158. Nor is it necessarily the case that awarding bodies should be expected to bear the whole cost. Many of the potential advantages of more extensive or consistent use of ICT would be for schools, colleges and students rather than producing efficiency savings or other benefits for the awarding bodies themselves. Indeed, for some of the possibilities outlined in this chapter it would be very difficult for the awarding bodies individually or collectively to justify the investment solely in terms of a cost-benefit analysis for their activities. The benefits are spread widely across the system and would contribute directly to wider policy objectives such as reducing administrative burdens on schools. For systems which use ICT as the medium through which students undertake examination and assessment tasks, there would be a very substantial element of investment directly in schools and colleges to provide the necessary equipment and expertise to ensure students' access.

159. On this basis there is a strong case for suggesting that at least some of the costs should be met directly from public funds rather than specifically from the awarding bodies' own resources.

Conclusions

160. From the evidence presented to me, I am persuaded that ICT offers the prospect of significant improvements in the efficiency, quality and flexibility of the examinations system.

161. Although such a conclusion goes well beyond my remit, I am also persuaded that ICT offers in time the prospect of linking students' achievement information with examples of their work and other information to provide students with an individually accessible comprehensive portfolio which illustrates their learning much more broadly than simply examination results.

162. Most of the technology to achieve all these things is available now - albeit at a price. But there are still some important technical issues which need to be resolved. Most notably, in the context of work submitted on-line, it is essential that the identity of students and their responsibility for the work they are submitting can be verified. Equally, reliable security would be needed to ensure privacy of personal records and control individuals' access to personal portfolios and other personal data.

163 I recommend that: resources should be made available from public funds to the awarding bodies, schools and colleges to ensure that full use can be made of the existing EDI systems, and for other ICT developments to improve the commonality of administration and reduce the administrative burdens on schools; an expert working group should be established by the regulators and the awarding bodies to examine and develop the use of ICT to improve the efficiency and quality of administration and marking, whilst ensuring such developments do not constrain innovation by individual awarding bodies or the options for longer term changes in the nature and format of assessment; the Government should prepare a longer term strategy for developing the use of ICT as the medium by which students take examinations and submit material for assessment, taking account of wider developments of ICT in relation to the curriculum and teaching and learning; resources for developing, piloting and implementing new ICT-based systems in the qualifications framework should be provided directly from public funds to an extent which properly reflects the public benefits of such systems and the financial circumstances of the awarding bodies.

Relevance for Northern Ireland and Wales

164. The issues of consistency between awarding bodies which I have set out in this chapter are equally relevant to Wales and Northern Ireland and to those schools and colleges which draw on qualifications from Welsh or Northern Irish awarding bodies alongside those from England. This work should be taken forward on a three country basis. Longer term development of the potential of ICT is also potentially relevant to all three countries.

^^Back to Top

SCHEDULE OF EVIDENCE PRESENTED TO THE INQUIRY

During the course of my inquiry I took oral submissions from:

Head Masters Conference (HMC)
Girls' Schools Association (GSA)
Secondary Heads Association SHA)
National Association of Head Teachers (NAHT)
National Union of Teachers (NUT)
National Association Schoolmasters/Union of Women Teachers (NASUWT)
Professional Association of Teachers (PAT)
Association of Teachers and Lecturers (ATL)
National Association of Teachers in Further and Higher Education (MATFHE)
Association of Colleges (AoC)
National Confederation of Parent Teacher Associations (NCPTA) OFSTED
Universities and Colleges Admissions Services (UCAS)
Universities UK
Standing Conference of Principals (SCOP)
Qualifications Curriculum and Assessment Authority for Wales (ACCAC)
Council for the Curriculum Examination and Assessment (CCEA)
International Baccalaureate Organisation
The Joint Council
CBI
The Chambers of Commerce
Institute of Directors
The Chief Executives of the three English Awarding Bodies: AQA,
Edexcel and OCR.
Ken Boston, Chief Executive of the QCA, Sir Alan Greener and QCA officers
Nick Stuart
Roger Porkess
Professor Steve Heppell - Ultralab
NCS Pearson
Professor Carol Fitz-Gibbon
Professor David Melville
Dianne Francombe
Dr Brenda Cross
Damien Green MP
Phil Willis MP
Celia Johnson, DfES
Rob Hull, DfES

I have also received written evidence and submissions from:

SHA
HMC
GSA
NAHT
NUT
ATL
NATFHE
NASUWT
NCPTA
AoC
The General Teaching Council for England
Universities UK
UCAS
SCOP
QCA
OFSTED
AQA
Edexcel
OCR
Damien Green MP
Phil Willis MP

In addition, I have received helpful written submissions from a very wide range of individuals, schools, teachers, examiners and others.

I held three regional meetings with teachers, parents, governors and students and would like to thank all that took part and who helped in their organisation, particularly: Professor Robin Millar, York University; Kath James, York University; David Moores, Deputy Head, Fulford School; Roger Dancey, Chief Master, King Edward's School; Sarah Evans, Head, King Edward's High School; Alan Jenkins, Principal, Varndean College; and Dr Russell Strutt, Principal, Haywards Heath College.

^^ Back to Top
 © MMV | News Sources | Privacy