No, Pearson. 99.2% of your grades were not accurate

For many months now, the website of Pearson, the owners of the edexcel examination board, has been boasting that “99.2% of our grades were accurate on results day”, a claim Pearson have been using to boost their marketing.

This claim is wrong. And it is amazing that Pearson have been allowed to continue this claim for so long, without being sanctioned by their regulator, Ofqual, an angry exam user, or indeed the Advertising Standards Authority. Especially so since the falsity was first identified in January 1918 by my friend and colleague, Rob Eastaway, in his blog How to Lie with Statistics, to which this blog is indeed indebted.

In early December every year, Ofqual publish a series of documents collectively entitled Reviews of marking and moderation for GCSE, AS and A level, which contain a large number of statistics relating to the previous summer’s examinations. Reference to this data for the summer 2017 examinations will show that Pearson awarded a total of 1,861,975 certifications at GCSE and A level, across England, Wales and Northern Ireland. The statistics also show that, as a result of the examination appeals process, a total of 14,210 Pearson grades were changed, from which we may calculate that the number of Pearson grades changed, expressed as a percentage of the total number of Pearson certifications, is 14,210/1,861,975 = 0.8%.

According to the official Ofqual data, it is true that 0.8% of Pearson grades were changed as the result of an appeal, implying that 0.8% of Pearson grades were wrong on results day.

Pearson then infer that 100.0 – 0.8 = 99.2% of their grades were therefore accurate on results day, hence the claim on their website.

This appears to be reasonable. But it is not. Not only is it wrong, it is an example of a statistical howler of which any organisation should be ashamed. But for this howler to be made, and publicised, by an organisation that sets, and marks, examinations – including examinations in statistics – is a disgrace of the first magnitude.

The howler becomes evident as soon as the penny drops that a grade can be changed only if an appeal has been made. If no appeal is made, the grade as awarded on results day may, or may not, be accurate – we just don’t know. There may be a mistake, lurking, undiscovered.

So another – important – number to be found in the Ofqual statistics is the number of appeals made in relation to Pearson grades, this number being 86,465 for GCSE and A level, across England, Wales and Northern Ireland. And it is within this population of 86,465 appeals that 14,210 errors were discovered, and subsequently corrected.

But since Pearson awarded a total of 1,861,975 GCSE and A level certifications, 1,861,975 – 86,465 = 1,775,510 certifications were not appealed.

The BIG – and unanswered – question is “how many grade changes would have been made had these 1,775,510 certifications been appealed?”. If this answer is zero, then Pearson’s claim of 99.2% accuracy is valid. But if there are any ‘potential’ grade changes within those 1,775,510 certifications that were not challenged, then Pearson’s claim is wrong.

I know where I’m putting my money...