1 school exam grade in 4 is wrong. Does this matter?
Thank you, HEPI, for publishing this series of three blogs
...which is the good news...
...and here are some solutions...
Conference presentation on grade (un)reliability
On 19th March 2020, I had been invited to give a presentation telling the story of grade (un)reliability, but due to the Covid-19 problem, the conference was -
understandably and wisely - cancelled.
Please click on the picture to download the presentation write-up.
Dennis on BBC Radio 4's Today programme, and also on More or Less
Following the announcement of the results of the 2019 A level and GCSE examinations, my claim that "1 grade in 4 is wrong" featured in the discussion with the Schools Minister, Nick Gibb, on BBC Radio 4's Today programme on 22nd August 2019:
And on the following day, I contributed an
an interview to More or Less (the first 10 minutes). So thank you, the BBC!
Spreadsheets to help determine 'Centre Assessed Grades'
I have developed some spreadsheets that use 2017, 2018 and 2019 grade distributions to suggest the numbers of candidates in the 2020 cohort that match the historical pattern.
Easy-to-identify colour codes show statistical 'outliers', and you can adjust the 'suggested distribution' too.
Spreadsheets are available for all levels, and also for aggregating sets within a subject, and subjects within a department or school.
For further details, click here.
Grade inflation at universities and schools
On 28th November 2018, the UK Standing Committee for Quality Assessment, the body that oversees quality standards for higher education, published a report enquiring into the observation that the number of 1st and 2:1 degrees awarded by many universities has been steadily rising over the last several
years. Pages 9 and 10 refer to my work on a similar phenomenon observed from 1988 to 2011 in relation to the number of higher grades awarded for GCSE exams in schools, as described in my document The race to the bottom is real.
Also, on 13th December, I will be speaking about this at a conference organised by Universities UK, following a presentation from the lead author of the report, Professor Andrew Wathey, the Vice-Chancellor and Chief Executive of Northumbria University.
Ofqual publish "qualification level" measures of grade reliability
Some two years after I began my campaign about the reliability of GCSE and A level grades, Ofqual have now published some official statistics in their document entitled
Marking Consistency Metrics - An update.
The key results are shown in Figure 12, on page 21, and reproduced to the left: the average grade reliability for each subject is shown by the vertical line within the blue-shaded box. As can be seen, the most reliable subject is (unsurprisingly) maths, at about 96%; the least, a combined
qualification in English language and literature at about 52%. These results apply to subjects graded A*, A, B..., and so the most recent GCSEs, graded 9, 8, 7..., are even more unreliable.
If the subject reliabilities are weighted by the (2016) subject cohorts, the average grade reliability across these subjects computes to about 74%, implying that for every 100 grades awarded each August, about 74 are right, and about 26 are wrong - so validating the assertion I have been making for some time that, on average, about 1 grade in 4 is wrong.
"1 grade in 4 is wrong" is no longer the assertion of a sole voice, it is an official statistic. So the BIG QUESTIONS now are "Does it matter?" and "Who cares?".
If you think it matters, and if you care, please contact me!
Simulation of GCSE and A level grade reliability
I’ve developed a spreadsheet that simulates GCSE and A level grade reliability for any subject, under any grading structure – so encompassing, and allowing comparison between, A*, A, B... and also 9, 8, 7...
Very briefly, for any given cohort size, the programme firstly distributes candidates across the mark range, so determining the number of candidates awarded any specific mark – for example, for a cohort of 250,000 students, 5,955 might be given 59 marks, corresponding to, say, grade 5.
The program then explores what happens when the scripts of the entire cohort are fairly re-marked. Since, as Ofqual states, “it is possible for two examiners to give different but appropriate marks to the same answer”, it is unlikely that all those 5,955 scripts originally given 59 will be given 59 on a re-mark: some might be given a lower mark (say, 57), and some a higher mark (say, 61).
The spreadsheet simulates this, and so the 5,955 scripts originally given 59 marks are ‘smeared out’ over a defined range. If the 4/5 grade boundary is 54/55, and the 5/6 boundary, 60/61, then some of the re-marks will result in a down-grade, and some in an up-grade. For those 5,955 candidates originally given 59 marks, the programme calculates how many of those have the original grade 5 confirmed (3,413), how many are awarded down-grades (568), and how many are awarded up-grades (1,974). The reliability of the original grade 5, corresponding to the original mark 59, is then calculated as 3,413 / 5,955 = 57%.
The accompanying chart, which is based on data for 2017 GCSE English Language, is an example of how the key results are presented: the 'wiggly line' shows how the reliability of the awarded grade varies according the original mark. As can be seen, grade reliability is worst at grade boundaries, somewhat better in the middle of grade widths, and reaches 100% only for good 9's and poor U's.
If you would like to use the spreadsheet, please contact me.
Thermodynamics made dynamic
To help students enrich their understanding of some of the most important principles of thermodynamics, I have written some spreadsheets that produce graphical representations of, for example, the Boltzmann distribution, the Van der Waals equation, chemical equilibrium, and the biochemical standard state.The spreadsheets allow the student to explore what happens when the relevant parameters change, and are now available on the OUP website, as additional material associated with Modern Thermodynamics.
Murder on the academic express
Why has there been a steady year-on-year increase in the number of 1st and 2:1 degrees awarded by British universities?
Can Hercule Poirot solve the mystery of the brutal murder of academic standards?
All is revealed in this blog, posted by the Higher Education Policy Institute...
Dennis featured in The Daily Telegraph again
The GSCE and A level school examination results are out again, and one year on from the press coverage last year, the reliability of the awarded grades has not become any better: rather, for those GCSEs now graded 9, 8, 7..., the grades are even more unreliable: I estimate around one million grades have been wrong this year. Oh dear.
So thank you The Daily Telegraph for your articles on 18th August and 25th August.
Also, in April 2018, Dennis gave a presentation at the annual conference of the UK System Dynamics Society - click on the accompanying picture to read about the perfect crime...
New thermodynamics book blog
Dennis has contributed a new blog to the OUP website. It is nearly 60 years since CP Snow gave his 'Two Cultures' lecture, in which he advocated that a good education should equip a young person with as deep a knowledge of the Second Law of Thermodynamics as of Shakespeare. A cynic might argue that his vision has successfully been achieved, for some young people might now be leaving school with little knowledge of either; on a more positive note, why did CP Snow choose this particular scientific law, as opposed to, say, Newton's Law of Gravitation, or Boyle's Law?
Some reviews* of Forest for the Trees...
"This is an extraordinary book about systems thinking. In this book it is possible to learn about how everything is connected and how just a little modification can change everything; literally, this book can open your eyes. Unlike most business books, this one is based on sound scientific research but it is not loaded with tons of jargon; instead Sherwood wrote a quite readable, funny and concise book."
"If you are serious about system thinking, and want to create causal maps, you need to buy this book. Yes buy it because you will revert to it over and over again. Brilliant examples, clearly explained."
"If you want to learn how to draw causal loop diagrams and use them in your work, this is the book for you. The book is extremely comprehensive, yet easy to read and written in a style that makes you feel like the author is personally teaching you about system dynamics in a one-to-one lesson."
"I have been using this textbook for 6 years in a module of a course on Causal Loop Diagrams (CLDs) for both Engineering Students and Futures Studies students. I have not been able to find any better book and have recommended it to many "Systems Thinkers" and they were all impressed."
"This has to be in contention for my book of the year for 2014 so far (out of 45 so far). This book introduces system thinking in a very light manner, then seamlessly drifts causal loops into the equation and before you are finished you are well on your way to understanding system dynamics."
"This is one of the best books for system thinking: very helpful, I really learned a lot by reading it."
"Rich in detail...a great learning resource. A first-rate book of substance and quality. Highly recommended."
* amazon.com and amazon.co.uk
Dennis featured in The Daily Telegraph
Dennis is the author of a feature in The Daily Telegraph, published on 22nd August 2017, and is also quoted in an article published the previous Sunday. Both pieces are about the reliability - or rather unreliability - of GCSE and A level grades.
According to a report published* by the exam regulator Ofqual, grades are pretty unreliable: over the last four years, some 15% of candidates in physics have been awarded the wrong grade, some 30% in English Language, and some 40% in history. And changing the GCSE grading structure from A*, A, B... to 9, 8, 7... makes things even worse.
For a more detailed discussion, see my blogs How reliable are GCSE and A level grades? and How to make GCSE and A level grades reliable, and also these documents:
■ The grading system is broken
■ Identifying marking 'errors'
■ The statistics of marking and re-marking
* Slide 7 from the presentation Quality of marking: confidence and consistency given by Dr Michelle Meadows at the Ofqual Summer Symposium, 29th June 2017, https://www.gov.uk/government/news/presentations-from-ofquals-summer-series-symposium-2017.
Great workshop for the KCL and ICL Reactive Plasmonics team
The Royal Society's Kavli Centre at Chicheley Hall was a splendid venue for a Creativity@ home event for the Reactive Plasmonics team at King's College, London, and Imperial College, London, led by Professor Anatoly Zayats. After an initial training day, the two-day off-site workshop was hugely productive, generating a host of great scientific ideas.
Some reviews* of Crystals, X-ray and Proteins
"This is one of the best crystallography books ever written, and it is with pleasure that I wholeheartedly recommend it. "
"In my opinion, this book would be the perfect textbook for a theoretical course on macromolecular crystallography."
"The first two-thirds of this book was like a thriller to me. Even though I knew the answer, I wanted to see how the author would address the next topic and I could not put it down."
"Perfection. This is the most underrated book ever. If you're familiar with fundamental molecular biology and have just a brief understanding of high-school mathematics/physics, this book is the best place to start. Hats off to the authors!"
And this review of the first edition appeared in Nature in April 1976.
* amazon.com and amazon.co.uk
Dennis speaks at Valve World
The big event in the annual calendar in the valve industry is the Valve World conference in Dusseldorf. At the suggestion of IMI Critical Engineering (thank you, IMI!), Dennis was invited to give a plenary presentation, and to run a workshop, and was also featured in Valve World Magazine. Those unfamiliar with the valve industry might think that valves - which have indeed been around for
hundreds of years - are now fully evolved, and so offer little scope for innovation. Not at all. Valves, and related components such as actuators, are of immense importance in controlling the flow of fluids in applications
from petrol pumps to power stations, from a hospital dialysis machine to a domestic
heating system. And the scope for successful creativity and innovation is huge.
New thermodynamics book
Dennis's latest book, co-authored with UCL Professor Paul Dalby, is a new textbook on thermodynamics, aimed at undergraduates in the chemical and biological sciences, and also at graduate students who need to brush up on this notoriously difficult subject. The book was published by Oxford University Press in May 2018, and is a second, and much revised, edition of Introductory Chemical Thermodynamics, published in 1971 in the UK by Longman, and in the US by Prentice Hall under the title Elementary Chemical Thermodynamics.