Proceedings of The Physiological Society
University of Oxford (2011) Proc Physiol Soc 23, PC9
A simple method for the ongoing evaluation of standard setting for one best answer exam questions
R. M. Lewis1, J. Field1
1. Faculty of Medicine, University of Southamton, Southampton, United Kingdom.
Introduction: Standard setting methods are now routinely used to determine passing scores in multiple choice examinations. However it is often difficult in everyday practice to know whether the standard setting process is reaching good judgements about the performance of borderline students. A new programme with a team of inexperienced examiners provided an opportunity to assess the ability of the examiners to predict student performance. Methods: Pass fail boundaries for multiple choice examinations at the end of Years 1 and 2 of the graduate entry programme at Southampton were determined using Ebel’s method over the programme’s first three years. The predicted performance of borderline students was compared with the actual performance of the borderline students. Results: Assessed difficulty reflected student performance from the outset. However, in the initial examinations the team did not always effectively distinguish between questions of adjacent difficulty levels (e.g. ‘easy’ vs ‘easy to moderate’). Over the course of three years the team’s ability to distinguish between these questions generally improved. Analysis of standard setting allowed feedback to the team which contributed to the improvement. Conclusions: Evaluation of the standard setting process showed that a relatively inexperienced team could predict the performance of borderline students. Ongoing evaluation may have aided better predictions and should increase the confidence of examiners and students in the process.
Where applicable, experiments conform with Society ethical requirements