At King’s College London the medical year 1 Cardiovascular and Respiratory System (CVR) course is largely assessed using linear examinations. The computerised adaptive testing (CAT) project aims to explore the possibility of turning these tests into online adaptive tests. The power behind adaptive testing is item response theory; each question or ‘item’ is analysed by psychometric software using accumulated past exam responses and consequently provides each item with separate parameters of difficulty, discrimination and pseudo-guess. An adaptive test uses these parameters to ‘tailor’ an exam to the ability of each candidate. Only a relatively small number of organisations are actively using CAT for high stakes exams, mostly on a large-scale continuous testing basis (e.g. National Council of State Boards of Nursing).
The first step towards evaluating the feasibility of implementing CAT exams was to purchase an online testing programme and after research, a software package called Quiz Studio was purchased. As the CVR exams have been optically marked for the past 3 years we have computerised responses for all the items in the bank. To establish parameters for each item these responses need to be calibrated. Quiz Studio has calibrated 57 items and we are now using a psychometric software solution (Xcalibre) to calibrate the remaining item bank. For Xcalibre to read the data, approximately 350 000 responses have been converted into a specific text format. A database (item bank) of 950 previous exam questions has been created and sorted according to set categories.
Previous exams have been made available online to the students using Quiz Studio, partly for revision purposes but mainly as an evaluation exercise. Online feedback forms returned 80 completed responses. The students who used Quiz Studio for revision purposes returned positive feedback. Of the survey responses collected, 80 % said the system was useful for learning, 79 % said they would use the system again and 80 % said they would like similar systems applied to other courses.
Implementing a CAT exam requires extensive preparation and is pivotal on having a large calibrated item bank. If exams move to being online, then sufficient data are collected, items can be calibrated by Quiz Studio. For the moment data will continue to be calibrated using data from the optical marker. Validation is an important consideration and careful planning will be needed to construct a starting point for the integration of this method. A useful by-product of this project is the production of a large calibrated item bank of CVR questions that will be used to provide questions for future linear tests. For further information see:
www.kcl.ac.uk/teares/gktvc/vc/CAT/