MRCGP CSA: are the examiners biased, favouring their own by sex, ethnicity, and degree source?

Abstract

BACKGROUND: Concern exists regarding differential performance of candidates in postgraduate clinical assessments by ethnicity, sex, and country of primary qualification. Could examiner bias be responsible? AIM: To explore whether candidate demographics affect examiners' judgements, by investigating candidates' case performances by candidates' and examiners' demographics. DESIGN AND SETTING: Data on 4000 candidates (52,000 cases) sitting the MRCGP clinical skills assessment in 2011-2012. METHOD: Univariate analyses were undertaken of subgroup performance (male/female, white/black and minority ethnic (BME), UK/non-UK graduates) by parallel examiner demographics. Due to confounding of variables, these were complemented by multivariate ANOVA and multiple regression analyses. RESULTS: Univariate analysis showed some differences between outcomes between the same-group and other-group examiners: these were contradictory regarding examiners 'favouring their own', for example, males received higher marks from female examiners than from males: maximum effect size was 3.6%. A six-way ANOVA confirmed all three candidate and examiner variables as having significant effects individually, identifying one significant interaction (examiner sex by examiner ethnicity). Stepwise regression showed candidate variables predicting 12% of score variance, parallel examiner demographics adding little (approximately 0.2% of variance). One 'transactional' variable proved significant, explaining 0.06% of score variance. CONCLUSION: Examiners show no general tendency to 'favour their own kind'. With confounding between variables, as far as the impact on candidates' case scores, substantial effects relate to candidate and not examiner characteristics. Candidate-examiner interaction effects were inconsistent in their direction and slight in their calculated impact.