Setting pass scores for assessment of technical performance by surgical trainees.

Abstract

One of the major challenges of competency-based training is defining a score representing a competent performance. The objective of this study was to set pass scores for the Objective Structured Assessment of Technical Skill. Pass scores for the examination were set using three standard setting methods applied to data collected prospectively from first-year surgical residents (trainees). General surgery residents were then assigned an overall pass-fail status for each method. Using a compensatory model, residents passed the eight station examinations if they met the overall pass score; using a conjunctive model, residents passed if they met the overall pass score and passed at least 50 per cent of the stations. The consistency of the pass-fail decision across the three methods, and between a compensatory and conjunctive model, were compared. Pass scores were stable across all three methods using data from 513 residents, 133 of whom were general surgeons. Consistency of the pass-fail decision across the three methods was 95.5 and 93.2 per cent using compensatory and conjunctive models respectively. Consistency of the pass-fail status between compensatory and conjunctive models for all three methods was also very high (91.7, 95.5 and 96.2 per cent). Consistency in pass-fail status between the various methods builds evidence of validity for the set scores. These methods can be applied and studied across a variety of assessment platforms, helping to increase the use of standard setting for competency-based training.