Interassessor agreement of portfolio-based competency assessment for orthotists/prosthetists in Australia: a mixed method study.

Abstract

Internationally qualified orthotists/prosthetists who want to practice in Australia must pass a portfolio-based competency assessment. Testing the agreement between independent assessors is important to engender confidence in the assessment, and continually improve the processes. To quantify interassessor agreement for all 68 performance indicators in the Australian Orthotic Prosthetic Association's Entry Level Competency Standards and where there was significant disagreement between assessors, to explore the reasons why. Mixed methods: explanatory sequential. Fifteen portfolios were assigned to independent assessors. Assessors determined whether the evidence presented met the requirements of each performance indicator. Interassessor agreement was calculated using Gwet's Agreement Coefficient 1 (AC1), and these data informed semistructured interviews to explore the reasons for disagreement. Most performance indicators (87%) had moderate to substantial agreement (AC1 > 0.71), which could be attributed to a variety of factors including the use of a simple assessment rubric with supporting guidelines and assessor training to establish shared expectations. The remaining performance indicators (13%) had fair to slight agreement (AC1 ≤ 0.7). Interviews with assessors suggested that disagreement could be attributed to the complexity of some performance indicators, unconscious bias, and the appropriateness of the evidence presented. Although most performance indicators in Australian Orthotic Prosthetic Association's Entry Level Competency Standard were associated with moderate to substantial interassessor agreement, there are opportunities to improve agreement by simplifying the wording of some performance indicators and revising guidelines to help applicants curate the most appropriate evidence for each performance indicator.