Inter-rater Agreement of End-of-shift Evaluations Based on a Single Encounter
Abstract
Introduction End-of-shift evaluation (ESE) forms, also known as daily encounter cards, represent a subset of encounter-based assessment forms. Encounter cards have become prevalent for formative evaluation, with some suggesting a potential for summative evaluation. Our objective was to evaluate the inter-rater agreement of ESE forms using a single scripted encounter at a conference of emergency medicine (EM) educators.
Methods Following institutional review board exemption a scripted video simulating an encounter between an intern and a patient with an ankle injury was created. That video was shown during a lecture at the Council of EM Residency Director’s Academic Assembly with attendees asked to evaluate the “resident” using 1 of 8 possible ESE forms randomly distributed. Descriptive statistics were used to analyze the results with Fleiss’ Kappa to evaluate inter-rater agreement.
Results Most of the 324 respondents were leadership in residency programs (66%). Despite that, few individuals (5%) felt they were experts in assessing residents based on EM milestones. Fleiss’ Kappa ranged from 0.157 - 0.308 and did not perform much better in two post-hoc subgroup analyses.
Conclusion
The kappa ranges found show only slight to fair inter-rater agreement and raise concerns about the use of ESE forms in assessment of EM residents. Despite limitations present in this study, these results and a lack of other studies on inter-rater agreement of encounter cards should prompt further studies of such methods of assessment. Additionally, EM educators should focus research on methods to improve inter-rater agreement of ESE forms or other evaluating other methods of assessment of EM residents.