Inter-rater Agreement of End-of-shift Evaluations Based on a Single Encounter
Skip to main content
Open Access Publications from the University of California

Inter-rater Agreement of End-of-shift Evaluations Based on a Single Encounter


Introduction End-of-shift evaluation (ESE) forms, also known as daily encounter cards, represent a subset of encounter-based assessment forms.  Encounter cards have become prevalent for formative evaluation, with some suggesting a potential for summative evaluation.  Our objective was to evaluate the inter-rater agreement of ESE forms using a single scripted encounter at a conference of emergency medicine (EM) educators.


Methods Following institutional review board exemption a scripted video simulating an encounter between an intern and a patient with an ankle injury was created.  That video was shown during a lecture at the Council of EM Residency Director’s Academic Assembly with attendees asked to evaluate the “resident” using 1 of 8 possible ESE forms randomly distributed.  Descriptive statistics were used to analyze the results with Fleiss’ Kappa to evaluate inter-rater agreement.


Results Most of the 324 respondents were leadership in residency programs (66%).  Despite that, few individuals (5%) felt they were experts in assessing residents based on EM milestones.  Fleiss’ Kappa ranged from 0.157 - 0.308 and did not perform much better in two post-hoc subgroup analyses.



The kappa ranges found show only slight to fair inter-rater agreement and raise concerns about the use of ESE forms in assessment of EM residents.  Despite limitations present in this study, these results and a lack of other studies on inter-rater agreement of encounter cards should prompt further studies of such methods of assessment.  Additionally, EM educators should focus research on methods to improve inter-rater agreement of ESE forms or other evaluating other methods of assessment of EM residents.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View