Inter-rater Agreement of End-of-shift Evaluations Based on a Single Encounter
Skip to main content
Open Access Publications from the University of California

Inter-rater Agreement of End-of-shift Evaluations Based on a Single Encounter

  • Author(s): Warrington, Steven Jay
  • Beeson, Michael S
  • Bradford-Saffles, Amber
  • et al.

Introduction End-of-shift evaluation (ESE) forms, also known as daily encounter cards, represent a subset of encounter-based assessment forms.  Encounter cards have become prevalent for formative evaluation, with some suggesting a potential for summative evaluation.  Our objective was to evaluate the inter-rater agreement of ESE forms using a single scripted encounter at a conference of emergency medicine (EM) educators.


Methods Following institutional review board exemption a scripted video simulating an encounter between an intern and a patient with an ankle injury was created.  That video was shown during a lecture at the Council of EM Residency Director’s Academic Assembly with attendees asked to evaluate the “resident” using 1 of 8 possible ESE forms randomly distributed.  Descriptive statistics were used to analyze the results with Fleiss’ Kappa to evaluate inter-rater agreement.


Results Most of the 324 respondents were leadership in residency programs (66%).  Despite that, few individuals (5%) felt they were experts in assessing residents based on EM milestones.  Fleiss’ Kappa ranged from 0.157 - 0.308 and did not perform much better in two post-hoc subgroup analyses.



The kappa ranges found show only slight to fair inter-rater agreement and raise concerns about the use of ESE forms in assessment of EM residents.  Despite limitations present in this study, these results and a lack of other studies on inter-rater agreement of encounter cards should prompt further studies of such methods of assessment.  Additionally, EM educators should focus research on methods to improve inter-rater agreement of ESE forms or other evaluating other methods of assessment of EM residents.

Main Content
Current View