Patients admitted to the hospital ward from the emergency department (ED) occasionally decompensate and require transfer to the intensive care unit (ICU). An emergency medicine (EM) curriculum focused on review of these ICU upgrade cases could improve resident knowledge related to patient acuity, critical illness, and appropriate disposition. Furthermore, initial identification of critical pathology in the ED and earlier admission to the ICU could reduce delays in care and improve patient outcomes.

We performed a retrospective analysis to determine the effectiveness of a resident quality improvement curriculum evaluating cases where patients require transfer from the inpatient floor to the ICU within 12 hours of admission from the ED. We compared postgraduate year 2 (PGY-2) EM residents who participated in the ICU upgrades curriculum during their first year to PGY-2 EM residents who did not participate in the curriculum.

Analysis of the 242 qualifying ICU upgrade cases from July 2019–October 2021 showed post-curriculum residents were responsible for an average of 1.0 upgrades per resident compared to an average of 1.54 upgrades per resident (P = 0.12) for pre-curriculum residents. Although there was no statistically significant difference in ICU upgrades between the groups, there was a trend toward decreased ICU upgrade cases for residents who participated in the curriculum. Common reasons for ICU upgrade included worsening respiratory distress requiring higher level of respiratory support, recurrent hypotension after initial intravenous fluid resuscitation requiring vasopressor support, and declining mental status.

This retrospective study showed no significant difference in the number of ICU upgrades for residents who completed the ICU upgrades curriculum compared to residents who were not enrolled in the course. However, the study was likely underpowered to detect a significant difference in the groups, and there was a trend toward reduced ICU upgrades for residents who completed the curriculum. ICU upgrade cases were frequently associated with worsening respiratory status, hypotension, and mental status. These findings highlight the importance of reassessment of vital signs and mental status prior to determining disposition from the ED. Additional, larger studies are needed to better determine the curriculum’s impact on resident proficiency in recognizing critical illness and reducing ICU upgrades.

Introduction: Emergency medicine (EM) milestones are used to assess residents’ progress. While some milestone validity evidence exists, there is a lack of standardized tools available to reliably assess residents. Inherent to this is a concern that we may not be truly measuring what we intend to assess. The purpose of this study was to design a direct observation milestone assessment instrument supported by validity and reliability evidence. In addition, such a tool would further lend validity evidence to the EM milestones by demonstrating their accurate measurement.

Methods: This was a multi-center, prospective, observational validity study conducted at eight institutions. The Critical Care Direct Observation Tool (CDOT) was created to assess EM residents during resuscitations. This tool was designed using a modified Delphi method focused on content, response process, and internal structure validity. Paying special attention to content validity, the CDOT was developed by an expert panel, maintaining the use of the EM milestone wording. We built response process and internal consistency by piloting and revising the instrument. Raters were faculty who routinely assess residents on the milestones. A brief training video on utilization of the instrument was completed by all. Raters used the CDOT to assess simulated videos of three residents at different stages of training in a critical care scenario. We measured reliability using Fleiss’ kappa and interclass correlations.

Results: Two versions of the CDOT were used: one used the milestone levels as global rating scales with anchors, and the second reflected a current trend of a checklist response system. Although the raters who used the CDOT routinely rate residents in their practice, they did not score the residents’ performances in the videos comparably, which led to poor reliability. The Fleiss’ kappa of each of the items measured on both versions of the CDOT was near zero.

Conclusion: The validity and reliability of the current EM milestone assessment tools have yet to be determined. This study is a rigorous attempt to collect validity evidence in the development of a direct observation assessment instrument. However, despite strict attention to validity evidence, inter-rater reliability was low. The potential sources of reducible variance include rater- and instrument-based error. Based on this study, there may be concerns for the reliability of other EM milestone assessment tools that are currently in use.