UC San Diego

Importance

Procedural proficiency is a core competency for graduate medical education; however, procedural reporting often relies on manual workflows that are duplicative and generate data whose validity and accuracy are difficult to assess. Failure to accurately gather these data can impede learner progression, delay procedures, and negatively impact patient safety.

Objective

To examine accuracy and procedure logging completeness of a system that extracts procedural data from an electronic health record system and uploads these data securely to an application used by many residency programs for accreditation.

Design, setting, and participants

This quality improvement study of all emergency medicine resident physicians at University of California, San Diego Health was performed from May 23, 2023, to June 25, 2023.

Exposures

Automated system for procedure data extraction and upload to a residency management software application.

Main outcomes and measures

The number of procedures captured by the automated system when running silently compared with manually logged procedures in the same timeframe, as well as accuracy of the data upload.

Results

Forty-seven residents participated in the initial silent assessment of the extraction component of the system. During a 1-year period (May 23, 2022, to May 7, 2023), 4291 procedures were manually logged by residents, compared with 7617 procedures captured by the automated system during the same period, representing a 78% increase. During assessment of the upload component of the system (May 8, 2023, to June 25, 2023), a total of 1353 procedures and patient encounters were evaluated, with the system operating with a sensitivity of 97.4%, specificity of 100%, and overall accuracy of 99.5%.

Conclusions and relevance

In this quality improvement study of emergency medicine resident physicians, an automated system demonstrated that reliance on self-reported procedure logging resulted in significant procedural underreporting compared with the use of data obtained at the point of performance. Additionally, this system afforded a degree of reliability and validity heretofore absent from the usual after-the-fact procedure logging workflows while using a novel application programming interface-based approach. To our knowledge, this system constitutes the first generalizable implementation of an automated solution to a problem that has existed in graduate medical education for decades.