Skip to main content
eScholarship
Open Access Publications from the University of California

UC San Diego

UC San Diego Previously Published Works bannerUC San Diego

Development, deployment, and continuous monitoring of a machine learning model to predict respiratory failure in critically ill patients

Abstract

Objectives

This study describes the development and deployment of a machine learning (ML) model called Vent.io to predict mechanical ventilation (MV).

Materials and methods

We trained Vent.io using electronic health record data of adult patients admitted to the intensive care units (ICUs) of the University of California San Diego (UCSD) Health System. We prospectively deployed Vent.io using a real-time platform at UCSD and evaluated the performance of Vent.io for a 1-month period in silent mode and on the MIMIC-IV dataset. As part of deployment, we included a Predetermined Changed Control Plan (PCCP) for continuous model monitoring that triggers model fine-tuning if performance drops below a specified area under the receiver operating curve (AUC) threshold of 0.85.

Results

The Vent.io model had a median AUC of 0.897 (IQR: 0.892-0.904) with specificity of 0.81 (IQR: 0.812-0.841) and positive predictive value (PPV) of 0.174 (IQR: 0.148-0.176) at a fixed sensitivity of 0.6 during 10-fold cross validation and an AUC of 0.908, sensitivity of 0.632, specificity of 0.849, and PPV of 0.235 during prospective deployment. Vent.io had an AUC of 0.73 on the MIMIC-IV dataset, triggering model fine-tuning per the PCCP as the AUC was below the minimum of 0.85. The fine-tuned Vent.io model achieved an AUC of 0.873.

Discussion

Deterioration of model performance is a significant challenge when deploying ML models prospectively or at different sites. Implementation of a PCCP can help models adapt to new patterns in data and maintain generalizability.

Conclusion

Vent.io is a generalizable ML model that has the potential to improve patient care and resource allocation for ICU patients with need for MV.

Many UC-authored scholarly publications are freely available on this site because of the UC's open access policies. Let us know how this access is important for you.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View