The Accuracy of Predictive Analytics in Forecasting Emergency Department Volume Before and After Onset of COVID-19
Abstract
Introduction: Big data and improved analytic techniques, such as triple exponential smoothing (TES), allow for prediction of emergency department (ED) volume. We sought to determine 1) which method of TES was most accurate in predicting pre-coronavirus 2019 (COVID-19), during COVID-19, and post-COVID-19 ED volume; 2) how the pandemic would affect TES prediction accuracy; and 3) whether TES would regain its pre-COVID-19 accuracy in the early post-pandemic period.
Methods: We studied monthly volumes of four EDs with a combined annual census of approximately 250,000 visits in the two years prior to, during the 25-month COVID-19 pandemic, and the 14 months following. We compared the accuracy of four models of TES forecasting by measuring the mean absolute percentage error (MAPE), mean square errors (MSE) and mean absolute deviation (MAD), comparing actual to predicted monthly volume.
Results: In the 23 months prior to COVID-19, the overall average MAPE across four forecasting methods was 3.88% ± 1.88% (range 2.41–6.42% across the four ED sites), rising to 15.21% ± 6.67%during the 25-month COVID-19 period (range 9.97–25.18% across the four sites), and falling to 6.45% ± 3.92% in the 14 months after (range 3.86–12.34% across the four sites). The 12-month Holt-Winter method had the greatest accuracy prior to COVID-19 (3.18% ± 1.65%) and during the pandemic (11.31% ± 4.81%), while the 24-month Holt-Winter offered the best performance following the pandemic (5.91% ± 3.82%). The pediatric ED had an average MAPE more than twice that of the average MAPE of the three adult EDs (6.42% ± 1.54% prior to COVID-19, 25.18% ± 9.42% during the pandemic, and 12.34% ± 0.55% after COVID-19). After the onset of the pandemic, there was no immediate improvement in forecasting model accuracy until two years later; however, these still had not returned to baseline accuracy levels.
Conclusion: We were able to identify a TES model that was the most accurate. Most of the models saw an approximate four-fold increase in MAPE after onset of the pandemic. In the months following the most severe waves of COVID-19, we saw improvements in the accuracy of forecasting models, but they were not back to pre-COVID-19 accuracies.