Forecasting NHS hospital demand under two structural breaks: A multiple-comparison-corrected audit of seven forecasting methods on 15 years of public data.

Abstract

Machine learning (ML) is increasingly used for hospital demand forecasting, but few published studies report multiple-comparison-corrected statistical tests against competitive baselines, and few characterise the structural-break context in which forecasts are evaluated. To audit seven AI/ML methods against statistical baselines on 15 years of NHS England activity data, with explicit attention to structural breaks and pandemic confounding. Monthly A&E activity (n = 189) and 52-quarter KH03 bed occupancy (2010-2026) were obtained from public NHS England releases. Andrews sup-F (q = 2 parameters, 15% trimming) and Bai-Perron breakpoint analyses characterised structural breaks. Seven forecasters were benchmarked on a 12-month chronological hold-out and 12-fold rolling-origin cross-validation, with Diebold-Mariano tests and Bonferroni-Holm + Benjamini-Hochberg correction. A pre-pandemic sensitivity (n = 115) assessed COVID-19 confounding. Detailed methodology is in Appendix A (separate file). 4-h wait failures grew from ∼ 3,700 to ∼ 123,000 per month (∼33-fold). Andrews sup-F = 25.4 placed the demand-side break at 2014-10 (compatible with NHS Five Year Forward View); an independent bed-occupancy break at 2020-09 aligned with NHS COVID-19 reorganisation. On the COVID-inclusive hold-out, only LSTM was significantly worse than naive under Holm correction (p = 0.029); six of seven methods showed no significant difference, but every modern-ML method had a worse point MAPE than the seasonal-naive baseline. Pre-pandemic, only the statistical methods ETS and Prophet significantly beat naive. Under horizon-matched rolling-origin one-step evaluation with Diebold-Mariano testing, no method significantly outperformed the seasonal-naive y[t-12] baseline; the apparent ML advantage held only against the weaker three-year-mean baseline. After horizon-matched, multiple-comparison-corrected evaluation, modern ML showed no reliable advantage over a strict seasonal-naive baseline on monthly system-level NHS data; the statistical methods Prophet and ETS were the only models to beat naive significantly, and only in the pre-pandemic regime. The apparent rolling-origin ML "recovery" reflected an easier one-step task and a weaker baseline rather than a genuine evaluation-scheme effect. The practical message is that complex ML should not be assumed superior for monthly NHS demand planning unless it demonstrably beats a strong seasonal-naive baseline under rolling-origin, multiple-comparison-corrected testing.