Skip to content

Benchmarks & synthetic data

Under testing

These benchmarks are how ClimAID is currently tested. Because the data are synthetic, good results here are necessary but not sufficient: real-data validation is still to come.

All synthetic datasets are generated by benchmarks/generators.py in the repository, with fixed random seeds, so every result can be reproduced exactly. Each dataset has a known truth, so model output can be checked against the right answer.

Datasets

Generator What it contains Used to test
climate() Monthly climate 2004–2022: seasonal temperature, monsoon-shaped rainfall, humidity, slowly varying ENSO Shared input
seasonal() Cases driven by temperature (1 month earlier), rainfall (2 months) and ENSO (3 months); calendar month explains ~70% of variation Forecasts, lag recovery
nonseasonal() Cases driven only by ENSO (4 months earlier) plus slow drift; no annual cycle Forecasts when seasonality is absent
enso_interaction() Seasonal truth plus El Niño amplifying the temperature effect ENSO interaction features
realistic_hard() Seasonal climate response with a threshold rainfall effect, saturating temperature effect, a reporting change (+60% from 2016), two serotype-driven outbreak years, a 2020 reporting collapse, ~5% missing months plus a 3-month gap, and two data-entry errors (10×) Robustness to real-world data problems
cmip6() Three imaginary climate models (own biases and warming sensitivities) × SSP2-4.5 / SSP5-8.5, 2015–2060 Scenario outlook, bias correction
multi_district() Six districts sharing one climate response but with different seasonal climates Multi-district pooling
thermal_optimum() A hot district whose transmission follows a thermal-suitability curve Temperature-curve option

Counts are drawn with more scatter than a simple Poisson model (negative binomial), as in real surveillance data.

Why clean synthetic data flatters the models

The clean datasets are built the way the models assume the world works: smooth climate effects, fixed lags inside the search ranges, a relationship that never changes, and clean noise. Real data also have reporting changes, testing campaigns, new serotypes, gaps and errors. realistic_hard() adds these, and is the more informative test.

Running the benchmark

python benchmarks/run_benchmarks.py --tuning fast

It makes 12-month forecasts from four start dates for each dataset through the full forecast_v2() path (tuning, backtests, calibration) and writes benchmarks/results/summary.csv and summary.md. The key column, WIS_vs_seasonal_naive, compares each model's probabilistic score with the "same as recent years" baseline: below 1.0 means better than the baseline.

Results are summarised on the Status & validation page.

--models a,b,... chooses the models (default: seasonal naive, Poisson, random forest, gradient boosting).


SARIMAX

SARIMAX was benchmarked alongside the four standard benchmark models (same run, Fast tuning, 12-month forecasts from four start dates; results in benchmarks/results/sarimax/):

python benchmarks/run_benchmarks.py --tuning fast \
    --models seasonal_naive,poisson,random_forest,gradient_boosting,sarimax --out benchmarks/results/sarimax

Score relative to the baseline (WIS ratio; lower is better) and coverage of the 80% likely range:

Dataset SARIMAX Best other single model Ensemble (with SARIMAX) SARIMAX 80% coverage
Seasonal (clean) 0.63 0.63 (random forest) 0.60 0.98
Non-seasonal (clean) 0.58 1.04 (Poisson) 0.86 0.94
Realistic (hard) 0.84 0.63 (Poisson) 0.63 0.72

SARIMAX was the best single model on the non-seasonal data, where the other models were no better than the baseline, and tied for best on the seasonal data. On the realistic dataset (reporting change, outbreak years, missing months, data-entry errors) it was clearly worse than Poisson regression and the tree models, and its 80% ranges contained the truth only 72% of the time. Adding it improved the ensemble slightly on the clean datasets and left it unchanged on the realistic one. Because the realistic case is the closest to real surveillance data, SARIMAX is offered as an optional model and benchmark rather than a default; this can be revisited once ClimAID has been tested on real data.