Do not ship a forecasting model just because its chart looks plausible. Forecast Proof is a zero-dependency CLI that checks a backtest CSV against a seasonal-naive baseline and audits probabilistic quantiles before the forecast reaches production.
It is model-agnostic: use it with TimesFM, Chronos, classical models, spreadsheets, or any system that can export CSV.
Foundation models make high-quality forecasts easier to generate, but deployment still needs three boring answers:
- Does the model beat “use the value from last season” on the same rows?
- Do quantiles stay ordered instead of crossing?
- Does a stated 90% quantile behave like 90% on held-out actuals?
Forecast Proof turns those checks into one command and a CI-friendly exit code.
Python 3.9+ is the only requirement.
python -m pip install .Or run the single file directly without installing:
python forecast_proof.py examples/retail.csv --group store --season 7Your backtest CSV needs increasing timestamps, actual and point-prediction columns, and one or more quantile columns named q1 through q99:
timestamp,store,actual,prediction,q10,q50,q90
2026-08-01,shanghai,100,104,101,104,107
2026-08-02,shanghai,110,106,103,106,109Run the included retail example with a weekly seasonal baseline:
forecast-proof examples/retail.csv --group store --season 7Write machine-readable output:
forecast-proof examples/retail.csv --group store --season 7 --format json --output report.jsonExit codes are stable: 0 passes every gate, 1 is a valid forecast that fails a gate, and 2 means the input is invalid.
- Point quality: MAE, WAPE, bias, and skill versus a seasonal-naive forecast on identical comparable rows.
- Probabilistic quality: mean pinball loss, empirical quantile calibration, central interval coverage and width.
- Structural safety: finite numbers, required columns, strictly increasing timestamps per group, and quantile crossing.
- Default gates: positive skill versus naive, zero crossing rows, and no quantile calibration error above 10 percentage points.
Use --calibration-tolerance when sample size or policy requires a different threshold. The tool never edits the input.
The CLI streams CSV rows into per-series lists, computes a lagged seasonal-naive value within each series, then evaluates the model and baseline on the same subset. Quantiles are discovered from headers and checked row by row. Rendering is separated from analysis so the same result can become Markdown or deterministic JSON.
The implementation is one standard-library Python module. There is no model runtime, dataframe library, telemetry, network call, or hidden upload.
- name: Gate forecast
run: forecast-proof backtest.csv --group sku --season 7 --format jsonThe included GitHub Actions workflow runs unit tests, compilation, an end-to-end passing example, and an invalid-input exit-code check.
- This validates a completed holdout/backtest file; it does not create temporal splits or detect leakage in feature engineering.
- Timestamp ordering is lexical, so use sortable ISO-8601 timestamps.
- Calibration is empirical and can be noisy on small samples. Choose a tolerance appropriate for your sample size and risk.
- The seasonal baseline is deliberately simple. Add domain baselines only when they change a real deployment decision.
- A passing report is evidence, not a guarantee of future accuracy or business value.
Forecast Proof runs locally and makes no network requests. It reads only the CSV path supplied and writes only when --output is set. CSV content may be sensitive; do not commit production data or generated reports without review. Formula-like spreadsheet cells are treated as invalid numbers and never evaluated.
See CONTRIBUTING.md. Security reports should follow the private process described there.
MIT. See LICENSE.
Suggested description: Zero-dependency CI gate for point and probabilistic time-series forecasts.
Suggested topics: time-series, forecasting, probabilistic-forecasting, timesfm, mlops, model-evaluation, python-cli.