Skip to main content
AI Fin Hub
Menu

Checklist, in Backtesting & Validation

Forecast Calibration Review Checklist

A trading or risk model that outputs probabilities is only useful if those probabilities are honest. This checklist reviews forecast calibration, the property that a stated probability matches the real frequency.

11 items. Published May 26, 2026. By AI Fin Hub Research.

Your progress

Move item by item and keep your place

Progress saves locally, so you can work through the page over multiple sessions without resetting your checklist.

0/11 complete

Sections

Work in focused batches instead of one long wall

Section 1

Phase 1: Proper scoring

3 items

Section 2

Phase 2: Reliability

3 items

Section 3

Phase 3: Discrimination

2 items

Section 4

Phase 4: Sample sufficiency

3 items

Tips

Small moves that make the checklist easier to finish

Accuracy hides overconfidence. A model that is right most of the time but says ninety-nine percent when it means seventy will size your bets disastrously, which is exactly what a proper scoring rule catches.
Calibration and discrimination are different skills. A forecast can rank outcomes perfectly and still state dishonest probabilities, and only the second property is fixable with a recalibration map.
Distrust a calibration verdict on thin data. With few observations per bucket, the reliability curve is mostly noise, so confirm the sample before concluding a model is well or badly calibrated.

Run the numbers next

  • Playgrounds Calibration Dojo

    Train your probabilistic intuition. Answer binary forecasting questions at any confidence level; track Brier score and reliability curve over time. All.

  • Calculators Returns Distribution Analyzer

    Paste a returns CSV. Histogram, normal QQ plot, skewness, excess kurtosis, Jarque-Bera test, tail-weight index. See why Sharpe alone misleads.

  • Calculators Position Sizing under Edge Variance

    Bayesian-Kelly bet sizing when your edge is itself uncertain. Compare deterministic Kelly, Bayesian-adjusted, and conservative lower-bound versions.

Sources

  1. Verification of Forecasts Expressed in Terms of Probability , Glenn W. Brier, Monthly Weather Review (1950)
  2. The Comparison and Evaluation of Forecasters , Morris H. DeGroot, Stephen E. Fienberg, The Statistician (1983)

Keep the topic connected

  • Checklist Trading Strategy Validation Checklist

    A sign-off checklist for validating a trading strategy before risking capital: data hygiene, out-of-sample testing, trial accounting, deflated Sharpe, and risk backtests.

  • Checklist Risk Model Validation Checklist

    Risk model validation checklist: backtest VaR with Kupiec and Christoffersen, check breach independence, validate fat tails, and stress the model.

  • Learn Monte Carlo Simulation

    Monte Carlo simulation in trading: when it's the right tool, when it's overkill, and the seed-discipline gotcha that ruins most published examples.

  • Learn Kelly Criterion

    What the Kelly criterion is, when full Kelly blows up, and why most working quants size at half- or quarter-Kelly.