Forecast Calibration Review Checklist
A trading or risk model that outputs probabilities is only useful if those probabilities are honest. This checklist reviews forecast calibration, the property that a stated probability matches the real frequency.
Checklist Progress
Move item by item and keep your place
Progress saves locally, so you can work through the page over multiple sessions without resetting your checklist.
Checklist Sections
Work in focused batches instead of one long wall
Section 1
Phase 1: Proper scoring
Section 2
Phase 2: Reliability
Section 3
Phase 3: Discrimination
Section 4
Phase 4: Sample sufficiency
Pro Tips
Small moves that make the checklist easier to finish
Try These Tools
Run the numbers next
Calibration Dojo
Train your probabilistic intuition. Answer binary forecasting questions at any confidence level; track Brier score and reliability curve over time. All.
Returns Distribution Analyzer
Paste a returns CSV. Histogram, normal-overlay, QQ plot, skewness, excess kurtosis, Jarque-Bera test, tail-weight index. See why Sharpe alone misleads.
Position Sizing under Edge Variance
Bayesian-Kelly bet sizing when your edge is itself uncertain. Compare deterministic Kelly, Bayesian-adjusted, and conservative lower-bound versions.
Sources & References
- Verification of Forecasts Expressed in Terms of Probability — Glenn W. Brier, Monthly Weather Review (1950)
- The Comparison and Evaluation of Forecasters — Morris H. DeGroot, Stephen E. Fienberg, The Statistician (1983)
Related Content
Keep the topic connected
Trading Strategy Validation Checklist
A sign-off checklist for validating a trading strategy before risking capital: data hygiene, out-of-sample testing, trial accounting, deflated Sharpe, and risk backtests.
Risk Model Validation Checklist
Risk model validation checklist: backtest VaR with Kupiec and Christoffersen, check breach independence, validate fat tails, and stress the model.
Monte Carlo Simulation
Monte Carlo simulation in trading: when it's the right tool, when it's overkill, and the seed-discipline gotcha that ruins most published examples.
Kelly Criterion
What the Kelly criterion is, when full Kelly blows up, and why most working quants size at half- or quarter-Kelly.