Calibration Dojo

How well a model's stated confidence matches its empirical hit rate. A 70%-confident forecast that's right 70% of the time is well-calibrated; a 90%-confident forecast that's right 60% of the time is overconfident. The dojo lets you self-train calibration using forecast feedback loops.

Brier score is the mean squared error between probability forecast and outcome. For a single forecast: (probability − outcome)². For a series: average across forecasts. Lower is better. A 50/50 random forecaster scores 0.25; a perfectly calibrated forecaster with discrimination scores below 0.20.

Calibration is whether your stated probabilities match observed frequencies. Resolution is whether you give different probabilities to different outcomes (vs. 50% on everything). A forecaster who always says 50% is perfectly calibrated but has zero resolution — they're useless. Brier score combines both.

At least 50, ideally 100+. The dojo shows the calibration curve (predicted probability vs. observed frequency in bins) — it's noisy below 50 forecasts. The methodology page documents the bootstrap CI on the curve.

A Brier score is meaningful only relative to a benchmark. Forecasting 'AAPL up tomorrow' at 50% gives Brier 0.25 with zero skill. The dojo always reports your score against the benchmark of forecasting the base rate, so you can see whether your work is adding signal or just hitting the average.