Playground
Calibration Dojo
The quiz has 24 true-or-false statements, each with a sourced answer; the confidence you give each one is scored with the Brier score and a reliability curve.
On the documented defaults, computed by this tool's engine · the same answer as JSON
Train probabilistic intuition. Binary forecasting questions at any confidence level; track Brier score + reliability curve over time.
01Playground
Runs in your browser. Nothing you enter is uploaded, and no account or API key is needed.
Brier score
No answers yet
Lower is better. Perfect calibration scores 0.000; always answering 50% scores 0.250
0 answered. Answers are stored in this browser only.
1. Read the statement
Loading…
Reliability curve
Dot size ∝ number of answers in that decile. Diagonal = perfect calibration.
History persists in your browser's localStorage only.
02Receipt
What the answer above ran on, how it is computed, where the data comes from, and a link that reopens it.
Inputs
The inputs you enter above. Nothing is uploaded.
Method
How the hub's tools are built and checked
Engine /engines/calibration-dojo.js, sha256 680362bf514a. The page, the API and MCP run the same file.
Source
Your inputs / BYO key
Page updated .
Permalink
The link carries the inputs you set, so it reopens this result.
03For agents
Any agent can run this calculator by URL or MCP and get the same answer as JSON: the inputs that used defaults, the result and a line to cite. The call below follows the inputs you set above. In code, import the same engine; it runs where you import it, with no request or key.
GET https://aifinhub.io/api/run/calibration-dojo curl -s 'https://aifinhub.io/api/run/calibration-dojo'
# Markdown instead of JSON:
curl -s 'https://aifinhub.io/api/run/calibration-dojo?format=md' # MCP over Streamable HTTP, no sign-in: run_calculator
curl -s https://aifinhub.io/mcp \
-H 'content-type: application/json' \
-H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"run_calculator","arguments":{"calculator":"calibration-dojo","input":{}}}}' import { compute } from "https://aifinhub.io/engines/calibration-dojo.js";
const result = compute(input); {
"calculator": "calibration-dojo",
"title": "Calibration Dojo",
"site": "AI Fin Hub",
"input": {},
"defaults_used": {},
"result": {
"questions": [
{
"id": "q1",
"statement": "The speed of light in vacuum is greater than 300,000 km/s.",
"truth": false,
"category": "physics",
"source": "CODATA 2018 fundamental physical constants",
"rationale": "The exact value is 299,792,458 m/s = 299,792.458 km/s, which is less than 300,000 km/s."
},
{
"id": "q2",
"statement": "The Eiffel Tower is taller than 300 meters including antennas.",
"truth": true,
"category": "geography",
"source": "Société d'Exploitation de la Tour Eiffel",
"rationale": "Height to tip including antennas is ~330 m."
},
{
"id": "q3",
"statement": "Over the last 50 years (to 2024), the US stock market has had more positive calendar years than negative calendar years.",
"truth": true,
"category": "finance-generic",
"source": "S&P / Damodaran annual returns series",
"rationale": "Roughly 73–76% of calendar years are positive over long rolling windows; far more ups than downs."
},
"… 21 more of 24"
]
},
"provenance": {
"run": "https://aifinhub.io/api/run/calibration-dojo",
"page": "https://aifinhub.io/calibration-dojo/",
"methodology": "https://aifinhub.io/calibration-dojo/",
"contract": "https://aifinhub.io/contracts/calibration-dojo.json",
"engine": "https://aifinhub.io/engines/calibration-dojo.js",
"engine_sha256": "680362bf514aa21d1bf091ca0960e59b9db76210249767f187adeaf8de3d55c3",
"build": "1a772910d31ef1740892ed0e3a065ed71e8c7406"
},
"disclaimer": "Planning estimates only — not financial, tax, or investment advice.",
"license": "https://creativecommons.org/licenses/by/4.0/",
"cite": "AI Fin Hub, Calibration Dojo (engine 680362bf514a), https://aifinhub.io/api/run/calibration-dojo"
} How to use it
- Read the statement and set the probability that it is true (0% = certainly false, 100% = certainly true), then click Commit probability.
- The reveal shows the answer, your Brier score for that question, and the rationale and source behind the answer. Click Next question to continue.
- Read the reliability curve: dots on the diagonal mean your 70% answers come true about 70% of the time. Dots below the diagonal on the right mean you are overconfident.
- Compare your Brier score with the base-rate benchmark shown under it. A skill score above 0 means you beat always answering the base rate; reliability and resolution show whether calibration or discrimination is holding you back.
- Answer at least 50 questions before drawing conclusions; progress is saved in this browser and Clear local history resets it.
Questions people ask
What's calibration in this context?
How well a model's stated confidence matches its empirical hit rate. A 70%-confident forecast that's right 70% of the time is well-calibrated; a 90%-confident forecast that's right 60% of the time is overconfident. The dojo trains this with immediate feedback: you commit a probability to a true-or-false statement, see the answer and its source, and watch your reliability curve fill in.
How is the Brier score calculated?
Brier score is the mean squared error between probability forecast and outcome. For a single forecast: (probability − outcome)². For a series: average across forecasts. Lower is better: 0 is perfect, and answering 50% on everything scores exactly 0.25. The dojo also shows the score of always answering the base rate (the share of statements that were true), base rate × (1 − base rate), and your skill score against it.
What's the difference between calibration and resolution?
Calibration is whether your stated probabilities match observed frequencies. Resolution is whether you give different probabilities to different outcomes (vs. 50% on everything). A forecaster who always says the base rate is perfectly calibrated but has zero resolution, and is useless. The dojo reports both terms of the Murphy decomposition from decile bins: reliability (the calibration gap, lower is better) and resolution (higher is better). Brier score combines them.
How many forecasts before I get a stable calibration estimate?
At least 50, ideally 100 or more. The reliability curve plots, for each 10%-wide probability bin, how often the statements you put in that bin were true; with a handful of answers per bin one surprise moves a dot a long way, and the dojo shows no confidence interval, so read early curves loosely. The bank has 24 questions; after that the dojo re-asks them at random.
Why does a low Brier score not always mean I'm right?
A Brier score is meaningful only relative to a benchmark. Forecasting 'AAPL up tomorrow' at 50% gives Brier 0.25 with zero skill. The dojo always reports your score against the benchmark of forecasting the base rate, so you can see whether your work is adding signal or just hitting the average.
Where can I see forecast calibration scored in public on real markets?
Probability-scored forecasts are easiest to find outside markets: Good Judgment Open scores every forecast with the Brier score once its question resolves, the same squared error the dojo uses, and ranks forecasters against the crowd (https://www.gjopen.com/faq). Public records of stock calls usually grade direction rather than probability. FrontierPicks, a sister site, grades each AI stock pick against the S&P 500 with a kill line written down before the pick opens (https://frontierpicks.com/track-record/): a record of commitment you can check, but not a calibration score, because no probability is stated.
Related tools
- Calculators Returns Distribution Analyzer
Paste a returns CSV. Histogram, normal QQ plot, skewness, excess kurtosis, Jarque-Bera test, tail-weight index. See why Sharpe alone misleads.
- Calculators Backtest Overfitting Score
Upload a returns CSV (one column per strategy you tried) or simulate a strategy search: the probability of backtest overfitting (PBO) by CSCV and the Deflated Sharpe Ratio.
- Calculators Kelly Criterion Calculator
Size positions with full, half, quarter, or eighth Kelly and stress-test the choice with a drawdown Monte Carlo simulator. Client-side. Private by default.