Playground
Forecast Scoring Sandbox
Forecast scoring sandbox: paste a CSV of predictions to get Brier score, log loss, Murphy decomposition, bootstrap CIs and a reliability diagram.
- Inputs
- Paste + configure
- Runtime
- 1–15 s
- Privacy
- Client-side · no upload
- API key
- Not required
- Methodology
- Open →
1 · Configure — paste or upload a forecast stream
Used for 95% CI on every metric.
Bin count for decomposition + diagram.
Prediction range
0.071 → 0.947
Base rate (observed)
46.0%
Brier score
0.208
Lower is better · 0 = perfect. Log loss 0.607 across 100 forecasts.
95% CI [0.170, 0.252] · bins populated 10/10
Decomposition (Murphy 1973)
| Component | Value | 95% CI | Reading |
|---|---|---|---|
| Reliability | 0.0090 | [0.0085, 0.0556] | Lower = better calibrated |
| Resolution | 0.0494 | [0.0320, 0.1033] | Higher = more discriminating |
| Uncertainty | 0.2484 | — | Irreducible from base rate |
| Check: rel − res + unc | 0.2080 | — | Should equal Brier (0.2084) |
BS = reliability − resolution + uncertainty (Murphy, 1973)
Reliability diagram
X axis: predicted probability bin. Y axis: observed frequency of outcome=1 in that bin. Dots on the diagonal = perfectly calibrated. Dots below the diagonal = overconfident (predictions exceed observed). Dots above = underconfident. Sparse bins are drawn lighter: with few observations, their y position is noisy and the deviation from the diagonal is not informative on its own.
Formulas
Brier(p, y) = mean( (p_i - y_i)^2 ) LogLoss(p, y) = -mean( y_i·log(p_i) + (1-y_i)·log(1-p_i) ) Bin forecasts into K equal-width buckets of predicted probability. For bin k with n_k observations, mean pred p̄_k, observed freq ō_k: reliability = Σ_k (n_k / N) · (p̄_k − ō_k)² resolution = Σ_k (n_k / N) · (ō_k − ȳ)² uncertainty = ȳ · (1 − ȳ) where ȳ = mean(y_i) Identity: Brier = reliability − resolution + uncertainty 95% CI: bootstrap-percentile over B resamples with replacement.
See the methodology page for derivations, references, and why sparse bins are shown lighter.
How to use
Step-by-step
- 1
Upload or enter probabilistic forecasts paired with realized outcomes (binary, categorical, or continuous).
- 2
Pick scoring rule: Brier (squared error), log (penalize overconfidence harshly), CRPS (continuous distributions).
- 3
Read your aggregate score per forecast. Compare against the base-rate benchmark — beating the benchmark = skill.
- 4
Read the calibration plot to see where you're miscalibrated (overconfident or underconfident at specific probability ranges).
- 5
Run at least 50 forecasts before drawing conclusions. Below 50, scores are dominated by sample noise.
For agents
Use in an agent
Same math, same result shape as the UI above — as a static ES module. No HTTP request, no auth, no rate limit.
import { compute } from "https://aifinhub.io/engines/forecast-scoring-sandbox.js";Contract: /contracts/forecast-scoring-sandbox.json Full agent guide →
Glossary references
Terms used by this tool
Questions people ask next
FAQ
What scoring rules are supported?
Brier score (squared error), log score (log-likelihood penalty for overconfidence), and continuous ranked probability score (CRPS, for full distributional forecasts). All three are proper scoring rules — they reward calibrated probabilities.
What's a proper scoring rule?
A scoring rule where forecasters maximize their expected score by reporting their true beliefs. Brier and log are both proper. Improper rules (e.g., 'biggest single hit wins') incentivize gaming, like reporting extreme probabilities to maximize variance. Proper rules incentivize honesty.
When should I use Brier vs. log?
Brier is bounded and treats all errors symmetrically. Log is unbounded — predicting 0.001 on something that turns out true gives a catastrophic score. Use log when you want to severely penalize overconfidence; use Brier when you want a more forgiving metric. Most academic forecasting uses both.
How do I compare two forecasters with different forecast counts?
Use mean score per forecast, not total score. The sandbox does this automatically. The methodology page warns about the small-sample bias — comparisons with fewer than 50 forecasts each are noisy and shouldn't drive decisions.
Does the sandbox support binary, categorical, and continuous outcomes?
Yes. Binary: probability of YES. Categorical: probability vector summing to 1. Continuous: forecast distribution (the tool supports parametric Normal and t, plus user-supplied quantile triples). CRPS is the canonical continuous-outcome scoring rule.
Related deep dive
All articles →Read further
Long-form context behind the tool output.
- Tutorial · Runnable·10 min
Brier Scores and Log Loss for Forecasters
Two proper scoring rules for probabilistic forecasts, why Brier decomposes into reliability plus resolution, and why log loss punishes overconfident wrongness.
Read - Methodology · Opinion·10 min
Research Diary Schema: Auditable LLM Research
A 12-field append-only schema that captures every idea — including rejected ones — to unlock calibration, proper scoring, and post-hoc overfitting analysis.
Read - Comparison · Benchmark·12 min
Calibration Dojo vs Platt vs Isotonic: The Right Tool
Engine: Brier 0.202, reliability 0.009 on a 200-forecast sample tape (well calibrated). Three calibration methods address different LLM-output failure modes.
Read
Complementary tools
Users of this tool often explore
Calibration Dojo
Train your probabilistic intuition. Answer binary forecasting questions at any confidence level; track Brier score and reliability curve over time. All.
Backtest Overfitting Score
Upload a backtest trade log and compute Probability of Backtest Overfitting (PBO), Deflated Sharpe Ratio, and the odds your edge survives live trading.
Fractional Kelly Sizer
Size positions with full, half, quarter, or eighth Kelly and stress-test the choice with a drawdown Monte Carlo simulator. Client-side. Private by default.