Playground
Forecast Scoring Sandbox
Forecast scoring sandbox: paste a CSV of predictions to get Brier score, log loss, Murphy decomposition, bootstrap CIs and a reliability diagram.
Runs in your browser. Nothing you enter is uploaded, and no account or API key is needed.
1. Configure — paste or upload a forecast stream
One forecast per line, probability first. A header row naming prob / outcome columns is optional. Rows with a probability outside 0-1 or an outcome other than 0/1 are skipped.
Used for the 95% intervals (seeded, so they do not jitter).
Equal-width probability bins for the decomposition and diagram.
Prediction range
0.071 → 0.947
Base rate (observed)
46.0%
Brier score
0.208
Lower is better · 0 = perfect. Log loss 0.607 across 100 forecasts.
95% CI [0.170, 0.252], bins populated 10/10. Always forecasting the base rate would score 0.248; skill score 0.161 (above 0 beats it).
Decomposition (Murphy 1973)
| Component | Value | 95% CI | Reading |
|---|---|---|---|
| Reliability | 0.0090 | [0.0085, 0.0556] | Lower = better calibrated |
| Resolution | 0.0494 | [0.0320, 0.1033] | Higher = more discriminating |
| Uncertainty | 0.2484 | — | Irreducible from base rate |
| Within-bin variance | 0.0007 | — | Spread of forecasts inside each bin |
| Within-bin covariance | 0.0003 | — | Discrimination inside bins (subtracted) |
| Check: rel − res + unc + wbv − wbc | 0.2084 | — | Equals Brier (0.2084) |
BS = reliability − resolution + uncertainty (Murphy, 1973), exact only when every forecast in a bin is the same; the two within-bin terms (Stephenson, Coelho and Jolliffe, 2008) close the identity for free-form probabilities.
Reliability diagram
X axis: predicted probability bin. Y axis: observed frequency of outcome=1 in that bin. Dots on the diagonal = perfectly calibrated. Dots below the diagonal = overconfident (predictions exceed observed). Dots above = underconfident. Sparse bins are drawn lighter: with few observations, their y position is noisy and the deviation from the diagonal is not informative on its own.
Formulas
Brier(p, y) = mean( (p_i - y_i)^2 )
LogLoss(p, y) = -mean( y_i·log(p_i) + (1-y_i)·log(1-p_i) )
Bin forecasts into K equal-width buckets of predicted probability.
For bin k with n_k observations, mean pred p̄_k, observed freq ō_k:
reliability = Σ_k (n_k / N) · (p̄_k − ō_k)²
resolution = Σ_k (n_k / N) · (ō_k − ȳ)²
uncertainty = ȳ · (1 − ȳ) where ȳ = mean(y_i)
within-bin variance = (1/N) · Σ_k Σ_i∈k (p_i − p̄_k)²
within-bin covariance = (2/N) · Σ_k Σ_i∈k (p_i − p̄_k)(y_i − ō_k)
Identity: Brier = reliability − resolution + uncertainty
+ within-bin variance − within-bin covariance
95% CI: bootstrap-percentile over B resamples with replacement.How to use it
- Paste or upload binary forecasts, one per line: the probability (0 to 1) first and the outcome (1 = happened, 0 = did not) second, or load the 100-case demo.
- Set the number of reliability bins and bootstrap resamples. Both the Brier score and log loss are always computed.
- Read the Brier score and log loss with their 95% bootstrap intervals, then compare the Brier score with the base-rate benchmark under it: a skill score above 0 beats always forecasting the base rate.
- Read the decomposition and the reliability diagram: dots below the diagonal on the right mean overconfident forecasts, dots above mean underconfident ones.
- Use at least 50 forecasts before drawing conclusions. Below 50, the intervals are wide and scores are dominated by sample noise.
Questions people ask
What scoring rules are supported?
Two proper scoring rules for binary outcomes: the Brier score (mean squared error of the probabilities) and log loss (mean negative natural-log likelihood). Both reward honest, calibrated probabilities. There is no CRPS or multi-category scoring: score continuous or categorical forecasts by turning them into yes/no events first, one event per row.
What's a proper scoring rule?
A scoring rule where forecasters maximize their expected score by reporting their true beliefs. Brier and log are both proper. Improper rules (e.g., 'biggest single hit wins') incentivize gaming, like reporting extreme probabilities to maximize variance. Proper rules incentivize honesty.
When should I use Brier vs. log?
Brier is bounded and treats all errors symmetrically. Log loss is unbounded: predicting 0.001 on something that happens costs about 6.9 on that one forecast (the tool clamps probabilities at 10⁻⁹, so the worst single loss is about 20.7). Use log loss when overconfidence must be punished hard; use Brier for a more forgiving, bounded metric. Reporting both is common.
How do I compare two forecasters with different forecast counts?
Use the mean score per forecast, which is what the sandbox reports, and compare the bootstrap 95% intervals: if they overlap heavily, the difference is not reliable. With fewer than about 50 forecasts each, the intervals are wide and comparisons should not drive decisions.
Does the sandbox support binary, categorical, and continuous outcomes?
Binary only: each row is the probability that an event happens and an outcome of 1 (it happened) or 0 (it did not). A categorical forecast can be scored one category at a time as yes/no events, and a continuous forecast through threshold events such as 'return above 0'. There is no distributional (CRPS) scoring.
Is there a real, public forecast stream I can look at as an example?
Yes. Good Judgment Open scores forecasters' probabilities with the Brier score once each question resolves (https://www.gjopen.com/faq), which is the input this sandbox expects: probability-tagged forecasts paired with realized outcomes. FrontierPicks, a sister site, keeps a public, dated record of AI stock picks graded against the S&P 500 (https://frontierpicks.com/track-record/), but its picks state no probability, so to score them here you would have to assign one yourself.
Guides, formulas and worked examples
- Guide How to Calibrate Probability Forecasts
- Comparison Brier Score vs Log Loss
- Comparison Cohen's Kappa vs ICC for Agreement
- Comparison Point Estimate vs Prediction Interval
- Formula Fundamental Law of Active Management Formula
- Formula Information Coefficient Formula
- Checklist Forecast Calibration Review Checklist
Related tools
- Playgrounds Calibration Dojo
Train your probabilistic intuition. Answer binary forecasting questions at any confidence level; track Brier score and reliability curve over time. All.
- Calculators Backtest Overfitting Score
Upload a backtest trade log and compute Probability of Backtest Overfitting (PBO), Deflated Sharpe Ratio, and the odds your edge survives live trading.
- Calculators Kelly Criterion Calculator
Size positions with full, half, quarter, or eighth Kelly and stress-test the choice with a drawdown Monte Carlo simulator. Client-side. Private by default.
Articles
- 10 min read Brier Scores and Log Loss for Forecasters
Two proper scoring rules for probabilistic forecasts, why Brier decomposes into reliability plus resolution, and why log loss punishes overconfident wrongness.
- 10 min read Research Diary Schema: Auditable LLM Research
A 12-field append-only schema that captures every idea — including rejected ones — to unlock calibration, proper scoring, and post-hoc overfitting analysis.
- 12 min read Calibration Dojo vs Platt vs Isotonic: The Right Tool
Engine: Brier 0.202, reliability 0.009 on a 200-forecast sample tape (well calibrated). Three calibration methods address different LLM-output failure modes.
Use it from code
The same calculation as a JavaScript module you can import. It runs where you import it, with no request, key or rate limit.
import { compute } from "https://aifinhub.io/engines/forecast-scoring-sandbox.js";