datasheet / backtest validation / pbo
Probability of backtest overfitting (PBO)
PBO is the probability that the strategy ranked best in your backtest ranks at or below the median of all the strategies you tried, out of sample. Bailey, Borwein, López de Prado and Zhu estimate it with combinatorially symmetric cross-validation (CSCV) on the same data the search used.
1Run a strategy search
Simulated daily returns at 1% volatility, through the hub's own CSCV engine, in your browser.
With no edge, PBO swings widely from one market to the next. That is why the paper pairs it with the deflated Sharpe ratio.
Use your own returns
Paste or drop a CSV: one column per strategy, one row per period (daily returns as decimals, 0.012 = 1.2%, or as percentages). A header row of names is optional. It runs in this tab with the same engine; nothing is uploaded.
Searching 8 strategies (none with any real edge) over 2 years of daily data, the best backtest shows a Sharpe of 1.19. In 66% of 500 cross-validation splits, the in-sample winner ranked at or below the median out of sample. PBO is 66%, above the random baseline (a random pick ranks at or below the median 50% of the time): the backtest winner did worse out of sample than a random pick. Across 11 simulated markets with these settings (this one and 10 more) PBO spans 11%–66%: one backtest can't settle it. With fewer than 10 strategies (8 here), read PBO as a rough guide.
2Validation Sheet
The result of this search on one drawing: PBO, the distribution behind it, the deflated Sharpe and the method, numbered from its inputs. For a research note, a pull request or a team channel.
3How CSCV estimates it
- Stack the returns of all N strategies you tried as columns of a T × N matrix.
- Cut the rows into an even number S of blocks (S = 16 here), each ⌊T / S⌋ rows long, with the remainder added to the last block: 2 years of daily rows (T = 504) give 15 blocks of 31 rows and a last block of 39.
- For every way to pick half the blocks as in-sample (12,870 ways for S = 16), the other half is out of sample.
- Find the strategy with the best in-sample Sharpe, then its rank r̄ out of sample (1 = worst, N = best; ties share their average rank) and its relative rank ω̄ = r̄ / (N + 1).
- PBO is the share of splits where its logit λ = ln(ω̄ / (1 − ω̄)) is at or below zero: φ = ∫ f(λ) dλ from −∞ to 0.
The deflated Sharpe ratio shown beside PBO uses the null variance of the Sharpe estimator, 1/(T − 1), for its benchmark; Bailey and López de Prado (2014) deflate with the variance of the Sharpe ratios across the N trials, so this figure is a simplification of theirs.
This page samples 500 of the 12,870 splits with a fixed seed, as the hub's calculator and API do, so the same inputs always give the same answer.
4In the paper
The authors' own example backtests a strategy search where every selected configuration showed an in-sample Sharpe between 1 and 3. About 78% of their out-of-sample Sharpe ratios were negative, and CSCV put PBO at 74%.
They note that a customary Neyman–Pearson approach would reject a model whose estimated PBO exceeds 0.05. Pair it with the deflated Sharpe ratio (Bailey and López de Prado, 2014), which corrects one Sharpe ratio for how many strategies were tried.
5On your own returns, or from an agent
The search above, sent to the hub's Backtest Overfitting Score. Paste your own returns in place of the simulated ones; the page and the API compute on the same numbers.
POST https://aifinhub.io/api/run/backtest-overfitting-score content-type: application/json { "returns": [ [0.019831, 0.007275, 0.004027, … 504 daily returns], … 8 strategies, 37.0 KB in all ] } PROPOSED, NOT LIVE today this GET returns HTTP 400; use the POST above. A seeded GET would make this search one link: GET https://aifinhub.io/api/run/backtest-overfitting-score ?n=8&years=2& edge=0& seed=23
curl -s https://aifinhub.io/api/run/backtest-overfitting-score \
-H 'content-type: application/json' \
--data-binary @- <<'JSON'
{"returns":[[0.019831,0.007275,0.004027,0.003515,0.009714,0.001885,0.00219,0.017418,-0.011892,-0.000317,0.009061,0.00824…
JSON
# Copy curl copies the whole command, all 37.0 KB of it# MCP over Streamable HTTP, no sign-in: one JSON-RPC 2.0 call to the run_calculator tool curl -s https://aifinhub.io/mcp \ -H 'content-type: application/json' \ -H 'accept: application/json, text/event-stream' \ --data-binary @- <<'JSON' { "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "run_calculator", "arguments": { "calculator": "backtest-overfitting-score", "input": { "returns": [ [0.019831, 0.007275, 0.004027, … 504 daily returns], … 8 strategies, 37.0 KB in all ] } } } } JSON
Inputs that fell back to a default
max_combos500s8seed42
Every agent surface
GET /api/toolsevery calculator, its inputs, defaults and run URLhttps://aifinhub.io/mcpMCP server (POST, for MCP clients): find_calculator, run_calculator, list_calculators/.well-known/webmcp.jsonWebMCP manifest/engines/backtest-overfitting-score.jsthis engine as an ES module/llms.txtthe map for AI crawlers