AI Fin Hub

datasheet / backtest validation / pbo

Probability of backtest overfitting (PBO)

PBO is the probability that the strategy ranked best in your backtest ranks at or below the median of all the strategies you tried, out of sample. Bailey, Borwein, López de Prado and Zhu estimate it with combinatorially symmetric cross-validation (CSCV) on the same data the search used.

1Run a strategy search

Simulated daily returns at 1% volatility, through the hub's own CSCV engine, in your browser.

With no edge, PBO swings widely from one market to the next. That is why the paper pairs it with the deflated Sharpe ratio.

Use your own returns

Paste or drop a CSV: one column per strategy, one row per period (daily returns as decimals, 0.012 = 1.2%, or as percentages). A header row of names is optional. It runs in this tab with the same engine; nothing is uploaded.

Best backtest Sharpe
1.19
The one you would pick of 8.
Probability of overfitting
66%
Overfitting likely: worse than random11 markets incl. this one: 11–66%
Deflated Sharpe ratio (DSR)
0.59
A 0.95 threshold is common; this falls short.The probability that the true Sharpe beats the best Sharpe expected from 8 no-skill trials.

Searching 8 strategies (none with any real edge) over 2 years of daily data, the best backtest shows a Sharpe of 1.19. In 66% of 500 cross-validation splits, the in-sample winner ranked at or below the median out of sample. PBO is 66%, above the random baseline (a random pick ranks at or below the median 50% of the time): the backtest winner did worse out of sample than a random pick. Across 11 simulated markets with these settings (this one and 10 more) PBO spans 11%–66%: one backtest can't settle it. With fewer than 10 strategies (8 here), read PBO as a rough guide.

rank 1 of 8: 91 splitsrank 2 of 8: 86 splitsrank 3 of 8: 78 splitsrank 4 of 8: 76 splitsrank 5 of 8: 59 splitsrank 6 of 8: 62 splitsrank 7 of 8: 36 splitsrank 8 of 8: 12 splits−2−10+1+2median, λ = 066% of splits at or below← at or below the median (λ ≤ 0)held up →
Figure 1. Where the backtest winner landed out of sample: the logit λ of its out-of-sample rank in each of 500 splits, one bar per rank (λ takes at most N values, from −ln N to +ln N). At or left of zero, it ranked at or below the median.
-3-2-10+1+2+3+401234no decaymedian winner+1.62 in sample, +0.13 outout-of-sample Sharpe ↑in-sample Sharpe of the winner →median out of sample +0.13
Figure 2. How its Sharpe held up: each split's winner, in sample against out of sample. The arrow is the median winner's drop; the orange line is the median out of sample.
PBO 66% for the example search: Searching 8 strategies (none with any real edge) over 2 years of daily data, the best backtest shows a Sharpe of 1.19. In 66% of 500 cross-validation splits, the in-sample winner ranked at or below the median out of sample. PBO is 66%, above the random baseline (a random pick ranks at or below the median 50% of the time): the backtest winner did worse out of sample than a random pick. Across 11 simulated markets with these settings (this one and 10 more) PBO spans 11%–66%: one backtest can't settle it. With fewer than 10 strategies (8 here), read PBO as a rough guide.
Figure 3. This search as a Validation Sheet, AFH-V-PV1K · example. It redraws as you move the sliders.

2Validation Sheet

The result of this search on one drawing: PBO, the distribution behind it, the deflated Sharpe and the method, numbered from its inputs. For a research note, a pull request or a team channel.

PBO = share of splits where λ = ln( ω̄ / (1 − ω̄) ) ≤ 0, with ω̄ = r̄ / (N + 1)r̄ is the in-sample winner's rank among all N strategies on the held-out half (1 = worst, N = best; ties share their average rank), so λ runs from −ln N to +ln N and λ ≤ 0 means at or below the median. A random pick lands at or below the median half the time for even N and (N + 1)/(2N) of the time for odd N (the median rank counts); above that baseline, choosing by backtest did worse than choosing at random.

3How CSCV estimates it

  1. Stack the returns of all N strategies you tried as columns of a T × N matrix.
  2. Cut the rows into an even number S of blocks (S = 16 here), each ⌊T / S⌋ rows long, with the remainder added to the last block: 2 years of daily rows (T = 504) give 15 blocks of 31 rows and a last block of 39.
  3. For every way to pick half the blocks as in-sample (12,870 ways for S = 16), the other half is out of sample.
  4. Find the strategy with the best in-sample Sharpe, then its rank r̄ out of sample (1 = worst, N = best; ties share their average rank) and its relative rank ω̄ = r̄ / (N + 1).
  5. PBO is the share of splits where its logit λ = ln(ω̄ / (1 − ω̄)) is at or below zero: φ = ∫ f(λ) dλ from −∞ to 0.

The deflated Sharpe ratio shown beside PBO uses the null variance of the Sharpe estimator, 1/(T − 1), for its benchmark; Bailey and López de Prado (2014) deflate with the variance of the Sharpe ratios across the N trials, so this figure is a simplification of theirs.

This page samples 500 of the 12,870 splits with a fixed seed, as the hub's calculator and API do, so the same inputs always give the same answer.

4In the paper

The authors' own example backtests a strategy search where every selected configuration showed an in-sample Sharpe between 1 and 3. About 78% of their out-of-sample Sharpe ratios were negative, and CSCV put PBO at 74%.

They note that a customary Neyman–Pearson approach would reject a model whose estimated PBO exceeds 0.05. Pair it with the deflated Sharpe ratio (Bailey and López de Prado, 2014), which corrects one Sharpe ratio for how many strategies were tried.

5On your own returns, or from an agent

The search above, sent to the hub's Backtest Overfitting Score. Paste your own returns in place of the simulated ones; the page and the API compute on the same numbers.

POST https://aifinhub.io/api/run/backtest-overfitting-score
content-type: application/json
{
  "returns": [
    [0.019831, 0.007275, 0.004027, … 504 daily returns],
    … 8 strategies, 37.0 KB in all
  ]
}

PROPOSED, NOT LIVE today this GET returns HTTP 400; use the POST above. A seeded GET would make this search one link:
GET https://aifinhub.io/api/run/backtest-overfitting-score
    ?n=8&years=2&edge=0&seed=23
Request body37.0 KBof the API's 64 KB

Inputs that fell back to a default

  • max_combos500
  • s8
  • seed42
Open in ClaudeOpen in ChatGPT

Every agent surface

Response200 application/json · 166.2 KB, large parts folded
"calculator": "backtest-overfitting-score"
"title": "Backtest Overfitting Score"
"site": "AI Fin Hub"
"input":
"returns": 8 items
"defaults_used":
"max_combos": 500
"s": 8
"seed": 42
"result":
"nStrategies": 8
"nObservations": 504
"strategies": 8 items
"pbo": 0.662
"combinationsTested": 500
"s": 8
"bestStrategyIndex": 2
"bestStrategyName": "STRATEGY_3"
"provenance":
"run": "https://aifinhub.io/api/run/backtest-overfitting-score?returns=%5B%5B0.019831%2C0.0…" +45,935 characters
"page": "https://aifinhub.io/backtest-overfitting-score/"
"methodology": "https://aifinhub.io/backtest-overfitting-score/"
"contract": "https://aifinhub.io/contracts/backtest-overfitting-score.json"
"engine": "https://aifinhub.io/engines/backtest-overfitting-score.js"
"engine_sha256": "5ffdea060c29d70ee4df3ac7249c337be2d3b9d8f9df9292989400610d19e84f"
"build": "1a772910d31ef1740892ed0e3a065ed71e8c7406"
"disclaimer": "Planning estimates only — not financial, tax, or investment advice."
"license": "https://creativecommons.org/licenses/by/4.0/"
"cite": "AI Fin Hub, Backtest Overfitting Score (engine 5ffdea060c29), https://aifinhub.io/a…" +45,997 characters
Listing 1. What that POST returns, computed in your browser by the calculator's engine from the hub repo (build 1a77291): PBO 66%, best strategy STRATEGY_3, the figures above. Its “s” is half the blocks: s = 8 is the page's S = 16.
Bailey, D. H., Borwein, J. M., López de Prado, M., and Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4), 39–69. doi:10.21314/JCF.2016.322. Working paper: SSRN 2326253.

Education, not investment advice. Past performance does not predict future results. How we check our numbers.