Probability of Backtest Overfitting (PBO): Bailey et al.

Why it matters

A backtest search over many configurations almost always produces an impressive in-sample winner, so a high in-sample Sharpe ratio is weak evidence on its own. PBO measures how often that winner fails to beat the median out-of-sample, using only the data the search already used. The paper notes that a customary Neyman-Pearson approach would reject models whose estimated PBO exceeds 0.05. PBO above 0.5 means the selection did worse than picking a configuration at random.

How it works

Following the paper's Algorithm 2.3: stack the N configurations' returns into a T x N matrix M. Split its rows into an even number S of equal blocks. Form all C(S, S/2) combinations of S/2 blocks; each is a training set J and its complement is the test set. For each combination, find the configuration n* with the best in-sample metric, take its out-of-sample rank r among the N, set w = r/(N+1) and logit = ln(w/(1-w)). PBO is the share of combinations with logit at or below zero; S = 16 gives 12,870 combinations.