Skip to main content
AI Fin Hub
Menu

Explainer, in Backtesting & Validation

Overfitting

Overfitting occurs when model parameters are tuned tightly enough to historical data that the strategy describes random structure rather than persistent signal. The hallmark: in-sample Sharpe is excellent, out-of-sample Sharpe collapses. The mechanism is multiple-testing — every parameter explored is a hypothesis tested, and with enough trials, random noise produces strategies that look profitable purely by chance.

2 questions. Published May 10, 2026. By AI Fin Hub Research.

Definition

Overfitting

Overfitting occurs when model parameters are tuned tightly enough to historical data that the strategy describes random structure rather than persistent signal. The hallmark: in-sample Sharpe is excellent, out-of-sample Sharpe collapses. The mechanism is multiple-testing — every parameter explored is a hypothesis tested, and with enough trials, random noise produces strategies that look profitable purely by chance.

Why it matters

Most retail and a meaningful fraction of institutional backtests are overfit. The strategies look great until they go live, at which point the ratio between live and backtested Sharpe (the haircut) typically lands at 0.3 to 0.5. Diagnosing overfitting is more valuable than designing a new strategy.

How it works

Hold out a true out-of-sample period and never look at it during development. Use walk-forward validation. Compute Probability of Backtest Overfitting (PBO) by comparing in-sample and out-of-sample rankings across parameter combinations. Apply Bailey-Lopez de Prado's deflated Sharpe ratio to penalize multiple testing. Treat any strategy whose deflated Sharpe collapses to near-zero as overfit until proven otherwise.

Example

Ten thousand random strategies on equity returns

Strategies tested

10,000

Best in-sample Sharpe

2.4

Same strategy out-of-sample Sharpe

0.3

Deflated Sharpe

0.1 (not significant)

The 'best' strategy from a 10k search has an in-sample Sharpe that looks impressive and a deflated Sharpe that says you found random luck. Live trading would lose money.

Key takeaways

1

Every parameter you tune is a hypothesis you test — multiple testing is real.

2

Deflated Sharpe is the cleanest single-number defense against overfitting claims.

3

If you can't reproduce the strategy on a held-out, never-touched dataset, it's overfit.

Run the numbers next

  • Calculators Backtest Overfitting Score

    Upload a backtest trade log and compute Probability of Backtest Overfitting (PBO), Deflated Sharpe Ratio, and the odds your edge survives live trading.

  • Calculators Deflated Sharpe Ratio Calculator

    Bailey & López de Prado deflated Sharpe — corrects observed Sharpe for selection bias across K trials. Reports deflated Sharpe, PSR (probability of skill).

  • Playgrounds Walk-Forward Validator

    Upload a returns CSV. Rolling or expanding IS/OOS windows, per-window Sharpe, walk-forward efficiency, and a concatenated OOS equity curve. Catches regime.

Questions people ask next

How many strategies do I have to test before I'm overfitting?

There's no clean threshold, but the deflated-Sharpe penalty grows with log(N_trials). Twenty trials cuts roughly 0.3 off your Sharpe expectation under the null. A thousand trials cuts roughly 0.6. Track and report N_trials honestly.

Does walk-forward eliminate overfitting?

It mitigates, doesn't eliminate. If your walk-forward window is short or you re-tune parameters frequently against the recent walk-forward result, you can still overfit at the meta-level. The ultimate defense is a true out-of-sample period the model has never touched.

Sources

  1. The Probability of Backtest Overfitting , Bailey, Borwein, Lopez de Prado, Zhu, Journal of Computational Finance 20(4), 39-69 (2017)

Keep the topic connected