Ten independent trials are enough to produce a backtest with an in-sample Sharpe ratio of 1.57 from strategies whose expected out-of-sample Sharpe ratio is zero. That figure is the expected maximum, not an unlucky tail, under the framework David Bailey, Jonathan Borwein, Marcos Lopez de Prado and Qiji Jim Zhu published in the Notices of the American Mathematical Society in May 2014.
The quantity doing the work in that sentence is the one most published backtests omit: N, the number of configurations a researcher evaluated before reporting the best one. A backtest is a simulation of a decision rule's performance on historical data. It stops being a test and becomes a search the moment more than one rule is evaluated, and a search across enough candidates will find an attractive equity curve in noise alone.
What exactly is backtest overfitting?
Backtest overfitting is the selection of a configuration whose measured historical performance reflects the number of variants tried rather than any repeatable effect. It is not a coding error and not lookahead bias. The rule is fitted correctly to history; the failure is that history was searched many times and only the winner was reported.
Out-of-sample means evaluated on data that was used neither to fit the model nor to select it. The second half of that definition is the part that fails quietly. A holdout period reused across dozens of candidate specifications has been used for selection, whatever the code comments say, and its results carry the same selection bias as the fitting sample.
Bailey and his co-authors put the reporting failure in blunt terms, writing that "not reporting the number of trials (N) involved in identifying a successful backtest is a similar kind of fraud" (p. 466). Their conclusion is equally direct: because most published backtests do not report N, many of them may be overfitted, and the paper warns that "positive backtested performance will often be followed by negative investment results" (p. 468).
How much history does a backtest actually need?
Enough history that the best result among N trials cannot plausibly be noise. The paper's Theorem 2 formalizes this as a Minimum Backtest Length: the years of data needed to avoid selecting a strategy with an in-sample Sharpe ratio of E[max N] among N independent strategies whose expected out-of-sample Sharpe ratio is zero.
The bound is MinBTL < 2 ln[N] / E[max N]². It scales with the logarithm of the trial count and inversely with the square of the Sharpe ratio being claimed, which is why it bites hardest on short samples and modest performance claims. The paper's own illustration, in its Figure 2 on p. 462, sets the expected maximum Sharpe ratio at 1 and asks how many independent trials a given sample length can absorb.
| Sample length available | Maximum independent trials before an expected in-sample Sharpe ratio of 1 arises from strategies with zero true Sharpe |
|---|---|
| 2 years | 7 |
| 5 years | 45 |
Both figures come from the same paper and the same assumptions: independent trials, Sharpe ratio estimates treated as Gaussian, and a true out-of-sample Sharpe ratio of zero for every candidate. Correlated configurations, which is what a real parameter sweep produces, are not the case the bound is derived for.
Why can more in-sample optimization make out-of-sample results worse?
Because under some conditions the two are negatively linked rather than merely uncorrelated. Where the paper introduces compensation effects, imposed by global constraints or by serial dependence in returns, it reports that "a strongly negative linear relation between performance IS and OOS arises, indicating that the more we optimize IS, the worse the OOS performance" (p. 465).
Its Proposition 3 states the ordering directly: with those global constraints imposed, a configuration with the higher in-sample Sharpe ratio has the lower out-of-sample Sharpe ratio. That inverts the intuition a backtest is usually asked to support. It does not mean optimization always destroys performance; it means the sign of the relationship is a property of the setting, not a constant, and cannot be assumed from the in-sample number alone.
What does the multiple-testing literature add?
It supplies the same correction at the scale of an entire research field. Campbell Harvey, Yan Liu and Heqing Zhu, writing in the Review of Financial Studies in 2016, catalogued 316 factors from 313 articles studying cross-sectional return patterns, then applied multiple-testing adjustments to the reported t-statistics rather than accepting each finding on its own.
Their headline threshold is a raised bar for new work: "a new factor needs to clear a much higher hurdle, with a t-statistic greater than 3.0" (p. 5), against the conventional 2.0. Applied backward to the existing literature, their tabulation finds that of 296 published significant factors, 158 would be considered false discoveries under Bonferroni, 142 under Holm, 132 under Benjamini-Hochberg-Yekutieli at 1% and 80 at 5% (p. 37).
The spread between 80 and 158 is not a defect in the analysis; it is the answer changing with the error rate being controlled. Bonferroni and Holm control the family-wise error rate, the probability of any false positive at all. The Benjamini-Hochberg-Yekutieli procedure controls the false discovery rate, the expected share of false positives among rejections. Neither result names an individual factor as false.
How do bank supervisors treat backtesting?
As monitoring, not discovery. The Federal Reserve and the Office of the Comptroller of the Currency issued joint guidance on model risk management, SR 11-7, on April 4, 2011, applying to supervised banking organizations in proportion to each one's size, nature and complexity. Backtesting appears there as a subordinate practice inside a larger validation structure.
The guidance defines outcomes analysis as comparing model outputs to corresponding actual outcomes, and describes back-testing as one form of it, involving the comparison of actual outcomes with model forecasts. Validation itself is defined as the processes intended to verify that models perform as expected, in line with their design objectives and business uses.
The letter sets out three core validation elements: evaluation of conceptual soundness, ongoing monitoring, and outcomes analysis. The ordering is the point for anyone reading a performance claim. A favorable backtest satisfies at most the third element, and only for the period tested; it says nothing about whether the model's design is defensible or whether it is still behaving as designed today.
What has to accompany backtested numbers in an advertisement?
In the United States, backtested results shown by a registered investment adviser fall under the Securities and Exchange Commission's marketing rule as hypothetical performance, which the rule prohibits in advertisements unless specific conditions are met. The rule's compliance date was November 4, 2022, eighteen months after its May 4, 2021 effective date.
Under the SEC's small-entity compliance guide, an adviser must adopt and implement policies and procedures reasonably designed to ensure that the performance is relevant to the likely financial situation and investment objectives of the intended audience, and must provide certain information underlying the hypothetical performance. The guide also notes that hypothetical performance does not include performance generated by interactive analysis tools, and that it does not qualify for the one-on-one communication exclusion unless provided in response to an unsolicited investor request or to a private fund investor.
That is a description of what the rule requires, not an assessment of any firm's compliance with it. What it establishes for a reader is narrower and useful: a backtested track record is a regulated category of claim, and the disclosures attached to it are part of the evidence, not decoration around it.
What none of this establishes
The Minimum Backtest Length bound is derived for independent trials. Real parameter sweeps are heavily correlated, so the effective number of independent trials is smaller than the raw count of runs, and the bound functions as a discipline on reporting rather than a precise threshold for any particular study.
Neither paper measures live trading results. The Bailey framework is a statistical argument about selection under simulation; the Harvey, Liu and Zhu tabulation is an argument about published t-statistics. Transaction costs, slippage, financing, capacity and market impact are outside both, and a strategy that survives every correction described here can still fail on those grounds.
Nor can any of it identify which specific published result is spurious. The false-discovery counts are properties of a set of 296 findings, and they move by a factor of roughly two depending on which error rate is controlled. No named factor, firm, or product is implicated by those numbers, and this article does not evaluate any.
Finally, a reported trial count cannot be verified from outside. A researcher who reports N is reporting only their own trials; specifications tried and abandoned by others on the same public dataset are invisible to every correction discussed here, which means the adjusted thresholds are floors rather than guarantees. None of the above is investment advice, and no figure in it describes what any market or model will do next.
