Out-of-sample means a model was scored on data withheld from every stage of fitting, and in financial forecasting that separation is the closest published proxy to live trading. The measure exists because the alternative is mostly memory: a model graded on its own training data can look skilled without being anything. The probability of backtest overfitting, formalized in the Journal of Computational Finance in 2014, can exceed 50 percent when many model configurations are tried on a single history.
Uzu News publishes information and analysis about forecasting methods, not investment advice. This explainer defines the vocabulary, shows how the split is supposed to be done, and states what an out-of-sample result cannot establish even when it is done correctly.
What does out-of-sample actually test?
Backtesting is the practice of replaying a strategy or model on historical data and recording its performance. An out-of-sample test scores the model on data that played no part in choosing its features, parameters, or stopping point. The claim being tested is narrow: on data the model has never influenced and never seen, did the measured behavior hold?
The definition has to be stated that strictly because leakage erodes it quietly. If a researcher tuned a hyperparameter on the test set, normalized features using statistics computed over the full history, or stopped training when test performance peaked, the test set is no longer out-of-sample in any meaningful sense. It has become part of the fitting process through the researcher's choices.
A useful copula definition: calibration is the match between a model's stated probabilities and observed frequencies. Calibration measured on training data is nearly uninformative, because flexible models can memorize the frequencies. Calibration measured on held-out data is evidence, bounded by how much held-out data exists.
How is the split supposed to be done?
The textbook procedure is a three-way split: fit on a training set, tune on a validation set, and report once on a test set touched last. Financial data breaks the textbook assumptions, because returns are close to serially dependent and regimes persist. A split placed at a random date leaks information across the boundary through overlapping windows and features computed over trailing periods.
The standard adaptation is walk-forward evaluation: fit on data up to a date, predict the next period, advance the window, and repeat. Marcos López de Prado's 2018 book Advances in Financial Machine Learning describes purged cross-validation, which removes training observations whose label windows overlap the test period. Both approaches reduce leakage; neither eliminates the deeper problem, which is that there is only one history.
That scarcity is the structural weakness. An equity model tested on 2010 through 2020 has seen one long bull market, two sharp drawdowns, and no repetition of either. The evaluator's question shifts from did it pass to what did it pass on.
Why is out-of-sample performance still overstated?
Because the split is usually reported as if it were the only test, when it was the last of many. Halbert White's Reality Check, published in Econometrica in 2000, addressed exactly this: when many candidate models are evaluated on the same dataset, the best one's apparent performance is inflated by selection. The test corrects for the number of trials.
The 2014 Journal of Computational Finance paper by Bailey, Borwein, López de Prado and Zhu gave the problem a working measure: the probability of backtest overfitting, the chance that the configuration chosen as best in-sample lands in the bottom half of configurations out-of-sample. Under their simulations, trying enough configurations on a fixed history makes a false discovery the expected outcome rather than the accident.
Bailey and López de Prado's Deflated Sharpe Ratio, published in the Journal of Portfolio Management the same year, adjusts a reported Sharpe ratio for the number of trials, the variance of trial results, and the sample length. A reported Sharpe ratio of 1.0 earned as the best of 100 trials on three years of data can carry a deflated value near zero. The correction is arithmetic, not opinion.
How should a reader check a paper's claims?
The check is a short list of questions, and a paper that answers none of them has not reported a test in the sense above.
- How many configurations were tried before the reported one, and were all trials disclosed?
- What defined the train and test periods, and were features computed without full-history statistics?
- What was the baseline, and does the reported metric beat it by more than the trial count would explain?
- What asset universe and date range, and how many distinct regimes does that range contain?
- Is the performance claim labeled as the authors' own result, or as independently verified?
Question five matters more than it appears. Replication attempts of published market-timing results routinely find weaker effects than the originals, and the site's standing rule applies to reading as much as to writing: claimed performance and verified performance are different things. The same discipline appears in regulation. The Federal Reserve's Supervisory Guidance on Model Risk Management, SR 11-7 of 2011, requires banks to validate models against outcomes the models did not influence, and to document known weaknesses.
What can an out-of-sample test not establish?
It cannot establish that a model will work live. The out-of-sample period is still one draw from a distribution, and the paper's own evaluation window may contain no regime resembling the reader's deployment period. Performance stated on 2015 through 2019 says nothing about a rate shock, whatever the metric.
It also cannot establish absence of overfitting when the research process itself was iterative. If the researcher saw intermediate results and adjusted the design, the number of effective trials exceeds the number disclosed, and every correction formula that assumes a known trial count undercorrects.
Finally, it cannot establish transfer. A model that calibrates well on US equities out-of-sample has been tested on US equities and nothing else. Cross-asset or cross-market claims require cross-asset or cross-market evaluation windows, stated explicitly. What remains unknown after a clean out-of-sample test is still substantial; the honest paper says so in its own limitations section, and the reader should treat one that does not as a warning rather than a promise.
