Backtesting without fooling yourself

Key takeaways

• Look ahead bias uses information that was not available at the moment of the simulated trade, and even a one day slip can turn a mediocre rule into a spectacular one.
• Survivorship bias comes from testing only on securities or funds that still exist today, which removes the failures and inflates returns.
• Trading costs scale with turnover: annual drag is roughly turnover times cost per dollar traded, and a high turnover strategy can lose most of its gross Sharpe to costs.
• Trying many variants and reporting the best inflates the Sharpe ratio; the deflated Sharpe ratio asks whether the best result beats what the best of that many random strategies would show.
• Out of sample and walk forward testing, with the out of sample data used only once, is the main defence against overfitting.

Look ahead bias and point in time data

Look ahead bias is using information in a simulated decision that would not have been known at the time. The crude version is a coding error: computing a signal from today's closing price and then earning today's return on it. The subtle versions live in the data. Company fundamentals are a common trap. A quarter ending 31 March is typically reported weeks later, and databases often store the figures against the period end date. A backtest that joins on period end will trade on earnings before they were published.

Restatements make it worse. When a company revises past figures, many databases overwrite the original. A backtest run today then sees the corrected number, which no investor could have traded on at the time. The cure is point in time data: every value stored with the date it became known, and every revision stored as a new row rather than an overwrite. Index membership, credit ratings, analyst estimates and economic data (which is heavily revised) all need the same treatment.

A useful habit is to write every signal as a function of what was known at the decision time and to lag positions explicitly: the signal computed at the close of day t sets the position held from day t plus 1. If a rule's performance collapses when it is lagged one extra day, it was probably depending on information it should not have had, or on execution it could not have achieved.

Survivorship bias

Survivorship bias arises when the test universe includes only securities, funds or managers that survived to the present. Companies that went bankrupt, were delisted or were acquired at depressed prices disappear from many free datasets. A stock strategy tested on today's index members is effectively told in advance which companies would not fail. Strategies that buy cheap or distressed stocks are especially flattered, because their failures are exactly the ones that vanish.

Fund databases have the same problem in a different form. Hedge funds and private funds report voluntarily, and funds that do badly tend to stop reporting or close. A peer group average computed only from funds still reporting overstates what an investor in the category actually earned. Backfill bias compounds it: a fund that joins a database often adds its earlier, successful history, while funds whose early history was poor never join.

Transaction costs and slippage

Every trade costs something: commissions, half the bid-ask spread, market impact (the price moving against you because you are trading), and for short positions the cost of borrowing stock. Together these are often expressed in basis points per dollar traded. For liquid large cap US stocks traded in moderate size, all-in costs of a few basis points are realistic; for small caps, emerging markets or large orders relative to daily volume, tens of basis points or more.

Slippage is the gap between the price the backtest assumed and the price achievable. Backtests that trade at the same close that generated the signal, or at the day's open after an overnight signal without any impact, assume execution no one gets. Impact grows with order size, roughly with the square root of the order as a fraction of daily volume in most empirical models, so a strategy that works with $10m may not work with $1bn. That is capacity, and it belongs in every serious backtest.

Turnover is what turns small costs into large ones. Annual turnover here means the total dollar value traded in a year divided by capital. A monthly rebalanced factor portfolio might trade 2 to 5 times its capital a year; a daily rebalanced short-term strategy can trade 50 times or more. The annual cost drag is approximately turnover multiplied by cost per dollar traded.

Overfitting, multiple testing and the deflated Sharpe

Overfitting is fitting a model to the noise in a sample rather than to a pattern that will persist. Every parameter adds freedom: the lookback window, the threshold, the rebalance day, the universe filter, the stop loss. With enough choices, almost any dataset yields a rule with an impressive historical Sharpe. The rule then fails out of sample because the pattern it found was never there.

Multiple testing is the statistical side of the same problem. If a researcher tests 100 variants of a strategy that has no true edge, some will look good by chance, and reporting the best one is selecting on luck. The estimated Sharpe ratio of a strategy has a standard error of roughly one divided by the square root of the number of years, for Sharpe ratios near zero and normally distributed returns. Over ten years that is about 0.32. The best of many such noisy estimates will be well above zero even when every true Sharpe is zero.

David Bailey and Marcos Lopez de Prado formalised the correction as the deflated Sharpe ratio. The idea: estimate how high the best Sharpe would be among N independent trials with no skill, given the sample length and the shape of returns, and ask whether the observed Sharpe clears that bar with confidence. The full statistic also adjusts for skewness and fat tails. The practical lesson is simpler: a backtest result means little without the number of variants tried to find it, and researchers should record every trial, not just the winners.

Out of sample and walk forward testing

The basic defence is to keep data the model never saw. Split the history: develop and tune on the earlier part (in sample), then test once on the later part (out of sample). The rule is once. If the out of sample result disappoints and the researcher goes back to tune again, the out of sample period has become in sample, and its protective value is gone.

Walk forward testing repeats the split through time. Fit the parameters on a training window, trade the next period with those fixed parameters, then roll both windows forward and repeat. Stitching together only the traded periods gives a track record in which every decision was made with data available at the time, which is closer to how a live strategy behaves. It also shows whether the chosen parameters are stable or jump around from window to window, which is itself a warning sign.

The final test is time. A paper trading or small capital live period, with real execution, catches what simulations cannot: data delivered late, orders that do not fill, borrow that is not available. Allocators weigh live track records far above backtests for this reason, and the gap between a manager's backtest and their live returns is one of the most informative numbers in due diligence.

Templates for this lesson

Included with a Premieur membership.

• NotebookBacktest notebookA moving average crossover end to end: lagged position, trading costs, Sharpe, maximum drawdown and the look ahead comparison.Download