We overfit a strategy on purpose

In this August 2026 experiment, the best of 72 BTCUSDT hourly EMA configurations by in-sample Sharpe ratio returned +17.96% in the selection window and −14.32% in the following evaluation window. Its Sharpe moved from 0.88 to −1.21. The result is a concrete example of multiple testing and winner selection; it does not estimate how every optimized strategy will behave.

The selected configuration also recorded 12.6% maximum drawdown across 91 in-sample trades. Its following evaluation period recorded 18.1% drawdown. At the moment of selection, that later period had not informed the winning parameter choice.

The rest of the grid matters as much as the winner, because testing more choices gives chance more opportunities to produce an attractive maximum.

The protocol

The surviving result artifact records:

• Strategy: EMA crossover on BTCUSDT, 1-hour candles
• In-sample: 1 January 2024 to 30 June 2025, the only data used to choose parameters
• Out-of-sample: 1 July 2025 to 1 August 2026, never touched during selection
• Grid: 72 fast/slow combinations (fast 5–50, slow 20–200, keeping slow at least double fast)
• Selection: highest in-sample Sharpe
• Costs: labelled “full realism” on every run

Then the study evaluated all 72 combinations on both windows. The champion's OOS period had not informed its initial selection, but it ceased to be untouched once the researchers inspected the grid. No later final holdout is recorded.

Bailey and coauthors' paper on the probability of backtest overfitting explains the general problem: repeated selection can exhaust an ordinary holdout. This article supplies one historical example rather than claiming to calculate that probability for AlphaProve's grid.

The champion, and the nine runners-up

(That's ordered by in-sample Sharpe, so the second row's slightly higher return isn't a typo. Ranking by raw return swaps the top two and changes nothing that follows.)

The ten configurations with the highest in-sample Sharpe all lost money in the recorded OOS period. A different selection rule could have chosen another row, so this table supports a claim about this ranking, not every possible rule.

Across the whole grid, 14 of 72 combinations were profitable in sample and two of 72 were profitable out of sample. The two positive OOS returns were +1.33% and +0.50% over the recorded period.

The part everyone gets wrong

Here's the obvious objection. Bitcoin rose 153% during the in-sample window and fell 41% during the out-of-sample one. Of course everything broke. That's regime change, not overfitting.

Fair objection, and the full grid answers it. Compare the distributions instead of the winner:

The median returns were similar across the two samples: −17.69% and −16.04%, while the best return fell from +18.06% to +1.33%. That pattern is consistent with optimism being concentrated at the selected top of the first distribution. It does not rule out regime change: Bitcoin rose strongly in the first window and fell in the second, and this experiment was not designed to identify how much of the gap came from market conditions versus selection.

The narrower overfitting lesson is that the maximum of a searched surface deserves more skepticism than its median and full distribution.

In-sample ranking isn't worthless, it's just not enough

Here's the nuance we didn't expect. If in-sample results were pure noise, the in-sample and out-of-sample rankings would be unrelated. They aren't. The Spearman rank correlation between them is +0.394 on return and +0.279 on Sharpe.

The rankings therefore had a positive association across these two samples. Slower parameter sets tended to rank better than faster ones, plausibly because they traded less and paid less; the experiment did not isolate that cause. See our cost ladder experiment for a separate historical cost comparison.

A positive rank correlation is not the same thing as a profitable selection rule. The two positive OOS configurations ranked 13th and 17th in sample. Choosing them after seeing that outcome would simply introduce another layer of look-ahead.

Does forward-walking fix it?

A common response to overfitting is to reselect parameters on a rolling basis: optimize on an earlier window, trade the next one, and repeat. The historical follow-up used BTCUSDT hourly data from April 1, 2025 through August 1, 2026, three-month IS windows, three-month OOS windows, and the engine's then-default 12-combination EMA grid. It recorded five OOS batches and 116 total trades.

Total: -20.35%, against the frozen champion's -14.32%.

Forward-walking didn't save it. We're including this because the marketing version of this article would have stopped one section earlier, and because the result teaches two things.

First, the third row records no allocation. That shows the selector could leave a batch empty under its historical rules. It does not establish that every candidate truly lacked positive expectancy, which is unknown.

Second, the selected parameters alternated between 12/50 and 30/100 around the empty batch. Instability is a warning worth inspecting, but it is not proof of noise: a changing market can also change which candidate ranks first. Forward-walk analysis makes that sequence visible instead of collapsing it into one optimized curve.

The lesson is not that rolling re-optimization is useless. It can expose how a selection process behaves as data advance, but no validation method manufactures an edge the rule family does not have. In this historical run, the crossover lost money under the rolling process too.

What to do with this

Define holdouts before selection. An OOS window should not influence the choice it evaluates. Once its result changes the research process, describe it as development evidence and reserve a later period for a final check if one is needed.

Report the distribution with the maximum. Here, 14 of 72 combinations were profitable in sample and the median lost 17.69%. Those facts make the selected maximum easier to interpret without claiming they identify why every row won or lost.

Be suspicious in proportion to how hard you searched. One backtest of a hypothesis you formed in advance is weak evidence. The best of 72 is much weaker evidence, because you've given noise 72 chances to look like skill.

Check whether the winners cluster. A broad plateau is less sensitive to a small parameter change than one sharp peak surrounded by losses. Neither shape proves a durable effect, but the isolated peak is the more fragile candidate.

Honest caveats

One asset, one strategy family, and two fixed windows cannot establish a general failure rate for optimization or EMA crossovers. The venue, exact engine revision, sizing rule, and numeric cost settings are unknown in the surviving grid artifact. The forward-walk artifact records its schedule and totals but likewise does not create a final unseen test.

That said, "the strategy had no edge to begin with" isn't a defence of the method. You can't know that before you run the test, and the sweep's +17.96% winner is exactly what "it has an edge" looks like from inside the in-sample window.

The finding this experiment supports is narrower: the selected top of this 72-configuration sweep failed in the next recorded period, while the distribution showed how exceptional its in-sample result was. The next practical step is to predefine a chronological process; walk-forward testing explained shows that procedure and its limits.