Real signal, or just luck?
A backtest that beats buy & hold is the start of the question, not the answer. With enough assets, settings and hindsight, luck alone produces winners — and they look identical to a real edge until you check. There are four independent ways a good-looking result is just luck. None of the checks below can prove a strategy works; each can only rule out one way it might be fooling you. That's still the most useful thing you can do with a backtest.
Start by trying to break it, not to confirm it. Take a result that beat buy & hold in your own backtest and run the four checks against it. More often than not the “edge” falls apart under one of them — that's not a disappointment, that's the check working.
Was the result just market exposure, not the rule?
Still doesn’t rule out: overfit settings, a lucky regime, or still lagging buy & hold
Did you just test enough assets to get lucky once?
Still doesn’t rule out: that winner’s own timing-luck or curve-fit
Do nearby settings work, or only this exact one?
Still doesn’t rule out: a regime that suited it; how it holds on unseen data
Does it hold up on data it was never tuned on?
Still doesn’t rule out: a future regime change, and live fills, spreads & discipline
Still a hypothesis about one stretch of history — not proof it works.
Illustrative — four independent checks, not a measured survival rate. They run in any order and each only rules out one way a backtest fools you. Clearing all four is genuinely rare and worth doing; it removes ways you were fooled, but it is not proof a strategy works, and it is not a signal to act.
Did the trade timing actually do anything?
The result might just be exposure. If a strategy is long most of the time, it inherits whatever the market did — the specific entry rule may have added nothing. The noise floor asks: would random entries with the same number of trades and the same holding lengths have done just as well?
The check: the noise floor — it runs free, right on your backtest result. If your strategy scores near the middle of a thousand random look-alikes, the timing wasn't doing the work.
If it passes, the trade timing (entries and exits, stops and targets included) wasn't only luck — not that the strategy works.
What it does NOT rule out: overfit settings, a market regime that happened to suit it, or that it still lagged buy & hold. Beating random entries is a low bar, not a pass.
Did you just try enough assets to get lucky once?
Run one strategy across every asset in the library, keep the best, and you've gone looking for luck — the top result of many tries is expected to look good even if none has an edge. The deflated Sharpe corrects the winner for how many assets you tried (it has to know that count, so it lives inside the batch scan).
The check: the deflated Sharpe on a batch scan (Batch scan) — it discounts the best result by the luck of having searched.
If it passes, we couldn't explain the winner away as the luck of trying many — not that it works forward.
What it does NOT rule out: whether that single winner is itself timing-lucky (filter 1) or curve-fitted (filter 3). It only addresses the searching.
Do the nearby settings work too — or only this exact one?
If RSI(14) beats buy & hold but RSI(13) and RSI(15) don't, you didn't find an edge — you found the one setting that fit this history. The robustness map re-runs the strategy across a grid of neighbouring settings and colours the shape of the surface (not a per-setting number — an in-sample figure is fit to the past). A broad plateau that beats is a shape worth testing further, out-of-sample; a lone spike is a curve-fit.
The check: the robustness heatmap and its overfitting probability (Robustness) — a low overfitting probability means the setting that looked best on the training half of history usually stayed near the top out-of-sample too, not just on the stretch it was picked on.
If it passes, the settings weren't cherry-picked from a noisy surface — not that the strategy is real.
What it does NOT rule out: that the entire family of settings only worked in this one market regime, and how it holds up on data it was never tuned on (filter 4).
Does it hold up on data it was never tuned on?
The strongest single check. Walk-forward chooses the settings using only the past, then tests that choice on the next slice it never saw — the way you'd actually have to trade it. Most edges that look great with full hindsight decay sharply once the settings can only come from what was already known.
The check: walk-forward / out-of-sample testing (Walk-forward) — compare the tuned-with-hindsight result to the honest out-of-sample one and read the decay.
If it passes, the edge held on data it wasn't fitted to — the best evidence a backtest can give, and still not a guarantee.
What it does NOT rule out: a future regime change, and everything that separates a simulation from live money — fills, spreads, and your own discipline. The past never has to repeat.
Passing all four still isn't “it works”
These four filters are independent, and clearing every one is genuinely rare — but it removes four ways you were being fooled; it does not add proof. A strategy that survives all four is still a hypothesis about one stretch of history: it can still fail when the market changes regime, and a simulation is never live trading. The honest next step isn't to trade it — it's to read what a backtest can and can't tell you and then test it forward, in real time, before any money is at stake. Those are the later stages of the same five-stage preparation — and stopping is a free outcome at every one.
These four checks are for a result you built yourself. Vetting a backtest or track record someone else is selling you is a different job — how to spot a fake backtest runs through the red flags.
Hypothetical / simulated results — past performance is not a reliable indicator of future results. This is an educational, analytical tool, not investment advice and not a recommendation to buy or sell anything. You make all decisions and execute on your own broker.