Real signal, or just luck?
A backtest that beats buy & hold is the start of the question, not the answer. With enough assets, settings and hindsight, luck alone produces winners — and they look identical to a real edge until you check. There are four independent ways a good-looking result is just luck. None of the checks below can prove a strategy works; each can only rule out one way it might be fooling you. That's still the most useful thing you can do with a backtest.
Start by trying to break it, not to confirm it. Take a result that beat buy & hold in your own backtest and run the four checks against it. More often than not the “edge” falls apart under one of them — that's not a disappointment, that's the check working.
Did the entry timing actually do anything?
The result might just be exposure. If a strategy is long most of the time, it inherits whatever the market did — the specific entry rule may have added nothing. The noise floor asks: would random entries with the same number of trades and the same holding lengths have done just as well?
The check: the noise floor — it runs free, right on your backtest result. If your strategy scores near the middle of a thousand random look-alikes, the timing wasn't doing the work.
If it passes, the entry timing wasn't only luck — not that the strategy works.
What it does NOT rule out: overfit settings, a market regime that happened to suit it, or that it still lagged buy & hold. Beating random entries is a low bar, not a pass.
Did you just try enough assets to get lucky once?
Run one strategy across every asset in the library, keep the best, and you've gone looking for luck — the top result of many tries is expected to look good even if none has an edge. The deflated Sharpe corrects the winner for how many assets you tried (it has to know that count, so it lives inside the batch scan).
The check: the deflated Sharpe on a batch scan (Batch scan) — it discounts the best result by the luck of having searched.
If it passes, we couldn't explain the winner away as the luck of trying many — not that it works forward.
What it does NOT rule out: whether that single winner is itself timing-lucky (filter 1) or curve-fitted (filter 3). It only addresses the searching.
Do the nearby settings work too — or only this exact one?
If RSI(14) beats buy & hold but RSI(13) and RSI(15) don't, you didn't find an edge — you found the one setting that fit this history. The robustness map re-runs the strategy across a grid of neighbouring settings and colours the surface. A broad plateau that beats is encouraging; a lone spike is a curve-fit.
The check: the robustness heatmap and its overfitting probability (Robustness) — a low overfitting probability means the setting that looked best on the training half of history usually stayed near the top out-of-sample too, not just on the stretch it was picked on.
If it passes, the settings weren't cherry-picked from a noisy surface — not that the strategy is real.
What it does NOT rule out: that the entire family of settings only worked in this one market regime, and how it holds up on data it was never tuned on (filter 4).
Does it hold up on data it was never tuned on?
The strongest single check. Walk-forward chooses the settings using only the past, then tests that choice on the next slice it never saw — the way you'd actually have to trade it. Most edges that look great with full hindsight decay sharply once the settings can only come from what was already known.
The check: walk-forward / out-of-sample testing (Walk-forward) — compare the tuned-with-hindsight result to the honest out-of-sample one and read the decay.
If it passes, the edge held on data it wasn't fitted to — the best evidence a backtest can give, and still not a guarantee.
What it does NOT rule out: a future regime change, and everything that separates a simulation from live money — fills, spreads, and your own discipline. The past never has to repeat.
Passing all four still isn't “it works”
These four filters are independent, and clearing every one is genuinely rare — but it removes four ways you were being fooled; it does not add proof. A strategy that survives all four is still a hypothesis about one stretch of history: it can still fail when the market changes regime, and a simulation is never live trading. The honest next step isn't to trade it — it's to read what a backtest can and can't tell you and then test it forward, in real time, before any money is at stake. Those are the later stages of the same five-stage preparation — and stopping is a free outcome at every one.
Hypothetical / simulated results — past performance is not a reliable indicator of future results. This is an educational, analytical tool, not investment advice and not a recommendation to buy or sell anything. You make all decisions and execute on your own broker.