Don't Trust, Verify: How to Backtest Like a Scientist

You have a strategy. “Buy when RSI is 30.” You look at the chart. “Look! It worked here. And here. And here!”

I call this Confirmation Bias. I know it well because I used to do it. You ignored the 10 times it failed.

Key Findings:

• Slippage Reality: In my backtesting analysis, I found returns are often overstated by 0.5-3% annually because traders ignore real-world slippage.
• Sample Size: I refuse to trade a strategy without a minimum of 100 trades to prove statistical significance.
• The Blind Test: I rely on Bar Replay because it is the only method that replicates the psychological pressure of live execution.

The Scientific Method

1. Hypothesis: “Buying RSI 30 works in an Uptrend.”
2. Experiment: Run a simulation over 2023 data.
3. Result: Win Rate 45%. R:R 1:1.5. Profit Factor 1.2.
4. Conclusion: It is marginally profitable, but not great. A ratio that thin only tells you something once you check how the profit factor was built, and the same caution applies to any risk-adjusted score quoted from a short test window.

The “Blind” Test

Use Bar Replay. Go back to a random date (e.g., March 12, 2019). Press “Play.” Make decisions candle-by-candle. You will feel the pain of drawdowns. You will feel the doubt. This is the only way to build Conviction.

Optimization vs. Curve Fitting

Be careful. If you tweak the RSI to “28” because it worked better in 2023, you are Curve Fitting. You are tailoring the suit to fit the mannequin, not the human. Robust strategies work across wide parameter ranges (RSI 25-35). The disciplined version of that check is walk forward analysis, where you tune on one block of history and score the rules only on the block that comes after it.

The data itself can lie to you as well. On futures-based markets the chart you test is stitched together from expiring contracts, so an oil backtest reads price steps that came from the rollover rather than from anything a trader could have traded.

Curve fitting is not the only way a great-looking test deceives you. If the indicator itself repaints, the problem runs deeper: the backtest is faked at the source because the tool quietly rewrites its old signals to sit on the winners. No parameter discipline fixes that; only a signal that locks at the candle close does.

Conclusion

I learned the hard way: if I haven’t backtested it, I don’t own it. I am just renting it from a guru. And when the guru is wrong, I will panic. I own my edge.

Does your strategy work in the market, or does it only work in your imagination?