← sources

How I stopped fooling myself with backtests, and it had nothing to do with finding a better metric

source: reddit (reddit:algotrading)published 2026-11-18 08:22ingested 2026-11-19 02:40

retained text

Follow-up to the overfitting stuff from a few weeks back, the strategy that looked flawless in the backtest and came apart the moment conditions changed. What it taught me wasn't some new metric. It was that I'd been letting the backtest tell me what "good" looked like, and a backtest will always crown the nicest looking curve. Which is exactly the curve you get from fitting the past.

The trap is simple. If you optimise for a single number, usually net profit, the search will happily give you a strategy that nailed the past and nothing else. The optimiser can't tell a real edge from a lucky coincidence, it just maximises whatever you point it at. Point it at profit and it finds you the one run that got lucky.

So I stopped judging a strategy on any single number. Now a candidate has to hold up across a whole spread of things at once, not just print a big return. How exactly I weigh that up is my own thing and still evolving, so I won't pretend there's a magic formula. The principle is the part that matters: a monster return sitting on a drawdown that would have blown me out in real life is a bad strategy, not a good one, whatever the profit line says. Getting the score to punish that is the whole point.

Honestly though, the scoring matters less than two things people skip.

First, none of it counts unless it survives out of sample and walk forward, on data the optimiser never touched. A great result on the data you built on is worth nothing. That's just the strategy reciting its own training set back to you.

Second, and this is the one that still catches me, you can overfit the selection itself. Try enough candidates and one clears whatever bar you set purely by luck. So I keep the out of sample window genuinely untouched, and I try to limit how many times I go back to the well. It's a discipline more than a formula, and it's the part I'm least sure I've nailed. "Try fewer things" is honestly about as far as I've got, so if someone here has a real defence against overfitting the selection step, not just the single strategy, I'd love to hear how you do it.

document fetched 2026-11-19 00:45 · canonical https://reddit.com/r/algotrading/comments/1xz9z7t/how_i_stopped_fooling_myself_with_backtests_and