Overfitting a trading strategy: what it is and how to spot it

What overfitting is

Prices move for many reasons, and a lot of every chart is noise: wiggles that won’t happen the same way again. Overfitting is when a strategy’s rules end up fitted to that noise. The rules match the past very closely, but what they match was never going to repeat.

Think of a key cut to fit one lock perfectly, down to its scratches. It opens that lock. It may not open the next one, even if the next one looks the same from the outside.

The two lines above show the idea. The first passes through every close, so it describes these candles perfectly. That’s the trouble: it has as many bends as there are candles, so it describes nothing else. The smoother line misses most closes, but it’s following something bigger than one candle. Trading rules can be bent the same way, one setting at a time.

You’ll also hear it called curve fitting. It means the same thing.

How overfitting happens, step by step

Here’s a made-up example. The rule: buy when a candle closes above the highest high of the 3 candles before it. The stop is 2 points below the entry and the target 3 points above. It isn’t a strategy, only a way to show what happens.

Neither new rule came from an idea about how markets move. Each was found by looking at a losing trade and asking what made it different. With only four trades, something always looks different. Then the picture at the top of this page runs all three rules on the next 16 candles. Of three trades, one reaches its target and two reach their stops. The rules had described the past, not something that carried on.

Real overfitting is rarely this plain. It hides in dozens of small choices: a length of 13 instead of 14, a stop a tick wider, one more filter. Each looks reasonable on its own.

Signs a backtest may be overfit

• One small change moves everything. Above, changing the look-back from 3 candles to 4 changes which trades are taken. If a result only holds at one exact setting, and falls apart one step either side, the setting may be fitted to noise.
• Many rules, few trades. Every rule you add can explain away a trade. With lots of rules and a small sample of trades, a clean-looking result is easy to get by chance.
• Rules with no reason. A filter that exists only because it removed a bad trade in the past, like rule 2 above, is a warning sign.
• Very specific numbers. A stop of 7.25 points and a length of 23 candles might be real choices, or they might be the one combination out of hundreds that suited this history.
• It only works on one stretch. Very different results on another year, another market or another timeframe.
• Many versions tried. The more versions you test on the same history, the more likely one of them looks good by luck alone.

How traders check for overfitting

Keep some history back. Build and adjust the rules on one part of the history. Then test them once, without changing anything, on a part you kept back. The first part is called in-sample. The part kept back is called out-of-sample. If the kept-back part looks very different, the rules may fit the first part’s quirks. Read about in-sample and out-of-sample testing.

Walk forward. Walk-forward testing repeats that split again and again, moving through the history: build on one window, test on the stretch after it, move on, and do it again. Read about walk-forward testing.

Look at the neighbours. Run the settings one step either side of the ones you chose. If the results change a little, the setting isn’t balanced on a knife edge. If they change a lot, it may be.

Write the reason first. Decide what a rule is for, and why, before you see what it does to the results. A rule added only because the results improved is the one to look at twice.

Test on new prices. Run the finished rules on a demo account as prices happen. That’s called forward testing. Read about forward testing.

None of these proves a strategy isn’t overfit. They make it harder for luck to pass as a pattern.

Often confused with

Look-ahead bias. A backtest using a price it couldn’t have known yet. That’s a mistake in how the test runs. Overfitting is a problem with the rules, even when the test runs perfectly. Read about look-ahead bias.

A small sample. Too few trades to say much. It makes overfitting easier, but a simple rule with few trades isn’t overfit, only untested. Read about sample size.

The market changing. A strategy that suited one kind of market can stop suiting it when the market changes. That can happen to rules that were never overfit. Read about market regimes.

Backtest and live differences. Fills, slippage and fees can make live trading differ from a backtest, overfit or not. Read why backtest and live results differ.

Where the term overfitting comes from

The word comes from statistics. When a model has as many moving parts as there are data points, it can match every point and learn nothing. Statisticians and, later, people building computer models of all kinds use it for any model that has learned the noise in its data instead of the pattern. Traders use it for strategies that have done the same with price history.

Testing your own rules on dates you kept back

See every trade your rules take, on the dates you choose

Wildflower’s Backtest runs your own rules over the dates you pick and draws every trade on the chart. So you can build on one stretch of history, then test the same version, unchanged, on dates you kept back. Compare two runs shows what changed between two versions of your rules and, on the same dates, the trades only one of them took. With a Backtest or Bot plan, Try a range runs your strategy across values you type and lays every run out in a grid. Nothing is marked or ranked. What you make of it is up to you.

Set it up and see how it works, free. Only pay when you backtest on real market history.

Join waitlistSee the Backtest

Common questions

Related

• What is backtesting
• Backtest vs live results
• Glossary

In the help centre

• Backtest your strategy
• My backtests and comparing runs
• Try a range