When Backtesting isn't enough

Asad Dossani
Backtesting is a fundamental part of creating an algorithmic trading strategy. Backtesting is when we test out a trading strategy on historical data, to see how well it would have performed, had we implemented it in the past. For any strategy we use in a live market, a successful backtest is critical.

But backtesting alone is not enough. More often than not, a successful backtest does not mean that the trading strategy will be successful live. This is due to a concept called over-fitting.

Suppose we test out a large number of trading strategies on our dataset. By chance alone, a few of these strategies will work very well. This will occur even if the trading strategy is no good.

For example, supposed I choose trades by flipping a coin. And then I test this coin flip strategy on my historical data. Ideally we want the backtest results to be bad. But sometimes, just due to luck, this coin flipping strategy will make money in the backtest.

The problem here is that if I implement this same strategy live, it will most likely lose money. Therefore, the successful backtest is misleading.

In reality, we don't know ahead of time if our trading strategy is good or not. Suppose we come up with a trading strategy. Then we backtest it and get successful results. Does this tell us that our trading strategy is good?

Unfortunately not. From the coin flipping example, you can see that even a bad trading strategy can have a successful backtest. Of course, it may be that our strategy is actually a good one. But we need more information before we can know this for sure. We certainly need to do more before we should implement it live.

The problem gets worse if we try out multiple trading strategies on the same set of data. If we backtest 10 variants of our bad trading strategy, it is likely that one or two of them will result in profitable backtest. If we backtest 100 variants of our bad trading strategy, it is practically certain than many of them will appear profitable.

This is the idea of over-fitting. We find bad strategies that happen to work on our dataset just by chance. But these strategies aren't going to work once we implement them live.

When I'm working on trading strategies for Alpha Trader, over-fitting is something I work hard to avoid. There are numerous steps I take before implementing a strategy live. In the next article, I'll reveal exactly what steps I take to ensure that a successful backtest leads to a successful trading strategy. And don't worry, it isn't anything fancy. Just common sense logical approaches to solve the problem of over-fitting.

Do you backtest your own trading strategies? Share your views in the Club or share your comments here.

Get Asad Dossani's Best Short Term Investment
Opportunities Delivered Straight To Your Inbox!

Sign Up For Profit Hunter Today... It's Free!
We hate spam as much as you do. Check out our Privacy Policy and Terms Of Use
We request your view! Post a comment on "When Backtesting isn't enough". Click here!
3 Responses to "When Backtesting isn't enough"
RS Rathore
28 Mar, 2015
In fact, just by flipping of a coil and to follow that is hardly going to bring snow-ball to any one. Only once in a Blue Moon coin-flipping may prove useful merely by sheer luck e.g. flipping of a coil by captain of any cricket team asking for Head or Tail, but there is no surety whether the asked for Head or Tail would be as wished for. However, I have not been able to understand at all the "When Backtesting isn't enough. Sorry, no further comments. Like 
Sadanand Joshi
27 Mar, 2015
flipping a coin example is a joke when you say strategy. Also could not understood why one should club multiple strategies to test. Logically only one strategy is expected to test at a time. Do you mean something different when you say multiple strategy? Like 
27 Mar, 2015
Sir I have back tested around 20 different indicators on a fixed data set. I have also used variants of these 20 indicators. Say 100 DMA instead of 200 DMA etc. The idea is to build a trading system by using signals from 5 - 7 indicators . Is this a wrong way to build a system? Regards VishalLike 
We request your view! Post a comment on "When Backtesting isn't enough". Click here!