Testing: How Much is Enough?

How much is enough? That's one of the key questions of trading system design: how much historic data is enough to give reliable results? Think of it this way: if you tested a trading system on two years of a bull market, what happens when the market turns down? The answer: probably not anything good. One reader with this on their mind recently wrote in with this comment: "If you can't back test this over at least three types of economic cycles, it's utterly coin flipping prone to failure." I answered in the comment space, but because this is an important issue I wanted to devote a post to it.

I'm going to refer again to Robert Pardo's great new book, The Evaluation and Optimization of Trading Strategies, which updates his 1991 bible of trading system development, Design, Testing and Optimization of Trading Systems. He addresses this issue extensively. What he says is quite interesting. He says there is no fixed period of time a system should be backtested over, so long as the period includes at least one of each of the major market types: bull, bear, congested and cyclic. As well, it should include lots of varying volatility periods. He says 10 years of data is sufficient. One especially interesting point he makes is that a longer test period, while incorporating more variety of market conditions and likely more robust going forward, will tend to be less responsive (i.e., less profitable) to current price action. It will also need less frequent reoptimization. Conversely, a shorter time frame will be more responsive but less adaptable to dramatic changes in market conditions. I tend to lean toward the longer test period, since it's often hard to know until it's too late when conditions have changed. But one alternative approach would be to include a price indicator that does that for you. That could allow a more responsive system.

Finally, Pardo writes about a key test for a trading strategy that is relevant to the length of the test window: ensuring the strategy uses no more than 10 percent of the available degrees of freedom (or datapoints). This includes the "overhead" number of weeks before you get your first signal. I've written about this not long ago in this post. For example, if we have 665 weeks of data (or available degrees of freedom) and a trading system with 12 trading rules, this means we can't use a moving average rule that exceeds 48 weeks [(12+48)*10+60]. To read more about all this, see pages 130-1, 140-4 and 171 of Pardo's book. Good luck this week!