CXO Study: Some Good Points... And Some Not So Good

Thankfully, Steve LeCompte over at CXOAdvisory.com has taken down his erroneous post about my S&P 500 trading setup and reevaluated the data in a new post today. He raises a good point about how "trading friction" (i.e. transaction costs) would have eaten into the past profitability of the setup, which I'm going to include in my ongoing revision process of my setups based on the Commitments of Traders reports. I think this independent look at my trading strategy is great and can only help me build a better system. I didn't take into account trade friction earlier because my original group of setups traded fairly infrequently. As my revision process comes up with setups that trade more often, this is definitely an important factor to consider. But I think some of Steve's other conclusions are again flawed. (See my post here about his earlier erroneous study of my strategy.) Here is the response I just sent him about his latest work:

Thanks for revising your erroneous post and reevaluating the data. You raise some good issues about this particular trading setup for the S&P 500, for which I thank you. But I also would like to draw you attention to other conclusions you draw that I believe are flawed and raise questions about some of your evaluation methods.

You are correct to say that trade friction is an important variable to take into account. As I've mentioned on my blog and to you, I'm going through a re-evaluation process of my setups right now to find those that are the most statistically robust. Trade friction would be a good element to include.

However, when you delve into the area of statistical robustness, your conclusions are weaker:

1) You say the performance of the SPX setup with trade friction mostly lags buying and holding the index in the last five years. You evaluate this by studying only the profit. That's probably the weakest measure you can use. It's easy to find incredibly profitable trading setups that aren't very statistically robust by other measures. One more robust measure to use, for example, is the Sharpe score. This tells if the return was achieved at the expense of great volatility. The 2003-07 Sharpe score for this setup is 2.8, while for buying and holding it is 2.0 - a large difference suggesting the setup achieved the same return with less tough-to-stomach ups and downs. Over the entire 1995-2007 period, the Sharpe for the setup was 3.4, while for SPX it was 1.2. That's not to say your point about trade friction eating up those profits isn't a good one. It's just that your evaluation method isn't based on a very robust measure.

2) You also say the COTs dataset for the SP500 has a small sample size that reduces confidence in the setup. Your conclusion doesn't seem to be based on anything very solid. I invite you to read up on how to determine this question by studying Robert Pardo's new book on trading strategy development. One way of checking the adequacy of the sample size is the number of trades. This setup has 150. That's well over the 30 minimum trades Pardo recommends for a reliable setup.

In his book, you will also see a simple method described to evaluate if you've got enough data for your strategy. I've blogged about this here. Using this method, this setup's nine-week moving average period uses only 3 percent of the available degrees of freedom of the dataset. (That is based on the 12 trading rules in the strategy.) That's well below the 10 percent maximum suggested by Pardo.

Pardo outlines other methods of reducing the risk of data-mining, which I've implemented or am in the process of implementing during my revision process. One is out-of-sample testing. This particular setup achieves an out-of-sample efficiency of 1.3 in 10 tests - meaning on average the out-of-sample performance was 30% higher than for the in-sample data for Sharpe, Robust Sharpe, compound annual growth rate, drawdown and regressed annual return.

3) Finally, I think you're incorrect to conclude that your study confirms that "the predictive power of COT report data may have diminished in recent years." I invite you to take another look at your own chart of annualized return by calendar year for buying and holding the SP500 and the COTs Timer Strategy with your trade friction. The best performances came at the end of that five-year period, in 2005 to 2007. As well, you draw this conclusion based on evaluating one possible setup. Again, I would suggest that's not a very robust conclusion. All that said, I thank you again for including me in your research and for raising some good issues to evaluate further.