The Black Dog

TOO MUCH Back testing is worth ZERO.

All systems that have been back tested can only show a good performance because the system has been tweaked to show a good performance. But markets behaviours change. Therefore even a good backtested system will not necessarily perform well…in 1 or 2 years !

NO trading system has performed well year after year for 20 years.

SIMPLE !

What people shall do is to look at the behavioural trades of the existing available trades. Then put these trades in perspective with their own capital size and define their own risk as to the number of contracts they are willing to take.

WHEN AND IF the new trades will not perform as per its history behaviour, the new trades should be put aside (not traded but watched) until you feel either the new trades are back to “normal” or that you conclude that the sytem has derailed from its history and is no longer worth trading with your real money.

Until then you can have numerous profitable trades. The last few trades would be negative by definition but so what ? This is the RISK !

If you dont have the ability to TAKE REASONABLE RISKS… SIMPLY DONT TRADE BECAUSE YOU HAVE A RISK AT EVERY TRADE YOU WILL ENTER.

For anyone who wants to see some examples of how backtested, optimized systems change after release of the system, check out the Feb 2006 issue of Futures Magazine “Today’s Top 10 Trading Systems.” This article shows equity curves before and after system release, and is pretty enlightening.



These systems are probably not the norm, but do show that results going forward can be comparable to backtested results.



I agree with Charles though that a good backtested system will not necessarily perform well - remeber, past performance is no guarantee of future results.

I would add that there can be two quite different reasons why backtesting may fail to yield good predictions. One reason is the one mentioned above, that the market conditions change. Another reason can be that the backtesting procedure was statistically flawed.



Unexperienced modellers often make the mistake to overparametrize their model, so that it is very flexible and fits very well to their specific sample (the backtest). This fit is achieved by capitialization on sample-specific characteristics, and the predictions will be much worse in a new sample (the forward test).



One way to avoid this is to use the method of cross-validation. This means that the parameters (e.g. stop loss levels) are estimated from data set A (say historic data from 1998-2000) and then tested on data set B (say historic data from 2001-2005). This means that, although only historic data are used, there is in a way both a ‘backtesting’ (on A) and ‘forward testing’ (on B) component. (I would rather call it an ‘estimation’ and a ‘testing’ phase; the statistical problem begins when the same data are used for both estimation and testing).



I don’t know how often system creators use this method, but I would trust models created by this method much more than models that estimate and test on the same data set.



Jules