What does a good backtest look like?

Trading and Markets
1

What do I look for in a trading system backtest to determine if it is good and worth risking money on?

2

Most important isn’t the results in the backtest but rather how does the system perform out of sample.

Always leave some data for that. For example, if you have 20 years of historical data, construct your system using the data of the first 18 or 19 years (IS = in-sample data) and then, once you are satisfied with the results, test it on the last year or two (OOS = out-of-sample data).

You do that once per system. If, after seeing the OOS performance, you then go back and adjust the system again and again until it performs nicely in the last year too, then you have turned the OOS data into IS data. This means that again, you will have no idea how your system will perform onunknown data.

Factors that will result in more robust systems that can perform well into the future, include:

  • Using as few adjustable parameters as possible (if you have 10 parameters and you just optimize them, it is highly likely you will manage to get beautiful results in sample, that will be worthless out of sample (issue of over-fitting your data, or “curve fitting”).
  • Your target function should consider low drawdowns and stability;of returns, this is more important than high overall yield.
  • See how tolerant your system is to noise in the data or noise in parameters’ values. If small changes to parameters values results in large changes in system performance then your system isn’t robust.
  • Place more emphasis on timing of entries and less on fancy exits. Simple exits will reduce the chance that the in-sample success is a fluke.
  • Your system must not lose money for too long at any given period. At worst, it should stay flat in periods it cannot handle. Alternatively, if it only works in specific periods and loses money in others, you should have a way to stop it when DD gets larger than what you expect in profitable times.
4 Likes
3

@GalBarak has a very good summary of conventional wisdom on backtests. These are best practice notes for system developers, but it is rare that an investor is invited to know how the OOS data was treated (and mistreated), how many adjustable parameters the system has, how tolerant the system is to noise, etc. It’s much less simple to divine future success in a system if you are a potential investor. I’m a developer; I’ll let other investors guide you further with practical advice.

4

Obviously my advice is for developers amd not for followers. The person
asking the question is a developer AFAIK

5

GalBarak,

Thank you for your reply. Your words are relevant for followers, investors and traders of systems developed by others as well as trading system developers.

This knowledge is necessary for success in this business. It may be that a follower does get important information like v1Trader mentioned above in reply; it is most likely the case that a system follower will not have this information available to them.

Developers must know these things if they want to be successful in the long-term.

Trading system followers will benefit greatly if they understand what you and v1Trader mention in this post. It is in the best interest of the trader to understand these things.

I am listening and willing to learn new things. Hopefully others will learn valuable lessons as well.

Thanks,
Andrew

1 Like
6

To me a backtest only determines if a system is worth further study, in other words it can be a starting point. A forward walk through, (on different data) followed by paper trading is much more valuable. Never expect the same results a backtest gives you alone.

7

Thank you for your reply. That is good advice. This discussion is sorely needed here in these forums.

Andrew