I think that people who don’t agree with backtesting are the ones that don’t do it or can’t do it. Of course, you can construct the perfect backtest that make the maximum amount of money with past data, this is called overfitting. But that is not the point of the backtest. If you have a trading idea and you test it on past data and it does not perform there, what is the point of even continuing with it? I’ve seen a lot of people proposing strategies and they seemed logical but the backtest showed something different. You would need a very good explanation (such as change in regulations) to go ahead with this regardless.
On the other hand, a well-performing backtest does not mean the strategy is going to work in the future. You may have just been lucky to pick the right parameters. How can you be sure your backtest has statistical relevance? Perform hundreds or thousands and check for statistical relevance. There are a ton of statistical tests you can do that show you if your backtests are valid. Just doing one is, in my opinion, a waste of time.
As it was previously pointed out, a good backtest can show you how much drawdown you can reasonably expect from your system and it will give you more confidence to carry on when things look a bit scary (or shut it down for the same reason).
Another thing that’s interesting is the fact that I’ve backtested a lot of strategies that have worked for the last 30 years but they have stopped working in the last 2 years. Without the ability to do such sanity checks you would blindly believe the ‘gurus’ that tell you that they’ve done this for decades and it works - and loose a lot of money.
Backtesting is not a sure-fire way to make profit but it reduces your risk of loosing by a very significant amount.
Yet another issue is the fact that people mistaken backtesting with testing certain (technical or fundamental) trading signals. However, even comparatively lousy signals with a good position management can produce good returns while the best signals with a bad position management does very little.
One particular thing that backtesting lots of systems has helped me with is to look at strategies here at collective2 and being able to tell straight away why they perform the way they do or even determining from the performance what kind of strategy it is.
Anyway, in my humble opinion, I would be cautious if people proclaim backtesting is useless and they do well without it. Perhaps they don’t have the skills, they have been lucky for a while, they are just ignorant or maybe total geniuses. But if the latter was true, why would they be here?
So basically what you are saying here is that this site is just for ignorant traders !
I wonder why the due diligence departments of large finance firms screening for potential investments in money managers say that it is a ‘red flag’ if any of the managers submit information pertaining to a backtest of their particular strategy?
The answer is simple. Money managers should have an established track record, not hypothetical results from messaging datapoints. In fact, I find it hard to believe that any institutional investor would waste time considering a money manager without an audited track record.
Brett,
you developed your trading approach during last 5 years and you never ever apply your trading approach (or parts of this approach) to the market data to see whats going on? In other words, you spent 5 years of your life and now run the system (on C2 in particular) knowing nothing about its possibilities? I can’t believe in it. ))
Hope you know that all your trades can be downloaded from c2 and reverse engineered?
Not sure I understand this; my decision to make a trade is ALWAYS based on the CURRENT market data, and what I foresee to be the FUTURE market data. More importantly is the current market environment and general psychology of the markets, sector, commodity, and the ‘average joes’, black box algo’s, and hedge fund managers who are the ones deciding how to put their money to work and influence price. I rely on my brain to decide when to fire off a trade at the end of the day, regardless of what ‘indicator’ or ‘trade signal’ or ‘system’ brought the trade to my attention. Any trade you see entered into my system, ‘Texas Doubledown (Collective 2 - The Alternative Alternative Investment – Collective2)’ is ultimately triggered by myself. I would never subject my clients to the risk of a ‘black box’ pure algo system going haywire and firing off erroneous trades into their accounts. I hold myself to the highest standards regarding my susbcribers capital, and will defend it ferociously. Allowing any system (regardless of the systems backtest/ track record/ etc.) to be the final ‘check’ before placing a trade into subscribers auto-trading accounts is downright negligent in my opinion.
I have personally seen smug traders sitting back at their computer with their legs kicked up watching their purely mechanical ‘algo trading systems’ print money for them on a daily basis for many months, just to watch it all go up in flames in a 2 day period because of a change in market conditions, or an extremely powerful news event that the system couldn’t account for. This could be viewed as a positive for the traders ego however, as he can always blame this loss ‘on the system’ and not himself…
The possibilities are endless and completely unknown. I do not believe that the market we will see in the next 1-2 years will be at all the same as any period in the past 20 years. I know certain lessons and strategies that I have had success with over varying market periods, and these are certainly in my tool-kit to use if I see those environments re-appear. I have used countless trading styles, strategies, algos, fibonaccis, bollingers, short-interest, news events, etc. to make trades over my trading career.
All of those techniques are filed away into a holistic view of my trading style, ultimately decided upon by myself, with input from my trading associates as well at times. I focus on being forward looking and evolving alongside the market, and technology for that matter. The past is certainly important, and to ignore it is dangerous, but I think many miss the lessons the past are teaching us.
The past shows how HUMANS reacted to the market environment of the time, how an extremely emotions based creature reacted to news, stimulus, and fundamental & technical factors over a period of time. This indeed is where the true value lies in backtesting. But without looking at these past fluctuations and patterns through the lens of Human emotion, I am not sure there is a lot of value.
For my particular system, it is the combination of only the most successful and consistent ‘pieces’ of the 100+ ‘systems’ or styles I have studied, and or used, in the previous 10 years. So I am not sure I can create a back test that would truly represent the totality of what I am doing with my current system.
I see WHY someone would want to use a backtest to make themselves more comfortable paying and autotrading for a C2 system. I just think that this ‘comfort’ is likely misguided and will just lead to unrealistic expecations, and/or possible disaster.
Long Term Capital Management I am sure had fantastic backtesting stats before their system went to 0 overnight…
Mr. Blinkov, will you do some reverse engineering of my system and let me know what you think it is I am doing? I am just curious what you could come up with as far as diagnosing my trading style, I know its not much data so maybe impossible… But I would be very curious to know!
Best of luck trading to all.
I am not going to do reverse engineering of your system, I don’t have time, knowledge and desire. You’ve noted it as a weakness of back testing earlier, and I’ve just noted that C2 systems can be easily reverse engineered, all data is open.
I would like to thank you for your last post. It shows your opinion in more expanded, and more polite (from the view of potential subscriber) way, comparing to your initial post. 
We just understand “back testing” term in different ways.
Anyway, wish you a good luck with your approach and profits for your system. Maybe I’ll be your subscriber, when you’ll have 1 year track.
ps Looking to LTCM equity curves, I would say that their money management was excessively optimistic.
Exactly the opposite. LTCM’s primary strategy was relative value fixed-income arbitrage. This type of strategy is predicated on trader’s unimpeded ability to transact at all times. For U.S. Treasuries, this is generally a reasonable assumption (although I’d still be curious to see how this strategy would have performed in 1987), but LTCM, under pressure from investors to reinvest, rather than distribute gains (which it nevertheless did anyway, despite investors screaming, but not nearly as much as it wanted to), ran the same strategies in significantly less liquid markets, from Danish mortgage-backed securities to Russian government bonds. The latter market, in particular, had been in existence for a total of six years when Russian government defaulted in August 1998, which was the first fallen domino in the series that led to LTCM’s implosion. Not a whole lot to backtest, methinks…
Hi TSH, What system do you manage? I would like to check it out!
Brett,
Let me first take a step back and explain my understanding of backtesting (my apologies if I am verbosely telling you things you already know). My background is in equities, so in my book, a good backtest goes like this:
- We start with a “universe”, that is, a pool of securities from which we will be drawing our picks. Usually, although not always, the universe is also the benchmark against which we would later judge the performance of the strategy. Examples are S&P 500, Russell 2000, or STOXX Europe 600. We also select a test period, which should cover a variety of market conditions, ideally including disaster periods.
- We define rules according to which we will take our picks and dispose of them. Obviously, we can’t disclose them to prospective clients exactly, but we do want to give an indication of what we’re trying to achieve. For example, we can say we’re looking for deep-value stocks which we expect to hold for three years, or stocks with an attractive combination of earnings surprises and price momentum to be held for three weeks on average, or stocks experiencing unusually large price movements against unusually low volumes to be held for three days on average, or what have you.
- We run the actual backtest against a database that includes now-defunct securities. If no such database is available (i.e., if the data is subject to survivorship bias), we make a prominent note to that extent to ourselves and prospects.
- We report gains and losses per position and per period, the latter accompanied by benchmark returns for the same periods.
- We take an educated guess at what our “implementation shortfall” (i.e., the difference between actual and simulated results) might be. Typically, a good guess is based on portfolio turnover and the nature of the universe; monthly turnover of 50% in a portfolio of large-cap stocks would drag you down considerably less than monthly turnover of 500% in a portfolio of micro-cap stocks.
And then we use our favorite analytical framework (or seven) to figure out whether this strategy is any good and let our prospects do the same.
Now, with that in mind, let’s go over some of your grievances.
I could create a backtest ‘analysis’ based on what I think you want to see…
Precisely. You may think you know what I want to see, and maybe you’d even be right some of the time. But the next guy may well want something different. For example, I may want consistent alpha, but the next guy wants straddle-shaped returns or at least a beta that’s significantly lower in falling markets compared to rising markets…
Would you like overall gains? Or risk adjusted gains?
I would like a degree of disclosure that would allow me to (1) spot inconsistencies between your description of the strategy and what’s actually been in your portfolio (e.g., you say you trade large-caps, but your portfolio is liberally peppered with small-cap names), and (2) take a reasonable guess at your implementation shortfall (e.g., you claim an annualized 10% alpha, but your portfolio turnover/universe combination leads me to believe implementation shortfall would be greater than that). In many cases, based on your description, I can even throw together a simplified (perhaps, a better word is bastardized) version of your system and see whether my peaks and troughs would roughly coincide with yours. The point is not to replicate your performance precisely, but to see whether your performance is actually driven by the factors you say it is driven by. Yet another possibility is that I’ve read a journal paper that proposed and tested something similar, so I can compare your work with that of the academics.
backtesting is backwards looking and IRELLEVANT. All that matters is what you believe you are RECEIVING IN THE FUTURE in exchange for your hard earned money!
What also matters is that you get to see how your prospective investment manager’s mind works, which does have a bearing on the future, at least from the standpoint of intellectual and emotional comfort.
We can all manifest a ‘perfect system’ if we have all the concrete data in the world at our disposal.
Maybe, but it’s much harder if you commit to disclosure of investing principles and position-level transparency.
Thank you, NC.
That was the answer I was looking for. Lots to think about.
Deleted post …
Hi it is called Daxer its new i am working on something good here , not suitable for everyone .
Will add another systems soon …
I’ll follow it in my watchlist. Understood that some approaches take time to shine. Still, it does seem like your trades allow a large equity swing before exiting…
I am working on something , my risk will be 1 - 1.5% per trade fixed .
It may feel meaningless and frustrating, and is both. However, you have to understand not everyone gets the system and fundamentals of trading. You may need to empathize or hire a personal assistant.
Looking back does not guarantee future results…
NOTHING guarantees future results, including your “getting the system and fundamentals of trading”, whatever that means. Markets are inherently noisy. Properly conducted (including an estimate of implementation shortfall) long-term backtests, meanwhile, help you evaluate your (or someone else’s) ability to separate signal from noise. They also help you establish the range of expectations for your system, including the extent and frequency of extreme drawdowns.
When a mechanical system developer declines to provide back test results I have to assume that he had big drawdowns in his historical back testing.
I think the backtesting its used more as test for Owner of the system… How it works?
- its mechanical or discretionary?
- If the DD its very low, system too optimized.
- If DD is too high, then system doesnt work fine.
- You can see if backtesting its similar to c2 stats, if its very different maybe system doesnt work fine.
- If only have backtest of 2-3 years, he doesnt develop very well the strategy.
etc…
You have a value information to know if the trader have his job or not.
It’s an impossible queston to answer
