The opinions expressed in these forums do not represent those of C2, and any discussion of profit/loss is not indicative of future performance or success. There is a substantial risk of loss in trading. You should therefore carefully consider whether such trading is suitable for you in light of your financial condition. You should read, understand, and consider the Risk Disclosure Statement that is provided by your broker before you consider trading. Most people who trade lose money.
I know lots of traders that don’t backtest, that are profitable.
I know lots of traders that do backtest, that are profitable.
I know a lot of old floor traders (typically types that would not do any backtests) who’s edge has stopped working and they have dropped out of trading.
I know lots of mech system traders (types that do backtests) who’s backtested systems have stopped working and they have dropped out of trading.
I will never trade any strategy that does not provide sound backtesting with a reasonable time period (at least 10 years look back) .
However, backtest results are necessary but not sufficient.
An important pitfall of backtesting is curve-fitting and data mining as said already, added to the fact already said previously that some strategies will work some time then not anymore as the edge provided by a strategy will disappears most likely due to changing market conditions or due to a crowding effect.
There are many ways to try to avoid data mining and do robustness checks of a strategy, but there is no way to know if the strategy will continue to perform in the future.
Also, the essence of backtesting itself, is first about data mining: you want to extract a profitable strategy based on the available past data you have.
So, when you do backtesting, you must first acknowledge the fact that you are basically trying to over-optimize some trading idea that you have, on a given data set.
Clearly, there is a really fine line between backtesting and curve-fitting that everybody tends to cross one time or another.
I personally rely heavily on backtesting, as I need some statistical evidence before trading any strategy, so I am subject to the same pitfalls as others regarding backtesting.
Some people say that, even if you conducted thorough robustness tests of your strategy, you should expect half the return and twice the drawdowns your backtest is showing.
This is a conservative approach, but it helps to set your expectations correctly regarding out of sample results.
I also understand that different people will follow different approach or different beliefs than mine regarding trading style or backtesting use, and will totally disagree with me.
Fine, this suits me perfectly, and this is actually the best thing that can happen.
To me. And to everybody.
People should be more worried about following gurus, than to follow their own personal edge even if nobody else believes in it or cares about.
Backtesting lets you have a set of stats that can give you some meaningful information. Its a lot like checking out a steak…I look to see how much grissle (drawdown) I look at the grade of the meat ( average profit per trade and % wins) I also look at how much there is available should i want give some to my friends ( how many trades per year) This all gives me an idea of whether or not i might like the experience of eating the steak.
Personally I have wonderful testing software, and some great systems…none of which I will use here. Instead I will be using something that I will put into software at a later time. If my system performs as I think it will, I will simply use it to show what I can do.
Hi TSH. Back testing most definitely is NOT curve fitting. It is often the only reliable guide to future performance a trader has if they are fully systematic. Choosing system parameters to make a backtest look better IS curve fitting. But you are right - unless they do the back testing themselves, a subscriber has to trust a developer that the backtest is authentic.
We do our back testing on tick data which, as Guy mentioned, is hideously expensive. Since we a short term traders we like to benchmark ourselves against the Newedge Short Term Traders Index - in particular the years post GFC when the index was in drawdown for about 5 years. The program we offer on C2, Sampo, performed strongly over this period and our out of sample results are better than our backtest results. This is largely because Sampo is ANTIMARTINGALE and the core systems it uses to make trading decisions are very simple with almost no optimisable parameters and therefore extremely statistically robust. But without proper back testing we would be lost!
Back-testing is not an indicator that a system may eventually work in the present. The problem with the forex market is it’s acute sensitivity to news releases and that sense even the greatest pundits and lauded analysts have had to run to cover and eat their words over some pretty woeful predictions. In reality, given that Heisenberg was absolutely correct with his observation of the Uncertainty Principle in quantum physics, the future can be anything, literally. Systems are fine as frameworks for analysis fed historic prices and programmed to watch for certain patterns; but the human spirit has never been chained so far as the history of human thought could ever recall. Therein lies the refutation principle and all crows are not black and the errata that can topple an inflexible adherent. Successful traders eventually acquire a 6th sense to move with the wind; to sense market direction whilst observing volume and generally being able to blink first with years of neural network memory to reinforce a decision with lightning speed. Its more than just math; its an issue of Mind vs Machine that were touching upon.They say that novice drivers are far more prone to accidents than experienced drivers because of the lack of reflexivity. A good trader is able to utilize a trading system structure but allow the flexibility to change the trade when years of experience coerces the human mind to have a serious rethink. Yours sincerely, Pieter Bergli.
Backtesting is for the strategy manager. If your backtesting hasn’t shown you something nice then why are you plunking down the listing fee!? That is to say, if it hasn’t worked in the recent past then what makes you think it will start working in the future!? That being said, unless you have discovered some sort of hither-to unknown arbitrage, the future performance of any system is always in doubt. Great backtests mean nothing if they lose money going forward. Forward testing is where you let it all hang-out in public.
