Many have commented about a systems that do well for a long time vs doing well for a short time. There are various causes. Many times the cause is in the systems development process.
There are several curses to avoid when developing systems. One of which is local minama. This is when one develops a system based upon a small set of correlations. Most novice systems developers do poorly by falling into the singularity of signal trap. This is when a signal works perfectly until it does not - and there is nothing to do by wait for it to work again.
Some use global minima. This gives systems the flexibility needed to survive. This gives the system the opportunity to survive in the real world - Provided one avoids the other curses.
Wrong, totally wrong. System success has nothing to do with local vs. global optima.
If you just optimize your system parameters and stick to some “optimum” you’ll almost surely produce a system which works very nice in the past and fails for every future.
The key is proper In-Sample vs. Out-of-Sample testing.
Let any “optimized” system run on a set of “unseen” data. Results are usually frustrating: The system produces no profits.
Only system which show good performance on out-of sample data are worth further consideration.
And there is one important obstacle: Whenever you run a system on “fresh” data, this data is no longer “out-of-sample” - hard to deal with…
Wrong totally wrong,
When developing a system one of the things you do is avoid a small set of correlations…local minima. You are much better off with global minima such as was used by Gann… http://bit.ly/gqd43I
Out of sample data testing comes several steps later.
Testing of any sort is a waste of time unless the system is properly constructed and filtered for market curses. Another curse is using hill top indicators verses flat top. This is when the indicators that are sought after are those that are virtually perfect in the development stage.
Once again out of sample data testing comes several steps later.
Alden
Guys, Out of sample is for whimps! Test what ever you have across the spectrum in futures-stocks-forex etc.
Best
"The key is proper In-Sample vs. Out-of-Sample testing. "
It took me over 4 years of model building to eventually realise the above statement!
I don’t follow what you mean. Please elaborate about the so called “singularity of signal trap”.
Notch, This is when there is a signal that is used for the system and surprise …the formula no longer works. Ask anyone who has used MACD and they will tell you that this or that crossover worked well for a while and then all of a sudden it stopped working. Usually the signal does not work any more because it is based upon some based upon a false theory. What is a false theory…"Look ma i found a moving average that works" is perhaps an exaggeration-but you get the point.
The challenge is proper curve fitting. In any system design there is curve fitting to some degree. How it is done is key to the system surviving. All of this occurs prior to any kind of testing.
The optimal test is the broad spectrum test. It is much better than walk forward.
Alden
Alden,
Your computers must be a humming night and day. To do a broad spectrum test on the 3,000-10,000 most traded instruments takes more than a weekend. How many years back do you go?
Best
When I have the data ten years. What takes a long time are the tests that involve various time frames. The computer has to look at one day at a time and various time frames at the same time for one system on one run. When a system has multiple sub systems it gets to do that for them all at the same time. When this is done in the forex the computer needs to be mindful of several crosses at once. Ten years ago this would have taken a Cray.
Your account has not been around long but it is doing well. To what do you owe your success?
Alden
"Your account has not been around long but it is doing well. To what do you owe your success?"
The system has made seven trades!!!
And it is only three weeks old! I agree, such a question at this point in time is meaningless. There really is no way to tell whether the success so far is simply by chance or if it’ll still be doing well one or more years from now.
Dean,
I agree that having a multi year track record is ideal. Sometimes you need to work with what you have.
Let us take a look at your three systems. They meet the requirements you stated.
Only examine the first seven trades. And of course press the personalized button. After looking at the first seven trades I can tell a lot about a system. I can see with the first seven trades which of the three is likely to do well. Note that your turbo system is a net money maker as you could have predicted after seven trades.
As you examine your three systems, do you see how after only seven trades you could have guessed which system will be the strongest? Which one is the strongest now, after a year?
Of course for statistical significance more than seven trades are needed, but then you can subscribe to my system for free. This will give you an excellent feeling for how my system works.
Best
Notch,
I was able to take a look at the first seven trades of one of your two systems. it is the one that is doing well. Your first seven trades tell me a lot.
If you look at the first seven trades of the system you let die, were the results of the first seven trades encouraging or discouraging?
Best
O m G,
In your analysis, you are picking two reputable vendors, which we only know after seeing their long track record. I can find 50 (more if you like) systems that had very good systems for the first 7 trades but are now below water or worse.
Would you really invest your retirement account with a hedge fund manager who has a 7 trade track record and nothing else? If so, I have a bridge to sell you, just send money.
CEO,
Thank you for stepping up to the plate. I am saying that if a system is free and it does well initially then it is worth following closely. And that way is to subscribe.
Looking at your two systems I can see that one you have been nursing for the last two years. It tells a lot. Have a look at the first seven trades. If you press the personalize button you will see that it represents nicely what has occurred for two years.
Now look at your OS system. Have a look at the first seven trades. Do you see the obvious difference?
There is an obvious difference in the trading of the two systems.
Best.
O m G
Congradulations are in order if you can evaluate a system based on 7 trades.
I need to see 100 trades or 1 year in the market (# Trades * Average Duration) before I will start to trust the system or vendor with my money. There are many other considerations I use.
Lets agree to disagree.
