A Cautionary Tale
Picking a good system is tough. The job becomes much easier once you eliminate all the junk systems out there.
Here are some warning signs to watch out for. I’ll use “Tradesignal” and “Forex YX Signal” as examples, since these were 2 recent systems I predicted would crash:
Mid August - Tradesignal was at $290K - I predicted <$100K by 10/31/2010 - system was <$100K on 9/30/10
Mid July - Forex YX Signal was at $57K - I predicted <$50K by 10/31/2010 - system was <$25K on 9/30/10
So, here were the warning signs I saw - and that I see with other systems:
Forum Posts
1. Boasting By Developer
"Tradesignal" would post almost daily his profit and winning percentage, crowing about how great he is. A lot of other developers do the same thing. This just shows insecurity. The best vendors don’t have to say anything - they just trade. The record speaks for itself.
2. Focus on Win Percentage (or some other meaningless stat)
REAL traders know winning percentage is meaningless for long term profit and loss. Newbies think win percentage matters - it does NOT. Focus on expectancy instead. Most other stats are meaningless for new systems. For example, Sharpe Ratio tells you nothing for at least the first 6 months. And no stat is meaningful until at least 30-50 trades are in the books. FYI, I count individual legs as trades - C2 doesn’t.
3. Ignorant Vendor Comments
One vendor once said that you should deduct $300 per trade for commission and slippage. Huh? Maybe in the old days, when brokers and the pit would hose the client. Not today. $50 is probably an upper limit. This comment was a tipoff that this vendor did not trade - he was probably just a system seller. That might be OK - just remember that good traders and good system sellers are likely not the same people.
4. Deletion of Posts or Removing Forum
"Tradesignal" actually said that drawdowns were meaningless - only profits mattered. Then, he deleted it. I saved it in My Analyst write up.
Another vendor posted “I am changing my strategy” then later took his forum private. BEWARE of vendors who try to hide things from you.
5. Discussion of Voodoo Science
Good trading doesn’t need “sacred” or “hidden” things, or recently discovered long lost papers, or advanced mathematics or anything like that. It is just sales talk.
6. Insults From Vendor
Today some vendor said I probably received my degree from “Banana Republic.” That was after he misspelled “Stanford.” Why on earth would a reputable, dependable vendor ever bother insulting potential clients? Answer: Mature vendors don’t. And immature vendors usually fail.
Review and My Analyst Comments
1. Shill reviews
If a review of My Analyst comment sounds like it was written by the developer, it is probably was. Or maybe by his friend. Anyone who writes a system is the “best” is probably a shill. Good vendors and good systems do not need shills.
Developer History
1. New Developers
Watch out for any new vendor without a track record. Especially one who appears on the scene with a whole slew of systems.
2. Other Systems
Beware of any vendor who has blown out a system before. Maybe this one will be different, but maybe not. “Fortuna” is the poster child for this.
2. Multiple Accounts
Some vendors have multiple accounts. There are legit reasons for this, but one BAD reason is to hide the failed systems.
System Trade Record
1. Adding to Losers
As a rule, avoid systems that do this. Almost all vendors who do this do it incorrectly.
2. Odd position sizing
Look for consistency. You do not want to see 1 lot trade followed by 200 lot trade. If it doesn’t make sense, why risk your money on it.
3. Performance with costs
Make sure you add in costs. One system out there charges $295 per month, and makes about 100 trades per month. This is a cost of about $1000 per month. If you have a $20K acct, that is 5% PER MONTH. Almost no system can consistently jump that hurdle.
Well said Myth
Well put indeed - this post is worth at least $10,000 in sage advice 
What a wonderful advice!
Should be a must-read for all new C2 customers (and probably the older ones too)
Just one minor comment. Under the section "Voodoo Science" you also mention "advanced mathematics".
I strongly think there is a clear cut between "voodoo" and "real" science and "advanced mathematics" along with "advanced statistics" clearly belongs to the realm of more useful things in trading.
Of course, science can be misused in "sales talk", this is another story.
I believe that these days a good measure of "advanced statistics" (if done well) is an important part of a working trading system.
I agree - I wasn’t clear, but I was directing this at sales talk that throws science in it to impress and confuse people. It is is impossible to understand many people will think “it must be really good!” Usually it is the opposite.
The “advanced mathematics” comment was directed at a system where the developer said he used “advanced matahmatics” (spelled wrong) that no one else in the world was using. That is obviously sales pitch talk.
His system crashed.
Let me add to your excellent post:
System Trade Record
4. Length of Track Record
Consider the length of the Track Record thoroughly. It can’t be overemphasized how important this length is. It is not only easy but unavoidable to see excellent systems with a short track record which is the result of pure luck (or more precise: an expexcted statistical anomaly).
With thousands of trading systems on C2, there are always a handful which have a few good days or weeks, which doesn’t mean a thing.
The longer a track record, the smaller the probability that results come from pure luck.
Today I’d start to look at a system’s performance stats if its older than three months and has more than 100 trades, ignoring all younger systems.
Some vendors provide backtest results to generate more trust in their (at this time too young) systems. But: You need a lot of trust in the vendor and his skills before you should believe a backtest result. It is very easy to generate a bogus backtest (even unconsciously) so watch out.
Some hints for judging a backtest
1. Trading costs and Slippage
Does the author state exactly what trading costs are included in his simulation? What does she say about slippage?
If you don’t find any information or just some foggy “trading costs and slippage included” be careful!
2. Precission
What data was used in the simulation? Where does the data come from?
Is the simuation based on End-of-Day data or intradeay (e.g. 1-Minute) data.
For some (slow) system EOD data may be enough. For faster systems it is critcal to simulate on intraday data. If the system uses limit orders and several open position at a time the backtest is almost surely bad if run on EOD data.
3. Scientific / statistical rigor
Does the author say anything about In-Sample and Out-of-Sample periods?
Something about Training periods and Test periods?
If none of these words appear in his descriptions be cautious!
After years of trading system development I find the treatment of Training/Test periods the single most important issue.
And one more:
Type of TradingSystem: systematic or emotional
Of course this is a nearly religious debate, but being a scientist I clearly prefer systematic trading which does not depend on personal judgement and daily moods.
How tell a systematic system form a manual system
Your first stop is the system description. The vendor should state clearly how he arrives at his trading signals.
A second hint comes form the system’s signals and their signal comments. You can see these comments in the “Recently Closed Trades” section if you open the details (click on the arrow at the right side of the section heading)
A systematic, automated system will have computer generated signal comments like these:
BTO Order: Limit Entry, PosID=2010-09-23
STC Order: PT1, matches Limit Entry, PosID=2010-09-23
As I said before, I’d always prefer a systematic, automated system.
Excellent additions, Rene, to a very good post. Maybe Matthew will add these to his book in a section called "What Community Members Recommend."
2 comments:
1) If you use Tradestation to deliver signals (as I do), from a systematic, automated system, I do not think you can add "signal comments."
2) As far as discretionary vs. systematic, I think both can be profitable. Since I have a math/science background, I like testable systematic systems. BUT, I am having some very good success now trading a discretionary system. One key, though, is to have rigid guidelines and risk control.
The key to me: both mechanical and discretionary are ok, as long as vendors disclose what they are doing, and STICK TO THE RULES. Many vendors abandon their system at first sign of trouble. And that is bad, no matter how you trade!
Kevin
So, do you have any system performance predictions that you think would perform well in the future?
Hi Rene,
Thanks for your excellent post in continueation of the excellent post by Myth Buster.
Even though I am not presently trading one of your systems I am aware that you have developped some great systems and people in the know can appreciate that. I also agree with a statement here made by Keith Fitschen some time ago that he thinks the best stock systems are the "dip buying" systems (and I assume he meant based on statistical analysis).
I hope I am not too presumptuous in asking you the following questions: There are now some other "dip buying" systems which incorporate inverse ETFs, in other words, they are also doing some "top buying" in certain market conditions. Do you think they have a valid statisitical underpinning?
Thanks,
Karl
To clarify:
Instead of “top buying” I should have said “top market selling”.
Also, I don’t mean to ask you to comment on any specific systems here but I wonder if you think that “top market selling” systems can successfully developed based on statistical analysis.
Some good points there, I would only say I think you’re wrong to count individual legs as trades and that C2 does it correctly, the people that do this scaling in and out without ever closing the position are playing some very clever games with their system statistics as a result. Surprised you haven’t seen through that.
Kevin Davey is Myth Buster. Look at his track record.
Hi Ian -
Sorry to disappoint you. I am pretty sure I know who Myth is, however.
Kevin
Hello Karl,
you asked: "There are now some other “dip buying” systems which incorporate inverse ETFs, in other words, they are also doing some “top buying” in certain market conditions. Do you think they have a valid statisitical underpinning?"
All I can say: I tried several times to find some worthwhile “Top Seller” system. But while the “Dip Buyer” is easy to find (and somewhat harder to improve) the inverse mechanics doesn’t seem to work at all.
I am sure, whoever researches this thoroughly (i.e. with valid statisitical underpinning) will find similar results.
My explanation goes as follows: During a dip people are driven by panic. This causes predictable behaviour which may be exploited to build profitable systems.
During a “Top” or Rally people are drifen by greed. While this is also a strong emotion it is much weaker than panic - the mass psychological effects are more hidden in the unavoidable noise. It is much harder to exploit greed for a working trading system.
Hello Ms Naivete,
you asked: "So, do you have any system performance predictions that you think would perform well in the future?"
I can’t see into the future. But I know the ingredients for a system’s track record to show some promise. To summarize the most importtant points:
* the Track record should be long (several months, > 100 Trades
* The systems stats should be “good,” i.e. risk adjusted profit should be as high as possible, look at sharpe ratios and annualized profits.
Hint: examine “Topaz” and “Pearl” 
[LINKSYSTEM_37299452]
Hello Rene,
I appreciate that you share your insights here on the forum.
Thanks,
Karl
Ren,
As you say the vendor should clearly state how he arrives at his signals.
I agree this way we can take his system and improve it and sell it on C2.
When will you give us the exact details on yours or should we all join you under the tent for an Oktoberfest beer?
Nick
Nick,
you said: When will you give us the exact details on yours
Here, and its an illustration of what I meant:
The signals for “Topaz NQ 100M” and “Pearl ][” are the result of rigorous statistical analysis of several years of stock-price-histories and related events. We use modern statistical methods (state-of-the-art Machine-Learning Algorithms, Bootstrap, etc) to identify statistical “patterns”.
(this involves a huge database with price histories and hundreds of indicator histories)
Contrary to what most expect the more important part is not finding such a “pattern” but testing its statistical significance. We invested quite some research to develop solid statistical test methodologies, the key of which is a clever usage of training and test periods. The primary goal of the whole undertaking is not “high profits” but persistance of predictability, i.e. for us it is important that a system does not only work for the years it was developped for but also the years to come.
You may judge our methods when looking at Topaz’ Track record.
(The inner plumbing of Topez is unmodified since it is published here at C2)
[LINKSYSTEM_25860974]
"… or should we all join you under the tent for an Oktoberfest beer?
This is an excellent idea. Instead of the crowded, tourist loadad Oktoberfest we could meet in a nearby Biergarten and discuss these things further over a decent “MaÔ (translates to a one-liter glass of beer)
Hi Rene,
Meeting over a Maà sounds good I am currently living in Bonn (no Maà served here though - you must be more Bavaria way) but moving back to London soon.
Dean.
