Matt,
I have two points to make.
1. This site provides a plethora of statistics and measurements by which reasonably intelligent and aware individuals can judge a system or system developer. Your rating system is merely redundant and at worst, is a tool that can be used by system developers using extremely risky, unsound techniques to enhance their ratings.
2. Secondly, the fact that you use Auto Trading Frequency as part of the rating algorithm seems borne out of self-interest. First of all, as a developer, I can easily set up an account or have a friend, relative or whatnot set up an account and make my system signals ALL auto-traded. Secondly, it’s quite convenient that the higher the rating, the more subscribers (presumably) and thus the higher auto-trading fees for you.
Let’s suppose that I create a system that uses a method for which auto-trading is not that great a benefit to the subscriber, and thus, I may have a stellar system that has numerous subscribers, however few of my subscribers use auto-trading. My rating is going to be pulled down because my subscribers don’t feel the need to use auto-trading.
I absolutely feel that using the auto-trading percentage as part of your algorithm in creating the rating gives a strong appearance of conflict of interest, and the fact that you don’t disclose the algorithm is not appropriate either.
If you’re going to be clean, BE CLEAN.
I must respectfully disagree.
1) Whenever we include AutoTrading participation in statistics, we include all members in the stats, from all AutoTrade service providers. (Perhaps it wasn’t obvious, but C2 and its affiliates do not charge or earn money from independently-provided AutoTrade service providers.)
2) I stand by my conviction that, over a long enough period of time, a system’s validity and quality can be largely (not entirely!) measured by the amount of real money trading it. Everything else is masturbation. Real money, from real people, speaks loudly. People don’t like parting with their money. Thus, it may be reasonable to think that the amount of dollars trading a system means more than any arbitrary opinion of Matthew Klein, or of Greg Bessemer, or of any other single person; and that it means more than any arbitrary mathematical formula. After all, you can design a formula to favor any metric or quality you arbitrarily decide to favor. It seems to me that letting the market speak about what matters to investors is a more fair way to do this.
Do investors always make smart decisions? Of course not. But over a long enough period of time, the markets have a way of working – of weeding out the incompetents and charlatans. So, I stand by my conviction that including the amount of AutoTrading in a system is one valid component of measuring a system’s quality.
But remember… it’s only one component among many.
And the reason I’ve thrown open this discussion regarding how to revamp the C2 Score is that I welcome our members’ opinions and suggestions. A lot of people on this site know far more about both trading and statistics than I do, and so I encourage anyone who has an opinion about rankings to add their voice to this discussion. (Which is taking place on the C2 Software Development Forum, actually.)
…over a long enough period of time…
…But remember… it’s only one component among many…
…including the amount of AutoTrading in a system is one valid component of measuring a system’s quality…
Well, until you have a new trading system without any autotraders
and until that “only one component” will not negate everything else,
like in “Hot lists” now, where systems without autotraders are completely excluded from “Hot” lists which is unfair.
Matt,
OK, so not ALL auto-trading generates revenue for C2 and it’s affiliates, but Gen 3 trading does indeed. So, the fact that all auto-trading doesn’t generate revenue does not refute my point that it’s in your interest to promote (more highly rate) systems that are more often auto-traded.
Again, as a system developer, it’s about as easy to have an “associate” write glowing reviews for my system as it is to have my system signals 100% auto-traded.
People don’t continue to subscribe to systems for long periods of time that they are not trading. You could use number of subscribers as a measurement of the satisfaction and use of a system rather than the auto-trading measurement in which you have a vested interest.
Please understand, I’m not impugning your reputation or credibility, I’m just saying, let’s be honest about the benefit you derive from having systems that create more auto-trading revenue for you, being more highly rated.
I am suggesting that AutoTrade component be dollar-based, not %-of-subscribers-autotrading based. That eliminates issue with gaming the score.
Matthew
I’d like to, also, recommend that vendors AutoTrading their own system be given considerable weight. No one knows the plusses and minuses of a system better than the person selling it. If a vendor isn’t willing to risk his own money, why should I.
Matthew, I sure don’t envy you trying to come up with something that the developers can’t game. I checked out a forex system that got a glowing review yesterday on the dashboard and I laughed so hard I almost spit coffee all over the screen. This guy has a 998 rating and what he’s doing is exiting profitable trades so he can book a winner and then he immediately gets back in in the same direction. His first 13 trades were really 1 trade where he was short the Euro for 4 months before he reversed to long. I admit it was a pretty nice trade but c’mon…
http://www.collective2.com/cgi-perl/c2systems.mpl?systemid=47345051
About autotrading, can C2 tell if TradeBullet is sending orders to a real-money account rather than a sim account at the subscriber’s broker?
I guess my bottom line is this in terms of keeping C2’s credibility and perception of imparitality at it’s highest, the best thing you could do would be get rid of the rating system, or at least make the formula public.
Again, it really serves no use other than allowing unscrupulous developers doing unsound things to boost their score. And any component that contains any potential self interest for C2 is questionable. Like I mentioned before, you’ve provided more stats than anyone could ever need, which I compliment you for.
If you want to be Consumer Reports like, then there must be no doubt in people’s minds about the impartiality and LACK of self interest of the rating system.
Hey Dennis,
I hear you, but for this particular instance, why does such an approach on said system necessarily constitute “gaming the ratings system?” Sure, those gains may in effect be from “the same trade,” but incremental gains are incremental gains, are they not? Why couldn’t such a trading style just be considered a form of risk management?
Christina
Hi Christina,
He’s boosting his percentage of winners at the expense of his subscribers. He’s exiting and reentering in the same minute so I don’t think it can be called risk management. He’s costing his subscribers extra commissions and/or the spread every time he does it.
Hmm… yes… I suppose… It’s true that sometimes he executes a trade at the exact moment he exits one at the same amount, but the trade could go in either direction (although sometimes goes in the same direction). Interesting how he’ll hold a losing position for a long time until he sees the trend change, and then will jump and change direction on it and generally recover the loss. I still think I chalk it up to personal trading style, even at the extra commission to his customers, rather than deliberate manipulation.
But the end result is the same, I guess.
You wrote :"I checked out a forex system that got a glowing review yesterday on the dashboard and I laughed so hard I almost spit coffee all over the screen."
Well this gentleman more than doubled his money in about a year (beating the S&P 500 by a large margin) while still maintaining his drawdown under 25%, so where exactly is the problem here?
Even Warren Buffet (or any trader in Wall Street) would kill for that kind of return, don’t you think?
Hey, no problem here. I find it funny but then I’m easily entertained. 
Hi Matthew,
Here is, for a change, a subscribers view about the developer’s ranking system on C2:
Over the years I have traded many systems on C2, not once did I look at the developer’s score or the systems popularity in order to come to a decision. The only things which count for me are the shape of the equity curve, the duration of the system, the statistics (including the new great feature of real brokers executions), the other systems the vendor has developed and the information he provides either on the system’s description page, his web page or through message exchanges. I look at the My Analyst page and the Reviews only for entertainment purposes which does not mean that there are not some good comments occasionally.
I would suggest that you send out questionnaires to all subscribers and ask them what improvements they would like to see since very few participate on the forum here.
It baffles me why vendors are obsessed with the ratings system, they probably want to use it for marketing purposes, but I doubt that any subscriber pays much attention to it. As others have pointed out C2 provides excellent statistics and is willing to improve upon them.
I agree with others who have recommended to eliminate the C2 score.
BTW, I can’t wait to see a Gen3 connection to IB, that should be a quantum leap forward.
Karl
