This is an open invitation to all Collective2 Members -
C2 is looking to revamp the C2 Score algorithm, and I would welcome your help and suggestions. Let’s use this thread to begin presenting and kicking around ideas. Ultimately, the more detailed and actionable the results of this discussion, the better. (Meaning: if we ultimately come up with a specific algorithm that can be programmed by translating the algorithm into computer code, that would be most welcome!)
The current C2 Score is calculated in the following way:
We build a collection of “qualities” we think are important. Examples of qualities that we think are important:
Sharpe Ratio
Annualized Return
Monte-Carlo’s Implied Probability of 10% account Loss
Monte-Carlo’s Implied Probability of 30% account Loss
# of autotraders
# of subscribers
etc.
(These are just examples; there are more.)
Each quality is calculated and multiplied by a coefficient, which is effectively the “weight” we give to the quality.
All weighted raw numbers are added together, to get a raw score value.
All people in the C2 Universe are next ranked in percentile order by raw scores. The C2 Score is their percentile ranking. (Score of 1000 is the hundredth percentile. Score 500 is the 50th percentile).
That is the current algorithm.
I welcome suggestions, ideas, proposals, etc. for a new C2 Score Algorithm. Please let the discussion begin!
From what I see above current C2 score is a wild mixture of:
a) system specific objective performance numbers (Sharpe, Drawdown)
b) system specific subjective numbers (# subscribers, # autotraders, reviews, analyst notes …)
combined to get a vendor specific number.
I’d suggest to add:
c) vendor specific objective numbers (#systems, reliability of signal delivery)
d) vendor specific subjective numbers (review stars, thumbs up in forum posts)
Then I’d generate Two Scores:
1) C2 Vendor Score
2) C2 System Score
to make things more clear and useful.
I’d suggest we start with collecting a number of numbers for each group a) to d)
Some ideas for category d) subjective vendor measures:
* add some buttons to the forum to give people a possibility to mark a posting as "useful" (some have a thumbs up button). Then calculate the usefulness of every user.
* measure the time it takers a vendor to respond to private messages or even let people judge the "value" of a PM response.
* install a Bug-Tracking / Feature Request system for C2. Give extra bonuses for every bug finally removed and every feature finally added.
* when a subscriber stops her subscription she could select a "customer satisfaction measure"
After all this is all about democratic social networking / interaction measures.
I think, the entire C2 Scoring system needs to be changed and be a System Based only… Feedback about the Vendor can be done in the Forum separately.
The C2 System again SHOULD NOT have the subscriber numbers as an input… that skews the result… example… a vendor can sell his system for 0.99 per month and have a huge number of subscribers and skew the C2 System, or for that matter, the current C2 Vendors who have a lot of subscribers have an unfair advantage over newbies.
I think, the length of the system as an input to C2 System Score is very good, so existing C2 Vendors who have a very stable system get the benfit of that.
More thoughts to come…
I’d find a way to heavily penalize Martingale-type systems. The system that’s currently at the top of the Hot Forex list is a disaster waiting to happen IMO.
2/27/11 19:14 STO 1 EUR/USD 1.37172
2/27/11 20:57 STO 1 EUR/USD 1.37329
2/27/11 23:15 STO 1 EUR/USD 1.37546
2/28/11 1:46 STO 3 EUR/USD 1.37675
2/28/11 2:15 STO 6 EUR/USD 1.37843
2/28/11 4:01 STO 11 EUR/USD 1.38017
2/28/11 4:12 STO 3 EUR/USD 1.38195
2/28/11 5:23 STO 6 EUR/USD 1.38413
2/28/11 5:59 BTC 32 EUR/USD 1.38289
I agree with Rene’ that I’d like to see separate ratings for the system and the vendor.
Personally, as a potential subscriber, I’m not too interested in the vendor’s social networking skills, just his trading skills. I guess you could rate the social stuff separately.
Thank you. Exactly my point.
I think C2 has a wonderful System Performance table for each system, something that I have never seen as detailed. I think the C2 System Score should be based on the key metrics from there rather than as Dennis pointed out on “social skills” of the Vendor.
Perhaps, if possible, 2 C2 Scores are in order
1. C2 System Rank
2. C2 Vendor Rank
The latter can be based on a concept like subscriber satisfaction etc much like eBay’s seller feedback, which i am citing as an example only, not that it is perfect in any sense.
Thanx
Certainly make a distinction between a vendor score and a system score. People searching “the grid” on balance would have no idea that C2 score currently refers to vendor. I think broad assumption/expectation would be a rating pertaining to a system. Isn’t a vendor’s overall performance track record readily apparent, anyway, due to the inherent premise of C2?
Keep the subjective/social stuff out of a system score: far too much there for people to complain about. (NOT that that happens on C2, I know… 
)
I think the Vendor score based on some of that stuff is a good idea in theory; however, you can’t discount the reality that some of the best systems might be the lowest maintenance. A few introductory interactions with the system vendor until you’re up and running may be all that’s needed, and the vendor may not otherwise need/want to spend a lot of time in the forums. “Helpful” or “thumbs up” ratings on actual posts, as exist in a lot of forums, are great, but as in other forums, should go across the board (users and vendors alike).
Christina
I agree with this.
Idea on how to penalize these types of systems:
Find the highest percent ever risked on a single trade (position size to stop price). This would be an interesting number to have in general.
If a system doesn’t use stops, give them a 0. Just kidding. Maybe still keep track of their most epic loss and use that (maybe with some multiplier?).
The two stats could not be used side by side, but they both would seem to express the ends the user will go to not take a loss.
Dennis wrote:
I’m not too interested in the vendor’s social networking skills, just his trading skills.
Let me reiterate:
We have lots of objective measures about systems on C2.
From the reviews I see that for many people a responsive and helpful vendor is an importing factor when subscribing to a system.
Of course, there are “silent” subscribers, they simply use the signals and pay their fees, they never talk to the vendor.
But I can say that there are others who have a lot of questions (even if all answers are in the FAQ) and need a lot of attention.
I personally feel it important to be helpful for these people.
I think C2 should encourage good vendor behaviour (to raise overall quality and Customer Satisfaction) and a “Vendor Score” could be such a move.
Rene:
I respectfully disagree. In choosing a system, from say a number of systems that a vendor offers, a subscriber should be able to quickly go thru the SYSTEM C2 score. They can then hone in the detailed system performance that C2 has.
By using the C2 VENDOR system as it stands now, a subscriber (and I am sure most subscribers would relate the current C2 system to be a measure of the SYSTEM and not VENDOR) would just be interested in C2 scores of a high magnitude. That is unfair. A vendor could have a high score and yet have broken systems in their portfolio of offering, where as a new vendor who has no subscribers yet, but has a good system in their offering never will get noticed.
I can understand where you are coming from, based on your score and you would like to keep the system as is, since it benefits you and you have certainly earned it.
But in all fairness to the subscribers, they should be paying for the System’s capability rather than the Vendors ethical behavior. The ethics of a vendor should certainly be a measured metric, but as a SEPARATE score, based on several factors like his portfolio of system performances, customer feedback, etc.
I do like the fact that someone suggested that when a system is killed, it weightedly decreases its value into the VENDOR system.
Thanx
No reason to disagree, I think we are on the same page here.
I also vote for a good, meaningful System Score.
Just as a helpful addition I’d like to see some of the soft skills added to a Vendor Score.
Of course I like my (current) C2 score of 1000, but if its based on the wrong calculations I’d rather have a lower score which is more meaningful.
(you hear the scientist speaking, he’s a bad business-man )
Well said Rene…
Best Regards
I think a system score is useful if it can be wickered to provide an "apples-to-apples" comparison of all C2 systems. The best way I know how to do that is to ratio periodic return to drawdown from an equity curve that has realistic slippage/commission deductions.
On say a monthly basis you divide the dollar-return by the max dollar drawdown to score that month. The system score is the average of all scored months. For new systems, a decreasing fraction of the score can be deducted until the system reaches maturity at say 1 year.
This method will catch the traders who average down because they will have relatively large drawdowns sometime during the month. It also takes care of the problem of comparing a system that takes high risk for relatively high return with one that risks less for less return, because leverage is out of the calculation.
If you want to compare traders, just average their system scores.
Matthew, I think the problem with the current C2 score is that it does not compare like with like (that is risk adjusted returns). Looking at headline return will no doubt favours the high leverage systems. Using the sharpe ratio should address this, however it is only meaningful if you take into account of age of the system (so technically speaking I would apply some sort of t-statistic on the sharpe). Other risk adjusted return metric could make use of drawdown to return (but again take age of system into account)
On the popularity front, it might be a good idea to measure Subscriber fees earned (say over the last 12 months) rather than number of subs. (say 1000 subs x $1/month would be no match a system that takes 10 subs x $200 and so on) . Better yet, if you could measure the duarability/attrition rate of the subscriber base (e.g. average number of years subscribes have been with the system to measure how fickle / long standing subscriber are, that should tell you how good a system really is from the users point of view)
Other measures such as kurtosis and skewness of returns, will tell you so much about the type of strategies a system deploys (trend vs mean reversion, low win rate but high R/R, vs high win rate but big tail risk etc). Your current stats on MAE, MFE are good starting points.
Would be happy to do some sort of collabration if you want more input.
[LINKSYSTEM_44436934]
Also, whatever you decide Mathew, you need to normalize the leverage factor. Perhaps, normalizing each system to start from a standard equity base and compare them at the same leverage. I am not sure how you would do that, but I have seen many systems here which use tremendous leverage using futures with small equity base… obviously, the performance would look great, but these are disasters waiting to happen. A subscriber should be able to compare two systems on a risk-adjusted basis.
Thanx
PS: When do you expect to close this thread and actually work on getting the C2 system implemented?
Hi,
You wrote : "Also, whatever you decide Mathew, you need to normalize the leverage factor"
Your idea sounds logical at first glance but in reality that would be like comparing apples and oranges.
You CANNOT "normalize" the leverage for a stock trading system and a Forex trading system for example and then draw any meaningful conclusion about the profitability of each system, it simply does not work this way.
Why ?
Very simple. Take the SAME trading system and apply it simultaneously to the stock market (leverage 1 to 1) and the Forex market (leverage 100 to 1). Starting with the same capital, the same trading system will ALWAYS make 100 times more money in the Forex market than in the stock market, on average, period!
Now if you "normalize" the leverage, both systems will end up with the same profit potential of course, give or take a few dollars, but nobody truly cares about that.
Remember, people want to make more money the fastest way (while still protecting their capital), that is their first priority. If the same exact system makes $100.000 a year in the Forex (due to 100:1 leverage) but only $1.000 in the stock market , starting with the same capital, the traders would not hesitate a second, they will follow the Forex system, end of story.
You wrote:
"…Now if you "normalize" the leverage, both systems will end up with the same profit potential of course, give or take a few dollars, but nobody truly cares about that.
Remember, people want to make more money the fastest way (while still protecting their capital), that is their first priority. If the same exact system makes $100.000 a year in the Forex (due to 100:1 leverage) but only $1.000 in the stock market , starting with the same capital, the traders would not hesitate a second, they will follow the Forex system, end of story. …"
Does that mean nobody cares about RISK-ADJUSTED RETURN? Then why not start trading just Forex with 200 to 500:1 leverage. Or for that matter, just trade your futures account value/margin requirement contracts of Futures.
You say - "Remember, people want to make more money the fastest way (while still protecting their capital), that is their first priority."…
That is exactly my point for having some way to disclose to the investor what the leverage usage is. Remember, it is always prudent to disclose the risk as much as you want to "focus" on the return. Not all subscribers go for the "high octane" returns. They try and find the answer to "What am I risking here to make such returns? And is that risk palatable to my portfolio?"… I personally would NEVER subscribe to a system that makes too much money too fast without understanding how it makes money… If your approach of "…people want to make more money the fastest way…", then they should be looking for Martingale types of strategy…
All I suggest that we should brainstorm and come up with comparing 2 systems based on a common benchmark of risk and reward. It is not an easy task for C2, but I am sure with so many bright subscribers and system developers we can get to a pretty close solution.
Thanx
The rating should most importantly reflect if you’d have likely made or lost money trading a system. What gets me is my system is one of the few where you would have likely made money and my rating is lower then others where you’d have likely lost money.
That makes no sense to me.
The first 3 months the developer should be shown as “Unrated” too. Developers that on average lose money should receive poor ratings.
MP,
you wrote:
my system is one of the few where you would have likely made money
Howe can you say that?
Did you ever click the “Monte Carlo” button below the chart of your system “Predictor Discretionary ETF”?
You’ll notice that in the worst case you’re just a tiny fraction above zero. In trading we have to expect the worst case.
I wouldn’t call this “likely made money”.
Hi Matthew,
Here is, for a change, a subscribers view about the developer’s ranking system on C2:
Over the years I have traded many systems on C2, not once did I look at the developer’s score or the systems popularity in order to come to a decision. The only things which count for me are the shape of the equity curve, the duration of the system, the statistics (including the new great feature of real brokers executions), the other systems the vendor has developed and the information he provides either on the system’s description page, his web page or through message exchanges. I look at the My Analyst page and the Reviews only for entertainment purposes which does not mean that there are not some good comments occasionally.
I would suggest that you send out questionnaires to all subscribers and ask them what improvements they would like to see since very few participate on the forum here.
It baffles me why vendors are obsessed with the ratings system, they probably want to use it for marketing purposes, but I doubt that any subscriber pays much attention to it. As others have pointed out C2 provides excellent statistics and is willing to improve upon them.
I agree with others who have recommended to eliminate the C2 score.
BTW, I can’t wait to see a Gen3 connection to IB, that should be a quantum leap forward.
Karl
