Call for C2 Score Algorithm

I think it would be wise to replace Sharpe Ratio component of your C2 Score with either UPI (Ulcer Performance Index) or Sortino Ratio since Sharpe Ratio penalizes for volatility regardless if its upside vs. downside volatility. IMO, UPI or Sortino does a much better job at penalizing for only the downside volatility which in the end, is what we really care about regarding risk of our own capital.



http://en.wikipedia.org/wiki/Ulcer_Index

http://www.investopedia.com/terms/s/sortinoratio.asp



JD

Hi,



You wrote : "Does that mean nobody cares about RISK-ADJUSTED RETURN? Then why not start trading just Forex with 200 to 500:1 leverage. Or for that matter, just trade your futures account value/margin requirement contracts of Futures. "



Of course not, no matter what financial instrument we are trading (stocks, futures, currencies, etc…), we should always use sound money management principles, there is no way around this.



All I am saying is that you CANNOT compare the results of a system that trades stocks (no leverage) with the results of the same exact system that trades currencies (up to 500 to1 leverage), it’s like comparing a small Cesna to the Concorde! Sure, both planes will take you from point A to point B, but the (now defunct) Concorde will get you there much much faster.



You wrote : " I personally would NEVER subscribe to a system that makes too much money too fast without understanding how it makes money… If your approach of “…people want to make more money the fastest way…”, then they should be looking for Martingale types of strategy…



Of course the trader/subscriber must at least understand the logic behind any trading system (usually the system details page reveals a few clues), but that does not mean that a system that makes money “too” quickly (??) is necessarily a martingale.



Again and I will repeat that, if a system has no mathematical/statistical edge to begin with (in other words it is a losing system making random calls and the spread and/or commissions will slowly but surely empty the trading account), NO martingale will ever turn it into a winning system.





"I think it would be wise to replace Sharpe Ratio component of your C2 Score with either UPI (Ulcer Performance Index) or Sortino Ratio since Sharpe Ratio penalizes for volatility regardless if its upside vs. downside volatility."



I disagree. Sharpe is profit/equity-curve-volatility. Upside volatility adds to profit so there is no “penalty” for making money and it doesn’t penalize winning and losing trades equally. OTOH it does penalize one-time, outlier profits, which are unlikely to be repeated, compared to slow, steady profits.



An example, I have a TS system that made a killing in the stock index futures during the flash crash. We’ll probably never see another one of those. The Sharpe went up after that but not as much as the sudden spike in the equity curve might suggest. And that’s how I think it should be.

Further, the problem I have with Sharpe is its reliance on standard deviation:



http://www.tangotools.com/ui/ui.htm

I respect the author of the Ulcer Index but either he doesn’t understand how Sharpe is calculated or he is “misspeaking.”



“The calculated value of SD depends on the time period used. For most investments, the SD of annual return is roughly 7.2 times the SD of weekly return (7.2 is the square root of 52 weeks per year). Since the time period is often unstated, this creates an opportunity for misunderstandings.”



"Unlike SD, the calculated value of UI is essentially the same regardless of the time interval per data point."




SD is annualized in the Sharpe calc, just like the return is annualized. Multiply the monthly SD by sqrt(12), the weekly SD by sqrt(52), etc.



That said, I do have a complaint about the way the stats are calculated at C2. They don’t include commissions and the calculated Sharpe is way too high for most systems. I probably shouldn’t complain because I’m a daytrader and my stuff (should I decide to publish it) would look great if I could ignore commissions. But it would give me an unfair advantage when comparing to a system that holds longer and for which commissions aren’t such a big deal. Commissions should be included but they should be realistic, not the ridiculously high “typical commissions.” Even the MBT commissions for futures are way more than what most of us pay.

I think a more important critique from Martin is below:



"The calculated value of SD is not affected by the sequences in which gains and losses occur. Thus, SD does not recognize the strings of losses that result in significant drawdowns in value. The three hypothetical investments in the chart below have the same annualized return and the same SD, but no rational investor would consider them as having the same risk."



UI solves this and thus UPI which makes use of UI is superior IMO. UPI = (Return - RiskFreeReturn) / UI. Whereas Sharpe = (Return - RiskFreeReturn) / SD.



Anyways, regardless of our debate above, I agree on your point that C2 should include commissions within its stats calculations which does skew the high frequency systems as being better then the others. Ideally C2 would calculate the stats for the different sets of commissions (e.g., Typical, OEC, etc.). Then when user selects his commissions option which affects the charts, the proper calculated stats is chosen as well (using the commissions value from that commissions option he selected).

The premise is faulty and so are the graphs on the UI page. He shows a graph that steps slowly up and says it has the same SD as one that goes way down and then goes back up to end at the same profit. I typed some numbers into Excel and had it calculate the SD (the last number in the list). Judge for yourself.



Edit: the forum formatting doesn’t make it clear but there are two columns of numbers with the SD at the bottom.



0 0

-1 1

-2 1

-3 1

-4 2

-5 2

-6 2

-7 3

-8 3

-7 3

-6 4

-5 4

-4 4

-3 5

-2 5

-1 5

0 6

1 6

2 6

3 7

4 7

5 7

6 8

7 8

8 9

9 9

10 10

--------------

5.28 2.80



The other thing is the order in which the trades happened in one historical run is only one possibility of many. The idea behind Bootstrap and Monte Carlo sims is to shuffle the trades into a different order, recalculate the MaxDD and repeat several thousand times. Sort those drawdowns and you get a better idea of the probability of having a drawdown of X% than a single sample can give you.

According to Peter Martin who I emailed about this issue of SD, his response to me regarding your last comment:



"If I understand him correctly, he’s calculating the SD of the values, not the SD of the returns!"



So perhaps you are not calculating that example of his correctly according to Peter. I’ll leave it at that as I’m not the expert with Peter’s own work here. Maybe he’ll jump in this thread to give more input since I pointed the URL out to him.



Cheers,



JD

We’re getting really OT here but here’s the same thing on the returns instead of the raw equity curve. It’s way simplified but the point is the same… the down-up curve has a worse SD than the one that just steps up.



This is like deja-vu all over again for me. :slight_smile: I was fighting the misconceptions about Sharpe on the old Omega List (Tradestation) back in the 90’s. I have TS code for calculating Sharpe the “right” way. PM me if you want it. My friend, the author, consulted with Dr. Sharpe when writing it. It’s way more complicated than the standard Sharpe calc but it’s true to Dr. Sharpe’s original intent and it covers all the possible ways you can scale a system.



-1 1

-1 0

-1 0

-1 1

-1 0

-1 0

-1 1

-1 0

1 0

1 1

1 0

1 0

1 1

1 0

1 0

1 1

1 0

1 0

1 1

1 0

1 0

1 1

1 0

1 1

1 0

1 1

---------

0.94 0.50

Going a bit further, the reality is that people would like to get a superior return to just Buy and Hold with minimum return volatility. The three simplistic measures that I think would solve this and can become an input to the C2 System Score are



1. % of Buy/Hold since inception.

2. Straightness of the equity curve (closer to a 45 degree)

3. % of Perfect that was extracted by the system. % Perfect being the theoretical max money you can make, knowing the signals well in advance. It is a look-back calc.



I have seen software that do calc these so the math is out there.



HI Dennis,

I’d love to see some “good” calculation of the Sharpe Ratio, because the standard method has some flaws.



In your example you should use the correct returns to compare the SD of dn-up vs. jiggle:



-1 -1

-1 1

-1 -1

1 1

1 -1

1 1



will result in exactly the same SD.



BTW, this one 1, 1, 1, -1, -1, -1 also has the same SD, despite most (naive) traders would describe it as “green, good, happy”

> PM me if you want it



There is no way to PM you on C2 as long as you don’t have an active system…

Funny.



How you guys want to measure systems or even vendors, if you are not even able to agree on one indicator/parameter?



My advice: Start measure your ego and use it as a top level parameter.



Everything will be much easier then.

Hi Rene’,



Oops, my noobness is showing. :slight_smile: I emailed it to your info address. The main difference from the standard method is it uses log returns so you’ll get about the same Sharpe whether you are trading constant size, constant risk or fixed fractional.

I’d like to point out another stat that may be of interest for C2 score called R-Cubed by Curtis Faith. See towards bottom of this page here:



http://www.forexspirit.com/2007/11/07/lies-damn-lies-and-statistics/



I also recommend his book “Way of the Turtle” which has lots of discussion on how to measure robust trading systems.



Regards,



JD

Hi John,



You wrote : "I also recommend his book "Way of the Turtle" which has lots of discussion on how to measure robust trading systems."



No a bad book but remember that the Turtles had to change their trading system because it is much less profitable now. Basically they traded off a 20 day simple Donchian system but over the years profits became smaller and smaller and they had to change the lookup period, from 20 days to 40 days.



For system developers or traders who want to build and/or evaluate robut trading systems I recommend The Evaluation and Optimization of Trading Strategies by Robert Pardo and of course the famous New Trading Systems and Methods by legendary futures expert and author Perry J. Kaufman.



Some great comments here. My wish list would include:



All Risk-Adjusted Performance; there is often too much focus on nominal returns, without normalizing for risk. (Even the "Gainers of the Week" emphasizes nominal performance…)



For risk, can use SD, down-side measures of risk (for instance, semi-deviation captures kurtosis, etc.), as well as drawdown measures.



Perhaps "average drawdown" and/or "worst drawdown" over various time frames (and/or drawdown that is normalized for age of system).



Some measures/monte carlo techniques can be applied to trade data (especially worst risk/trade that C2 already has) – to help weed out/penalize Martingale approaches.



Age of system is important, but should be balanced with meaningful risk-adjusted measures.



I like splitting out a trader score as well as a vendor "customer service" score.



[LINKSYSTEM_51201525]



Mathew:

Any plans or agreement on the C2 Score algo development?

I agree with the following point:

"I think, the entire C2 Scoring system needs to be changed and be a System Based only"



Also you could introduce a "Subscriber Retention Score". This look at the age for your subscribers.

[LINKSYSTEM_46932324]

I think my first question to you Matt would be why is the calculation methodology secret? It is hardly a particularly valuable statistic from the viewpoint of intellectual property. Can you just give the exact formula, inputs and weightings to aid the improvement discussion.



I honestly think that the statistic as it stands is counterproductive as anything that repeatedly gives extremely high scores to systems that have immense (e.g. +75%) drawdowns is harmful to the reputation of C2.