Matthew - I have raised this issue previously. I still don’t understand how the ‘Probabilities of future account loss’ numbers are calculated.
Could you please enlighten me on how a system such as ‘Super Star Trading System’ that (according to C2 monthly return figures and performance graph) is currently, as at 11/August/11, in a drawdown of greater than 55% have a C2 Chance of 10% future account loss of NIL?
People look to your figures for guidance in investment decisions, and these are not helping them. If these figures are being correctly calculated per the C2 algorithms can I respectfully request you pull these numbers from the site as they are downright misleading to potential investors.
Regards
[LINKSYSTEM_56316039]
Hi, Michael:
1 Monte Carlo simulations are run on a periodic basis. Thus, probably what is going on here is that our servers haven’t gotten around to re-running the simulation for this system. Note that each Monte Carlo simulation burns a lot of CPU time, and CPU resources, which are not infinite, and so we schedule Monte Carlo runs in sequence, one system at a time, as frequently as we can, which turns out mean that each one is run approximately once per week.
2 I think the numbers are helpful and are meant to uncover things that aren’t obvious. But they don’t always do so, nor do they predict the future. We say this very clearly on the explanatory text.
3 The numbers are meant to supplement already-available information. The already-available info obviously includes the current results to date. So, taken together, the current chart for this system (which is obviously dreadful) plus the Monte Carlo (which is not yet dreadful, but probably will be when the simulation gets re-run shortly) tell you a lot.
4 I think it might be a “cut off your nose to spite your face” kind of demand to insist the Monte Carlo sims be removed from the site because there’s a lag time before recalculating them on a system-by-system basis. Why would it help people if we removed an entire set of available data just because in some cases the information is delayed and thus looks silly?
All this being said, I think it’s a good suggestion to perhaps improve the scheduling algorithm for Monte Carlo simulation runs, however. Perhaps we can do something like reserve a certain segment of available CPU time for those systems that have experienced large changes in equity recently. Thus, in the case of this particular system, the site might recognize: “Hey, this system had a steep fall recently; let me re-run the Monte Carlo simulation today rather than usual time it would be re-run, 3 days from now.” Or something like that.
Or show the date statistics were calculated.
Matthew
I didn’t consider the scheduling/timing factor in these calculations. I would not demand anything from you, this is your site, not mine!
Thank you for your response.
[LINKSYSTEM_56316039]
But I agree your comments are good, and so is Frank’s. I’ll try to make things better.
To me, the label probability of future account loss calculation implies a statistical analysis. A monte carlo simulation is based entirely on past data and so for new strategies it can be very misleading. In my experience, strategies with the potential for large gains also have the potential for large losses, even if they have not yet occurred. Monte Carlo tells you nothing about strategies such as the Super Star that were new and had only large gains. On the other hand, if you were to calculate the annualized rate of return, and standard deviation, you could easily calculate probability of 10%, 20%, 30%, loss. It might be more meaninful and accurate. Another way to improve accuracy would be to wait until you have 6 months to a year of data before making any calculations.
If it is recognized that the data is "silly" in some cases, how are users supposed to recognize the times that it is not, and how many might be mislead when it is? Just my 2 cents. I pay no attention to the probability of future loss because I recognized the data was not an accurate estimate in a number of cases.
I second that.
Monte Carlo is among the best things to do but if the input data is less than six month you get GIGO (garbage in/garbage out)
I’d display a clear “No Meaningful Data Available” in this table as long as the system has less than one year of history and less than 100 trades.
perhaps at least 6 months data may benefit newer systems.
We are liking the monte carlo sim results on our system. it appears we cannot get any worse and only better.
Hello. I am a newbie here, currently in process of publishing a c2 system. Also I have the perspective of a new client looking for systems to subscribe to. So here is this newbie’s perspective on the “loss probability” feature:
1. I really like the idea. Both for developers and clients, this is an extremely desirable feature to have–if it can be made credible.
2. In addition to the often-outdated CPU-intensive calculation–why not also make a simple always-in-date calculation based on displayed results–then show the worse or the average of the two?
3. Absolutely, this is meaningless when there is little data. But I believe there is a solution. I suggest judicious use of “average for category” loss probabilities.
Less than 3 months old:
"This is a very new system. Insufficient data to determine loss probabilities. However the eventual loss probabilities for all new systems of this category have been…"
3-6 months old:
"This system is only 3-6 months old. Insufficient data to determine loss probabilities. However we have taken the average loss probabilities of all systems in this category, beginning after 3 months, and balanced these figures against the results of the first 3 months of this new system. We give 2/3 importance to the “category average” and only 1/3 importance to the initial results of this new system. Thus favoring the “category average” result by a 2:1 ratio. Here are the results…"
6-9 months old: Favor the category average 3:2.
9-12 months old: Equal ratio 1:1.
12-18 months: Favor the system performance 3:2.
Etc.
Of course, this is just an example of the general idea. Other ratios may work better. Nonetheless let me point out that this “average-weighted” calculation and its description:
a) Possibly makes it more clear that c2 is not liable for “predicting performance” of any particular system
b) Possibly provides more meaningful figures, as well as a better understanding of what those figures mean.
c) Possibly limits the need to run CPU-intensive probability calculations more than once every 3 months.
I don’t think anyone wants their system “grouped” with others. As we all know, most systems fail. So stating “the eventual loss probabilities for all new systems of this category have been…” wouldn’t be a good idea. At least I don’t think so.
Since updates are only done every couple of weeks or so, I think they should simply put a date, as someone else had mentioned previously, as to when the calculation was made. Each system should have their own metrics, period.
agreed
Point well taken that "grouping" might not work. However if most new systems fail, then either this should be made abundantly clear in any claim of "probability" for any new system, or there should be no implication of any "probability" that might be positive. Otherwise it is deceptive.
