You may notice a new feature on C2: Monte Carlo Simulations for each system.
The addition of the Monte Carlo Simulation is one facet of C2’s overarching effort to help subscribers learn more about the real risks of a trading system before they start trading it.
What is a Monte Carlo Simulation?
The name comes from the gambling capital of Monaco. A Monte Carlo Simulation is a common analytical technique in which you run thousands of random simulations and study the distribution of results. In the case of Collective2, we analyze the risk profile of each trading system, and then feed that risk profile into a simulation which repeats history thousands of times. The idea is that the equity curve you see on Collective2 is in fact only one path that could have been taken. It may have been a particularly lucky path, or a particularly unlucky one. By looking at the distribution of equally-likely results, you can learn more about the nature of each trading system. Is the distribution of results scattered all over the map? That means that – regardless of the results displayed on C2 – it is equally-likely that the system could have taken a worse, adverse path.
Like everything in life, it makes a lot more sense when you look at the pretty pictures. Monte Carlo Simulations are available for most trading systems. There’s a switch under the equity graph that allows you to see a simulation plot. There are still a handful of systems which have not been calculated yet, but they will be filled in over the course of the next day or so. This stuff takes a lot of CPU cycles. So be patient.
Over the next few weeks you’ll be seeing several important improvements and additions to C2 which are designed to allow subscribers to find lower-risk (but still successful) trading systems.
I am concerned that currently, C2 gives too much prominence to high-risk systems which trade recklessly and then blow up. My job is to make C2 a useful resource for traders. You’ll be seeing some further enhancements soon.
MK
Matthew: Great feautre, but I’m a touch confused. Shouldn’t the grey area be between the red and green lines? From the pop-up legend, I would assume that the red and green represent the best and worst outcomes, but some of the simulations lie outside these boundaries - at least for some systems.
Here’s the source of confusion. Thousands of simulations are run, but only a sample of them are drawn on the chart. That sample which is chosen to be drawn always includes two very special cases: “worst drawdown” and “highest high-water mark”. Note that those special simulation runs mean simply that a particular simulation had the best or worst at some point in its historical simulation. It does not mean that the run itself was the “best” or “worst.” (You can imagine a scenario in which one particular simulation rockets upwards, scores the “best” high-water mark, but then plummets and ends up as just a mediocre simulated run.) So the red lines and green don’t trace the frontier of possible runs. Then simply show one very special simulated run among many.
The main thing to look at (in my personal, non-mathematician’s opinion) is where the C2 actual run (the bright blue line) lies in relation to the bulk of the gray area (i.e. the equally likely area).
If you see that C2’s blue line is much above the gray area, that indicates the system results on C2 were essentially lucky, and that things could have easily been much worse. A blue line that lies within the gray area indicates that the C2 results were pretty “likely” from a probability point-of-view.
I’m sure Jules and others with better statistics backgrounds than I have can elucidate this further, or can correct me if I am saying something knuckle-headed.
Matthew
In my work with Monte Carlo simulations, we formed composite time-series curves whose ordinates were, at each abscissa (or point in time), chosen as a given percentile of all values recorded for that point in time. Thus, one could display the composite equity curve at a few percentiles (e.g. 1, 5, 50, 95, 99) to characterize the range of outcomes possible. These 5 curves, by their construction, would never cross one another. This is somewhat more robust than relying upon single-point estimates of ‘best-case’ gain or ‘worst-case’ drawdown to determine the two ‘extreme’ curves to be displayed. However, this technique may mask too much of the sequential time-dependency of trading; at least each of your curves was a coherent, realizable instance…
Wonderful. I don’t want to toot my own horn, but the Monte Carlo simulation for Pinnacle Trading just confirms my belief that my trading methodolgy is infact a sound and robust way to engage the markets rather than resembling a lucky streak. (As you mentioned, the blue line is well below the gray lines, and the red line is in outer space). That’s fabulous Matthew.
Matthew and Tarek,
I don’t know whether you see the same as I, but what I see is that the blue line of Pinnacle is a flat horizontal line at $0. So the “actual C2 fill” suggests that this system started and ended with $0. This does not agree with the picture that I get without the MC simulation. I don’t understand how that is possible. Is it a bug?
Matthew:
How do you construct an outcome that is “equally likely”? Is that on basis of the actual fills, back-reported by TradeBullet from the subscriber?
Jules
Hi Jules,
You know statistics better than I do. Matthew is the one running these simulations so I have no control over them. I did notice however, that after my first post this afternoon about MC, an hour or so later, the graph changed considerably, yet still showing very positive results, as if Matthew ran another simulation but just on my system. Why would he do that, I have no clue.
Tarek
I think the reason for the blue graph looking “flat” is because the high end of the scale represents 120 million dollars as the highest profitable outcome. I’m flattered with that representation to say the least. Will I achieve that – only time will tell.
I toggled on the MC simulation for the top 20 systems at C2. One of my system Midas Ultra Med-Term has 24 million dollars as the highest profitable outcome in 35 weeks. Yours has 120 million dollars in 113 weeks. Next is Timac with 8 millon dollars in 27 weeks. Compounding can change the picture dramatically. I’m surprised that others are not even close. Why is that?
I’m surprised that it so happens that Yours and Timac are the systems I have been watching for a long time. Maybe I’m good at identifying similar systems? I guess, If these results indicate robustness of a system, as you say, this should be proof enough for it. I have been running MC simulations for a long time but never had the chance to compare it with other systems. I don’t see any reason why the real-world results should not resemble this. All these hard work at C2 must pay off ultimately.
How do you construct an outcome that is “equally likely”? Is that on basis of the actual fills, back-reported by TradeBullet from the subscriber?
No. But, let me try to answer this question addressed to MK or rather let this gentleman’s article answer it at::
Monte Carlo Analysis : Day Trading Strategies : Forex Trading Systems : Adaptrade Software
I hope this helps.
Tarek, you’re right. I didn’t see the K in 120,000K. That explains it. Thanks,
Jules
Pal,
Thanks, that paper answers my question. As I understand it, the trades are the same, only their sequence is randomized. So the grey and the red lines are still based on the ‘best possible’ C2 fills.
I understand that many users have a problem with getting fills that are close to C2 fills, due to all kind of delays in the signals. So it would also be interesting to simulate with random delays, as I described above. The realism factor already gives an indication of this, but I can imagine that there is a large variation in this, with overall loss for some and overall profit for others, even with the sequence fixed.
This is not an attempt to critisize the present simulation, which is informative in its own way! Interesting to see that such an arbitrary change can produce such different outcomes.
What I don’t understand (but I didn’t think long about it yet) is how it is possible that some real curves lie consistently above most of the grey curves. This suggests that the system has been exceptionally lucky all the time. That is of course possible, but is also very unlikely. Therefore I wonder whether there is not some stable and essential charachteristic of some systems that is destroyed when you randomize the order.
For example, suppose that a system deliberately spread its risk by buying negatively correlated stocks at the same day and selling them at different times. Could it be that these two trades end up at very different positions in a randomized sequence? If so, the sequence with the largest dragdown may not be ‘equally likely’ at all, because the simulation destroyed the risk management of the system and the system would never do that in reality.
I don’t know whether this can happen; it depends on how the simulation is programmed. Again, this is not to critisize. I just want to know the limitations of the method (every method has limitations) and the consequences for the interpretation of the results.
Jules
What I don’t quite understand about these simulations is that if it’s randomly rearranging the trades to see other possible “equity paths”, shouldn’t ALL possibilities finish at the same end point? I mean, if we start with $100K, have trades of +1000, -2000, +4000, shouldn’t you end up with $103K regardless of how you rearrange them?
Interesting tool, nonetheless.
Hans.
I think (but I’m not sure, please let someone correct me if I’m wrong!) that the difference is caused by compounding. E.g. if a trade of +2000 is made when the ‘real’ account was 200K, then it will be rescaled to a trade of +1000 if it is positioned at the beginning of a simulated sequence, where the account is only 100K.
Jules
Jules,
MK did not specifically say that C2 was randomly rearranging the sequence order of the trades, only that ‘we analyze the risk profile of each trading system, and then feed that risk profile into a simulation’. I do not know exactly what comprises a ‘risk profile’, but would hope it includes consideration for missed fills and slippage. Matthew, would you care to clarify exactly what is your random variable in these Monte Carlo simulations, and from what type of distribution (e.g., normal, log-normal, uniform) are the draws being made? Some of us are curious as to how we should interpret your simulation results.
Alan
You’re right. MK didn’t say that (and I didn’t say that he said it), I infered that from the link that Pal gave. It says among other things
"In using a Monte Carlo approach to calculate the drawdown, the historical sequence of trades is randomized, and the rate of return and drawdown are calculated for the randomized sequence."
and
"If you choose a sequence of trades where five losses occur in a row, you could get a very large drawdown. The same trades arranged in a different order, such that the losses are evenly dispersed, might have a negligible drawdown."
But I think that I focussed to much on these two sentences. If you view the set of historical trades as a pool from which you can draw, then drawing without replacement would be the same as rearranging the sequence. But I think it would be better to draw with replacement. If that is done, it would be much easier to understand why the grey curves don’t end at the same point.
But then I still don’t understand why some real curves lie consistently above the simulated ones. Alan, do you have an explanation for this, other than my “destroyed risk management” hypothesis above?
I would also hope that missing fills and slippage are covered, but the site mentioned by Pal makes clear that this is not necessarily the case. Indeed, the only one who can answer this is MK.
Jules
Another factor in having the various equity curve plots mean something in terms of real account expectations is not being considered now, and that is how limit order systems are modeled when autotrading using Tradebullet (which, I’m assuming, is how most subscribers are autotrading any C2 system).
If no fill management is used in TB then some limit orders will be missed on entry. As has been discussed here many times, these will almost always be winning trades while losing trades will almost always be hit because of the direction of price movement. So there is a distinct bias for hitting a larger percentage of losing trades than winning trades, thereby lowering the profit potential compared to the C2 equity data despite the present adjustments being made for RF, slippage, etc.
If fill management is used in TB to convert limit orders (after X seconds, X being just about anything) to market, or MIT, once C2 reports a fill, then the slippage is almost always considerably worse than C2 shows for its fills (presumably due to the same price movement that always fills the losing trades but misses some of the winning trades). Add to this the additional (not insignificant) losses that occur from TB getting out of synch with C2 with limit order systems (ie. extra long or short positions that happen due to software errors, disconnects, etc.) and the real-world results for limit order systems always appear much better in the equity curves than they really can be in a real account autotrading using the C2/TB combo.
I realize there is far too little subscriber feedback on real trading account results using C2/TB autrading to do this, but it would be very interesting to see these curves added to the C2 equity plots (I’d be happy to supply my own account data). MK is doing a great job here continually updating the site and adding new features, but I think there is a long way to go before the equity curves accurately represent what can be expected in real world trading through the interfaces that are available now.
Jules,
I agree with your analysis of the maximum-drawdown with respect to sampling from historical trades with or without replacement; either case has the potential for violating risk-management strategies.
Looking at the Monte Carlo curves for ER2 Cash Cow, I sumise that there is a random draw for each day on which the system had one or more historic trades; notice the paucity of breakpoints in the gray curves. For an implausible set of curves that leaves me puzzled, see those for Dave’s Goofiz ATM, where the historic curve dramatically dominates the ‘best-case’ Monte Carlo curve until the very end. I cannot speculate how this could occur in a robust Monte Carlo simulation, and await further information from MK before putting too much confidence in this new feature.
RE: "For an implausible set of curves that leaves me puzzled, see those for Dave’s Goofiz ATM, where the historic curve dramatically dominates the ‘best-case’ Monte Carlo curve until the very end."
Click on the Realism Factor graph. That curve would (I think) fit within the MC simulation area. Maybe he’s using a Realism Factor adjusted MC? No idea, just guessing.
Hans,
I just tried that, and the first third of the historic equity curve adjusted by Realism Factor still lies way above the ‘best-case’ Monte Carlo. I remain unconvinced that the MC simulations are spanning the entire range of possibilities for this system.
