Matthew,
I have noticed that some of the EXLENCE drawdown stats are not correct. Particularly, I have noticed the profit on trades marked n/a for drawdown is not included in the APD calculation. The EXLENCE APD is currently understated by approximately 0.04. Also, the EXLENCE max drawdown is overstated.
1) Please correct the EXLENCE system Max Drawdown (currently showing at 73.51%; previously showed 23.84%).
2) Please adjust the Trade Drawdown for the@CDU9 trade closed on 7/21/09 to $90 (my time and sales show a lowest trade price of 0.8637 immediately after entry at 11:30:41 Eastern time).
I appreciate a prompt turnaround on this issue.
Chris
This is known shortcoming of the way ADP is currently calculated – that trades without any drawdown are not included in the ADP calcs. While we would all like a fast turnaround of this issue, the reality is a bit less happy: it is a known problem; it is slated to be fixed in the medium term future; but I have no current estimated time of completion for its being handled. I hope we can get to this soon, though.
Matthew,
Thanks for the prompt reply. Are you able to correct the EXLENCE total system Max Drawdown soon? Were any other systems affected with a similar change in Max Drawdown stats?
Chris
trades without any drawdown are not included in the ADP calcs
Umm, that does not include trades with zero drawdown. That means Trades that say "no calc" If you include trades with "no calc" - that means a profit or loss is added to the Profit, but there is nothing to add to the Drawdown, skewing the results. So these trades have to be ignored
Index, trades that have only drawdown and no profit are similarly excluded in order to avoid skewing results in the opposite direction, correct?
no
especially for scalping systems, drawdowns are sometimes not calculated for trades, so the trade lists "no calc"
These trades need to be removed, since they show a profit/loss, but the drawdown is unknown. They might win or lose which goes to the total "Profit" value, but their drawdown is unknown and not included. THAT is why they would be skewed.
I see, thanks for that. Has Matthew explained at any point why C2 is unable to calculate the drawdowns for those trades? I hadn’t raelized this before, and as this issue doesn’t only affect APD, but also the primary drawdown stat, it seems that it should be a relatively high priority to fix.
In statistical terms this is a problem of missing data. The currently used solution, ignoring the trades with ‘no calc’, is appropriate if the missing data are missing completely at random (MCAR). That is, the missingness mechanism does not depend on the observed or unobserved data. If this condition is not satisfied, for example if trades with 0 dd have a higher probability of getting a ‘no calc’, then ignoring these trades may actually create a bias. The best solution is: Get the drawdowns of all trades.
I don’t know how big this problem is. It would be interesting to compute in a sample of systems a kind of ‘best case’ APD, where all trades are used and where 0 is substituted for each ‘no calc’, and see how much the outcome deviates from the regular APD.
This issue we are discussing does not affect the primary drawdown stat. (That is, it does not affect the “Max Peak-to-Valley Drawdown” stat.) Rather, the problem affects only individual trade drawdown statistics. Specifically: if a trade has no drawdown whatsoever, it is not included in APD. As I said before, this is a known design flaw, and needs to be fixed, but it’s not a simple tweak and therefore can’t be resolved right away. I agree it’s important, and so I hope we’ll be able to implement in the not-too-distant future.
Matthew,
Thanks for the reply. APD is such an esoteric metric that I’m having trouble understanding it. It sounds like we have two issues here: trades that have zero drawdown, and trades where the drawdown isn’t calculated correctly (the “no calc” trades). In the first case, the trades with zero drawdown, if they are excluded from APD, shouldn’t trades with zero profit also be excluded from APD so that the calculation is symmetrical?
And in the case of the “no calc” trades, can you please provide some insight into why C2 is unable to make the calculation?
Trades with zero drawdown are not excluded from APD. Only trades with unknown drawdown are excluded. What I said is that, since we do not know why the drawdowns are unknown, it is possible that small drawdowns are more likely to become unknown, which would create a bias. This is a possibility, not a fact.
"APD is such an esoteric metric that I’m having trouble understanding it."
Not really. For any system:
1) throw out all rows that have "no calc"
2) For remaining rows:
…a. Sum P/L column
…b.Sum DD column
3) APD = 2a. / 2b.
Many systems in the past took huge trade drawdowns “holding & hoping” the trade went back to positive. This punishes such systems. They were prone to blowups upon disruptive market events, and the vendor would often just start a new system.
Other systems are generally random, but the vendor generally closes out trades to bank profits. If a “vanilla” system (probably 90% of the C2 systems), a very low APD is relational to a relatively random system. IF someone is profitable, but has an APD of 0.12, it means he risks $100 on average per $12 gained.
These above are major examples, not all cases. Many vendors do not grasp that trading is ONLY about Risk-Adjusted Reward. It is NOT about “winning%” or “total profit”. You can differentiate a knowledgeable trader from a newbie trader, if the latter goes blank or evasive when you start discussing Calmar, Sharpe, Profit Factor, or other stats.
Note: Trades with 0 drawdown that are NOT “no calc” should NOT be excluded, UNLESS there is known problem in C2. They represent the best trades.
Jules: "Trades with zero drawdown are not excluded from APD."
Matthew: "Specifically: if a trade has no drawdown whatsoever, it is not included in APD."
Jules, how do you reconcile this?
Index,
I agree with everything you said here. What we haven’t yet discovered is why “no calc” appears at all. What is preventing the calculation of the drawdown? If the entry point, exit point, and all prices in between are known? Or if it is the case that all prices in between are not known, why not?
The phrase "no drawdown whatsoever" is ambiguous. You interpreted it as a zero drawdown, but I think Matthew meant that the C2 database does not contain a value for the drawdown. That is how APD was originally defined by Index. Excluding the trades with known zero drawdown would definitely be wrong and it would be easy to fix.
To be sure that I am not carried away by my own phantasies, I just checked the computations in one example: Magic Moment System. The trade details of this system contain both drawdowns that are n/a and drawdowns that are 0. You can download the trade details of this system as a csv file via the beta version of C2, and compute the APD yourself.
- If you exclude the trades with n/a drawdown but include the trades with 0 drawdown then the outcome is -0.1486.
- If you exclude both the trades with n/a drawdown and the the trades with 0 drawdown then the outcome is -0.1885.
- C2 reports an APD of -0.15.
Conclusion: The computation of C2 includes the trades with zero drawdowns (as it should) and it excludes only the trades with non-available drawdowns.
PS. In this example you can also see how great the influence of n/a drawdowns might be for APD. If all n/a drawdowns of this system were actually 0, then the APD would be +0.91 instead of the -0.15 that is currently reported. That is a big IF, of course.
The trade details of this system can also give you a clue of the reason why many drawdowns are not available. This mostly happens with trades that were closed within 1 or 2 minutes after opening. However, correct me if I am wrong, but if such fast trades end profitable then it hard to imagine that they had a serious drawdown within these two minutes. This is why I suggest that it might be reasonable to include these trades in APD and to replace the n/a drawdown by a 0 drawdown.
Jules,
I appreciate your detailed answer. I’m afraid there’s been a miscommunication–I have no disagreement with anything you’ve said. But one issue remains unanswered, which is why the n/a calcs appear in the first place. If it’s true what you say, that this issue is due to the speed of the market, then I would agree with you that zero is a reasonable substitute for n/a and should be implemented (perhaps with a footnote for those trades affected as such). But if there is another reason why the n/a appears, I think it’s important that we find out why. Part of the genesis of this thread is that Chris Freeman noticed incorrect max drawdown stats for his system, but the resolution took place privately, so it’s unclear if the APD “n/a” and the max drawdown issues are related. Matthew said they are not, but what if the max drawdown takes place within that single trade (especially easy for a leveraged forex system)? Then it would appear that APD and Max Drawdown are related.
I think this could be cleared up if we could get some insight into why the “n/a” calcs are showing up in the first place.
There are three classes of cases in which the trade-specific drawdown stat is marked ‘n/a’ here at C2.
In the first, C2 simply has not yet calculated the drawdown stat. Each stat is calculated offline, on a delayed basis, because its an “expensive” operation, computationally speaking, and thus may be calculated up to several days after a trade is closed.
In the next (more uncommon) case, the stat has been calculated, but our historical price data is suspect in some way. This can happen in cases where there are stock splits, for example, and our data provider does not change the historical price data to reflect this. (Thus, we might see the price of an instrument go from 80 to 40 in the course of one minute.) This is not the only circumstance in which we find invalid data; sometimes there may be a few random bad ticks scattered through the day, which throw the calculation off.
In general, this second class of ‘n/a’ cases are truly random, and are probably evenly distributed between winning and losing trades, and so should not really affect the validity of the ADP stat in general.
The final case in which a C2 drawdown stat is marked ‘n/a’ is one where the drawdown does occur, but it occurs within a smaller granularity than our data lookup. So, for instance, a trade in which the drawdown occurs in minute number three of the trade, but our data lookup is done on the basis of 5-minute bars, might be marked with an ‘n/a.’ Here I’m less comfortable with the n/a stat, because I think that these cases tend to happen more frequently to profitable trades, and thus this problem skews the APD stat to be more unfavorable for some systems than it should be. This problem is solvable, but it’s not a trivial fix, and requires significant rewriting of some C2 software.
As I said in a previous post, we are aware of this problem and have it on our list of “issues to solve” (a better sounding term than “bug database,” I think). While I agree it’s important, and can understand why the system owner of a system affected by this, such as EXLENCE, would want this to be a top-level priority, it probably will not be fixed in the immediate future, and may take a bit of time.
I promise to keep you posted as we get closer to a fix of this issue.
"But one issue remains unanswered, which is why the n/a calcs appear in the first place. "
Matthew is the expert, regarding why no calcs appear. This has nothing to do with APD or any other C2 stat. It is simply where the drawdown does not appear, as Matthew described above.
Note that this per-trade drawdown column is also called in industry, MAE (or maximum adverse excursion). It is a hope someday for a few of us, that MFE (max favorable excursion) for each trade is listed. That would help people calculate a profit targets and system behavior. But I realize that is a significant user of resources.
