Sorry, I didn’t understand that you meant that. That claim was indeed not subject to a significance test, and even if I did, there are alternative explanations possible that would require data that are much harder to obtain. But then I don’t see the point of testing this claim for the sake of our discussion, because I believe you said this yourself about a 100 times without showing any test. Is there another reason why you want this test?
"…about a 100 times without showing any test"
That is somewhat polemic (how come, in this forum?), please discard it. What I wanted to say is that I believe that you have many times stated the same claim, or a very similar claim, about the influence of luck on system statistics. So I doubt that a more rigorous analysis will help in our discussion, because this is imho not the point on which we disagree.
I didn’t really seek the test. I was only explaining my original comments.
Reflecting further upon your questions and what Ross said:
We should make a distinction between three elements in what I suggested:
1) Taken across systems, the variance of the Sharpe ratios decreases as N grows
2) The same is true for many other performance statistics that are based on means and / or variances.
3) The Sharpe ratio converges to 0 as N grows.
I think the first point is close to a statistical necessity under fairly general circumstances. The second point too, but it depends of course on the definition of the specific statistic that you consider. The third point is for me no more than an incidental empirical pattern of C2 forex systems, suggested (not proven) by the plot - although the efficient market hypothesis implies that it has more generality.
With respect to point (1), consider first the Sharpe ratio of a single system. If the distribution of returns stays constant over time, and if some fairly general other conditions are also satisfied, then the estimate of the Sharpe ratio converges to its true value as N increases, and the standard error of the estimate decreases with a factor 1/sqrt(N). So if we have a lot of systems with the same true Sharpe ratio but of different ages, and we make a plot like I did in the first post (Sharpe ratio on the vertical axis, N of days on the horizontal axis), then it will schematically look like this (here I cannot display small variations around the asymptote, but they will exist and they will gradually vanish):
Sharpe
*
**
******
*****************
******************************************************* -> N of days
****************
*****
**
*
Next, consider a set of systems with different true Sharpe ratios. If we make similar plot for these systems, this will be a superposition of plots like the one above, but with different limits. So it will look like this:
Sharpe
*
**
******
*****************
*******************************************************
******************************************************* -> N of days
*******************************************************
****************
*****
**
*
So if we select the system with the highest Sharpe, it will most likely be a young system. Moreover, this effect continues to exist if we confine ourselves to systems above a certain age, although the effect becomes gradually weaker. And if we confine our discussion to systems with above-average Sharpe, this means that we consider only the upper half of the last figure, which looks like this:
Sharpe
*
**
******
*****************
*******************************************************
-> N of days
Within this selection, the older systems will on average have a smaller Sharpe than the younger systems.
For this effect it is not even necessary that the distribution of returns is constant over time. The only thing that you need to assume is that the statistic becomes a more reliable estimate when N increases. Whenever this is true, then the described effect is a logical consequence of it. Thats why I think that the effect does not only apply to the Sharpe, but also to many other statistics.
With respect to the plot that I gave in my first post: Although you can think of better ways to obtain the data, I dont think that this can change the essential conclusion that any performance statistic that satisfies a law of large numbers (i.e. becomes more reliable when N grows) will also introduce a selection bias against old systems.
Jules,
I like your explanation. It seems a bayesian approach might be appropriate, where a prior distribution is based on the entire universe of C2 systems.
I dont think that this can change the essential conclusion that any performance statistic that satisfies a law of large numbers (i.e. becomes more reliable when N grows) will also introduce a selection bias against old systems.
The more I think about this, facets of what different people said lead me to think this is really the driving forces behind your diminishing Sharpe observation(s):
Theorem 1
Almost all vendors and subscribers confuse luck with skill.
Theorem 2
As time passes, different market conditions expose that very few systems outperform, generally. The market is actually quite efficient.
Theorem 3
As time passes, the luck that drives most systems, that originally look good, fails. Holding & hoping, leveraging a strong trend, money management, and other parlor tricks only push off the Day of Reckoning for this crowd.
In gambling establishments, this is called the House Advantage.
Conclusion
When you find an exception, stay with them.
Yes, that is basically what I want to say. If I may go into the specific formulation:
Wrt. Theorem 1: Well, I don’t know what most vendors and subscribers actually think, because I am not clearvoyant :-). That’s why my formulation is more boring to read…But if they simply select a system with high performance statistics and above a certain minimum age, then they will most likely confuse luck with skill - at least partially. Indeed, that is one of the things that I am trying to communicate.
Wrt. Theorem 2: Yes, this was what I tried to illustrate with the plot of my first post, corresponding to point 3 of my previous post. But as you and ST pointed out, it is not a proof. And the plot is about C2 forex systems only.
Wrt. Theorem 3: Exactly. That was what I tried to say in my last post (except that I didn’t claim to know which trading methods are responsible for this).
Wrt. Conclusion: Yes. In addition, it will be very hard to find an exception, because it will probably be dominated by some younger systems that had more luck.
It is like trying to find a stealth plane on a radar screen full of white noise.
But if they simply select a system with high performance statistics and above a certain minimum age, then they will most likely confuse luck with skill
Do you mean they think the vendor is lucky, while in fact he has some skills, or the other way around?
the other way around
With respect to your suggestion of Bayesian statistics: Yes, I agree. I am not specialized in Bayesian statistics, but my intuition of it is that it would produce something comparable to my suggestion of the t-value. Basically it would adjust the statistic of a young system toward the mean, i.e. downward if the value is above average. When you compare systems, this equivalent to adjusting older systems upward - which is what the t-value does. But anyone with more expertise in Bayesian statistics is invited to correct me on this…
I think you’re right. I’ve been thinking about doing a Bayesian analysis for a while, but it will take quite some time to gather all the data necessary to estimate the prior. The nice thing about a Bayesian analysis is that you can include system characteristics that are not directly observable from the returns series, e.g. the amount of leverage, trading frequency, whether it trades stocks, forex or futures etc. As the system is very young, the relevant statistic of interest (e.g. Sharpe) is shrunk entirely towards the mean of all other systems, conditionally on these system characteristics. Once it gets more history, this prior information becomes less and less important and the system is judged primarily on its own merits.
Wrt the “luck or skill” discussion, I do think it is possible to simply pick a system based on some criteria, but it requires a much larger number of observations than most people are willing to wait for. At the same time I also think that subscribers should be very serious about regression to the mean and take into account that a future scenario with twice the historical risk and half the reward is quite plausible.
There’s also a theory that says that a trading system (in general) has only a limited life time. At some point the “secret” (i.e. market inefficiency) is discovered by too many people and traded out of the market, after which the system goes flat or downward. That could be another explanation of your empirical results. If that is a valid explanation, it makes things a lot more complicated, as you need to subscribe to a system as quickly as possible (i.e. with a minimum of data to assess performance) before it loses its edge.
The equity curve is often used to identify a traders skill, but its impossible to determine the time frame required to establish whether returns generated are due to skill or luck.
If markets are efficient and current prices fully reflect all information, then buying and selling securities in an attempt to outperform the market will effectively be a game of chance rather than skill. If a market is efficient, no information or analysis can be expected to result in outperformance of an appropriate benchmark.
Analysis on each individual trade often provides insight into whether returns generated are due to skill or luck.
Most that do not have the capability of making any meaningful money in the market simply state that it could not be done and anyone who has done it is merely lucky.
Buffett was right in viewing the efficient-market hypothesizers with profound skepticism. “Observing that the market was frequently efficient,” he wrote of them in 1988, “they went on to conclude incorrectly that the market was always efficient. The difference between the propositions is night and day.” [Emphasis added.] Quote from:
http://www.capmag.com/article.asp?id=2623
According to John Allen Paulos, author of a wonderful new book, A Mathematician Plays the Stock Market (Basic Books), that’s all to the good. The paradox, he told me in an interview last week, is that, if all investors were convinced that markets were efficient, they would simply sit on their holdings, never buying or selling. In such a world, new information about stocks would never enter the market, moving the prices of stocks in an efficient way.
“The EMH is true,” he said, “to the extent that people believe it to be false and so, by their exertions, bring about efficiency.” Cool.
This is what Paulos does best – identify paradoxes and puzzles, not just in the market but also in everyday life. He is the author of one of my favorite books of all time, Innumeracy, published in 1988. The book’s title is a term he invented to describe the mathematical illiteracy of Americans – exploited daily by such institutions as state-run lotteries, which Voltaire, more than 200 years ago, called a "tax on stupidity."
Quote from:
http://www.capmag.com/article.asp?id=2883
Efficient Market Hypothesis (EMH) states that at any time, the price fully captures all known knowledge. Since, all known knowledge is used optimally by market participants, price variations are random, as new information occurs randomly. Thus, prices perform a “random walk” and it is not possible to beat the market. Despite its rather strong statement that is untrue in practice, there have been inconclusive evidence in rejecting EMH. In practice, stock market crashes such as in 1987, contradict EMH because they are not based on randomly occurring knowledge, but arise in times of overwhelming investor fear. if EMH is valid, then all analysis should lead to no better performance than random guessing. The fact that many have beat the market consistently is an indication that EMH is not true in practice. It may be true in the ideal world with equal knowledge distribution.
I would take issue with the belief that the market is efficient (EMH), and all available knowledge has been incorporated into the price at any given moment. Perception is everything. Misperceptions make for inefficiencies and Chaos, and inefficiencies make for good trades. Contrary to EMH, several researchers claim that the market and other complex systems exhibit Chaos. Chaos is a nonlinear deterministic process which only appears random because it can not be easily expressed by traditional technical analysis or fundamental analysis alone. Chaos theory analysis a process under the assumption that part of the process is deterministic and part of the process is random. Chaos is a nonlinear process which appears to be random. Various theoretical tests have been developed to test if a system is chaotic (has chaos in its time series.) Chaos theory is an attempt to show that order does exist in apparent randomness. By implying that the market is chaotic and not simply random, chaos theory contradicts EMH. A chaotic system is a combination of a deterministic
and a random process. The deterministic process can be characterized using regression fitting, while the random process can be characterized by statistical parameters of a distribution function. Thus, using only deterministic or statistical techniques will not fully capture the nature of a chaotic system. A NN’s ability to capture both deterministic and random features makes it ideal for modeling chaotic systems.
I am not sure whether adjusting systems is the point here. It is possible, but all this may do, is to normalize systems, so they look more alike. I dont know that it exposes better systems much.
I think the much bigger problem, is trying to weed out lucky systems, so that the few truly systems remain. I have tried to make this point in the past, but it is hard to get through the general forum noise.
Keeping the 3 theorems above in mind, my current heuristic includes some of these elements:
1) Reasonable equity curve
2) Reasonable Sharpe, APD, Profit Factor
(At this point, I am onto a list of systems that are lucky or good)
3) Watch system for at least a few weeks, to see if system continues to outperform or whether they start behaving randomly. This somewhat eliminates systems that are lucky. After all, how many people say "when I start following it, it stopped working!!!" or "the drawdown started when I started trading it." As Science Trader said, people are really not willing to wait long enough…
(At this point, there are VERY few systems left. If it outperforms, then it is likely in my list). I doubt whether you can really separate out ALL the lucky systems. But you can get closer.
" If markets are efficient and current prices fully reflect all information, then buying and selling securities in an attempt to outperform the market will effectively be a game of chance rather than skill. If a market is efficient, no information or analysis can be expected to result in outperformance of an appropriate benchmark."
If a trader buys efficient moves up and sells efficient moves down the market will be outperformed. Or for that matter replace the word efficient with inefficient and you will have the same results.
There’s also a theory that says that a trading system (in general) has only a limited life time. At some point the “secret” (i.e. market inefficiency) is discovered by too many people and traded out of the market, after which the system goes flat or downward. That could be another explanation of your empirical results. If that is a valid explanation, it makes things a lot more complicated, as you need to subscribe to a system as quickly as possible (i.e. with a minimum of data to assess performance) before it loses its edge. (ST)
You are correct. I tried to balance between two different explanations for the phenomenon:
1. A statistical cause (estimates becoming more reliable implies to regression to the mean, which is essentially running out of luck)
2. A substantive cause (market inefficiencies fading out, changing market conditions).
The first one is close to a statistical necessity. The second one must be established empirically, and there may be exceptions to it.
However, I believe these two explanations are actually the same on a somewhat higher level. For example, suppose that a system exploits a trend channel that continues to exist for 6 months. It will look as if this system beats the odds. But at some point in time this trend channel will cease to exist, the market will becoming more chaotic, and the advantage of the system will disappear if it does not adapt. You can describe this as a fading market inefficiency, but on a larger time frame you can also describe it with a statistical model that is a little more complicated than assuming uncorrelated returns from the same distribution. For example, the model could be that the market can be in different states (market conditions), and that there is a different distribution of returns associated with each state. The systems starts in one state and on each day there is a small probability that it jumps to another state (change of market condition). This will lead to the regression effect if you consider a large number of systems, but it is also compatible with the idea of fading market inefficiencies.
The consequence of this would indeed be that, for the systems to which this applies, it could be profitable to subscribe before it loses its edge (and to unsubscribe when it seems to lose its edge). That would be a sort of trading the trading systems. Obviously this strategy has many dangers, and you may be better off by trading the market directly.
The equity curve is often used to identify a traders skill, but its impossible to determine the time frame required to establish whether returns generated are due to skill or luck. (Sam G)
That is one of the problems too, and I think it applies to other statistics as well. Indeed the regression effect continues to exist at all time frames.
Watch system for at least a few weeks, to see if system continues to outperform or whether they start behaving randomly. This somewhat eliminates systems that are lucky. (Ross)
One thing that follows from my analysis is that waiting some time is not enough. The regression effect continues to exist at all times, although its size decreases. For example, if you confine yourself to systems older than 6 months, then it will still be true that some younger systems seem to have an advantage over older systems. One reason for this is: Suppose a system was indeed lucky for 6 months, and received a high Sharpe because of that. Suppose that the returns become random with mean 0 after this. Then the Sharpe will still remain high for a very long time. So this system will still seem to dominate many older systems.
When trying to eliminate this problem you can do something like this: First, wait x months and select systems with a high performance. Then, wait another x months and assess the performance again over only the last x months. However, I think this is quite similar to waiting 2x months and assessing the performance over 2x months. The problem is that you dont know how large x should be, as Sam G pointed out. My guess would be that a different value of x can be needed for each system (see also the Markov model that I sketched to ST).
“The EMH is true,” he said, “to the extent that people believe it to be false and so, by their exertions, bring about efficiency.” Cool. (Pal)
That is cool indeed. Tax on stupidity is cool too.
Could it be that any “system” that has consistently and significantly outperformed or “beaten” the market, is perhaps having one incredibly lucky streak?
And why do most investors buy at the top and sell at the bottom. It’s perhaps because when they finally get comfortable with the evolving trend - they buy…and much to their amazement/expectations a reversal of the trend causes dismay and panic whch prompts selling at the bottom.
The same can be applied to C2 systems…buy into them when they are in their rocket ship phase - all the while holding fast to your stop target that closes out all your positions - and then move on to the next. Seems reasonalby enough for one to handsomely “profit” from these “lucky streaks”. No?
Also a word regarding equity curves…somewhere between mine (for now take KC.com charts) - which are a little jumpy but acceptable and one like Steve Auger’s and you perhaps will have an evolving successful “system”. (I’m pulling for him to have some method to retain most gains if we go into a prolonged downturn, we’ll just have to see.) The rest - just on sight - can be dismissed, except for the trading process outlined above.
Gilbert
OK…if it pleases, forget my charts - but you get the gist. In time one can tell simply by the equity curve if there is any hope for any length of success. Then weigh the vendor’s progression in comparison to 2 or more “normal” (10-20% drops to a major iaverage - that take 2-3 months to actually resolve/bottom).
From that data alone, you can make a reasonable decision with your hard earned money to perhaps compound it into anything worth writing home about.
The rest is just noise. Unless someone comes along and can actually rocket upward - all the while keeping up the compounded (say 6 month) percent return (even Dustin had a short time frame, from what I can see) - and to everyone’s dismay has a comparatively small (minimal) drawdown (re: <10%) during market corrections or at any given time.
OH! and that the trades can be resonably duplicated with auto-trading.
If you know of any such “lucky” systems post your favorites here and we can discuss (cite possible flaws alongside the excellent points) them further (just referring to my above assertion that I don’t even believe otherwise what’s the point?). Gilbert
"Could it be that any “system” that has consistently and significantly outperformed or “beaten” the market, is perhaps having one incredibly lucky streak?"
Sure they are, specially when you define LUCK = When preparation meets opportunity. In this case, your luck is unusually persistent (stubborn) and will last a very long time. That’s the kind of luck I like to embrace
"And why do most investors buy at the top and sell at the bottom. It’s perhaps because when they finally get comfortable with the evolving trend - they buy…and much to their amazement/expectations a reversal of the trend causes dismay and panic whch prompts selling at the bottom. "
A Very good question. They do that because they are fearful of losing if they buy in an obvious downtrend and end up looking (and feeling) stupid. More than likely, it stems from an unpleasant experience that has been deeply etched in their subconscious mind. What’s so funny about this particular behavior is that when they finally pick a comfortable point to enter, they refuse to admit they’re wrong until it hurts their wallet really bad. Of course by then, it’s too little too late.
As far as the same applies to subscribing to systems, this suggests that being too conservative (waiting a long time before you subscribe, waiting again a long time before you unsubscribe) can be just as bad or even worse as being too liberal.
That’s why I said a C2 customer can track all systems and filter them based on a well-rounded set of criteria like stocks.
Then as sytems make the top 10 list subscribe - with a limit on when all positions should be closed due to drawdown.
Then take that percentage of available capital - all the while keeping your watchlist updated as some will filter up and others fall right off - and place it in the next decently performing system.
Always use limit losses with each particular vendor and keep your trading head on straight. Perhaps a good way to utilize the somewhat sporadic performance from what at first appears as a good C2 vendor.
Many may not even reach the free trial end date - and still provide a nice profit!
Gilbert
But, as I said earlier, there are a lot of dangers with that strategy. For one, this is in fact a trading system too, but a trading system for which you have no performance statistics (yet). For all we know it can be the same as subscribing randomly, or worse, or better.
