Okay, it’s confirmed. Gary is no doofus.
But, Gary, I bet the statistical exercise you ran and described in your previous post raised some gnawing doubts in your mind.
Namely: is the backtesting data provided by C2 “clean,” or does it suffer from survivorship bias?
For those readers who are not familiar with the term, let me describe survivorship bias. It’s a common pitfall that novice quants can fall into. Imagine you backtest a stock-picking strategy over the past ten years, and it returns amazing results. Chances are - unless you took specific action to avoid it, your results were positively skewed by survivorship bias.
How does survivorship bias infect your work? Imagine this: you sit down to run your backtest in 2023. You feed the database of all the available stock symbols into your computer program. You get amazing profitable results!
But wait. That “database of all the available stock symbols” are those symbols that still exist in 2023. Not included in your database are all those symbols that disappeared between the year 2013 and 2023. You know which stocks disappeared during that time? The stinky ones. The bankrupt companies. The losers. The stocks that went to zero.
If you had run your strategy back in 2018 (i.e. in real life), some of those companies would have been available in your 2018-vintage stock-symbol database. And your strategy would have invested in them. And you would have lost money.
But because you weren’t careful and thoughtful, when you ran your backtest in 2023, your software didn’t have any possibility of picking one of those loser stocks. They weren’t in your 2023-vintage database. That’s why your results were so great. (When I write “your” I’m not talking about Gary, obviously, but rather the mythical quant person who learns the hard way about survivorship bias.)
Anyway… that long explanation about survivorship bias is the necessary preamble for me to talk about Collective2’s Scoring Workbench.
When I built the C2 Scoring Workbench, I worked to avoid survivorship bias. Specifically, I made sure that any strategy that existed, and was “subscribable” in the past, would be included in the database that is used by the backtester. That includes “killed” strategies. And it includes strategies subsequently made private. In other words, if a strategy was available for subscription in 2021, for example, then the C2 backtester would include it in the tests/analysis – even if the strategy no longer exists as of today, when the test is being run.
So the C2 Score Workbench does not suffer from survivorship bias, right?
Well, there’s still a subtle problem that quants have a hard time overcoming. It affects all quants and backtesters, not just people who use C2 or who poke around on the Scoring Workbench.
What is the problem?
The problem is that you know.
You know what happened. You know there was a financial meltdown in 2008, and that real estate stocks and banks tanked. So necessarily, you never even bother to create a “long real-estate strategy” and run a backtest for the past 20 years.
You know that Covid happened. You never even bother to create the “buy travel and airline stocks” strategy and run that test over data including 2019-2020.
There are dramatized examples, of course, but the same issue happens in less obvious ways. Like this: you sit down in front of the C2 Score Workbench for the very first time, and you play around with a couple formulas and see how they perform on today’s data. The first two formulas are stinkers, which you attribute to the fact that the software is complicated, and you have no clue how to use it. But the third one returns a really nice leaderboard! And so you backtest it. And it works really well over a 12-month window.
But there is a subtle survivorship bias in this. You started you work only after you came up with a formula in the Workbench that happened to pick today’s winners. It was more than likely that the same formula would return the same strategies, and these strategies were also winners a mere 6 months ago. Etc.
Anyway, I’m not trying to cast shade on Gary’s work here. Frankly, I’m always surprised and delighted when someone bothers to sit down and actually use my software. And it does seem like Gary’s formula returned good results!
But I just wanted to offer two pieces of information here:
That I am aware of the possibility of survivorship bias, and I designed the C2 Scoring Workbench to avoid the most obvious kind. (i.e. When you backtest, all the “failed” and “disappeared” strategies are in the historical database.) That’s the good news.
The bad news is that, even so, it’s still very hard to be a human being and try to design a trading strategy that doesn’t suffer from a subtle form of bias, because you kinda sorta know how things turned out in history, and your brain will naturally gravitate to create and test strategies that happen to do okay in the world that transpired.