The opinions expressed in these forums do not represent those of C2, and any discussion of profit/loss is not indicative of future performance or success. There is a substantial risk of loss in trading. You should therefore carefully consider whether such trading is suitable for you in light of your financial condition. You should read, understand, and consider the Risk Disclosure Statement that is provided by your broker before you consider trading. Most people who trade lose money.
The subject matter may be a little dry and esoteric for some, but if you’re of a mathematical bent, you’ll be interested in the work of Daniil, one of C2’s quant analysts. He just published a blog post here:
https://medium.com/collective2
It describes his work so far trying to create an optimal portfolio of Collective2 trading strategies.
Interesting read to market smart portfolios. Is the top-3 ranking formula available for testing? @MatthewKlein
Hi,
Yes, there is a simple formula derived from this tool and is currently testing in real time with a smart portfolio.
A few words on how the formula is derived:
Even without taking the boundary values of the attributes, to go through only weights from 0 to 5. We get 2176782336(6^12) combinations. Agree that this is a lot even for a machine search.
So there is a lot of art in developing the formula at this stage.
I’ve tried quite a few complex variants of the formula (many attributes) and a number of simple ones (few attributes).
Of the variants I have tried (a drop in the ocean of all possible combinations), the simple formulas have shown the most stable results in terms of different portfolio sizes. What I mean is that sometimes the formula for the top 5 portfolio shows much worse results for top 4 or top 6 portfolios. This is an indicator of over fitting.
There is no such thing in the case of simple formulas.
Thank you for the info Daniil (and Matthew), I truly admire “quants” like yourself, they can find and exploit tons of market inefficiencies that the average trader cannot possibly see.
Keep up the good work. 
Thanks mate. Very useful.
Kindly keep us updated here of the monthly results of the top3 and top10 portfolios 
Sure, thanks for the interest.
Hello,
New update for the tool and real-time testing report.
Optimization of calculations. Managed to keep the performance of the tool at the same level (before the addition of the Summary tab)
Renamed the main Overview tab to Workshop. Now the name corresponds to the content
Do not hesitate to propose any of your desires, will implement as far as possible and expediency.
Oooh! The power of overly optimized backtesting! Nice curve.
While it’s quite possible we made errors in implementation, the design of the methodology was meant to be very thoughtful about hindsight bias.
A few specifics:
When we say, “Let’s take the top 5 strategies in January 2022 and look at their subsequent 3 month performance” we actually go back to the strategy rankings as of January 2022 – without any particular knowledge about how the strategies scored after that date.
We also designed the methodology to be aware of survivorship bias – a very insidious but surprisingly common flaw in many people’s backtesting. A common example of this is someone who wants to analyze at a database of “the entire set of stock symbols” as of June 2022. Perhaps he says, “Gee I wonder what would have happened if I went back in time and used Method X to select from this database of all known stock symbols.”
The inexperienced analyst tells himself: “It’s cool – there’s no bias, because, after all, my database contains the entire universe of today’s stock symbols. I’m not cherry-picking only the best stocks.”
The flaw here is not obvious. Do you see what it is? It’s that lots of shitty companies were delisted and disappeared between January 2022 and June 2022… so that “database of all known stock symbols” that you “choose from” with complete dispassion is actually filtered to have previously gotten rid of the crap.
So, yeah, we designed our methods to avoid this. We never “delist” strategies from our database. Crappy strategies that “disappear” are still in the database, etc.
Again, I’m not guaranteeing Daniil and the team here didn’t make a bonehead mistake – these kind of mistakes are very common when doing this sort of analysis – but I am saying that we’re not complete nubes, and yeah we understand how it’s easy to over-fit data with the benefit of hindsight. We try to avoid those pitfalls.
I want to believe these results. They just seem too good to be true. I would love to see the results of a real account dedicated to just following this screening system. Then it would be great to see the results for the real account compared to a backtest over the same time period.
Hi,
Please include the drawdown of the top 3% and top 10% systems (plus the drawdown of the S&P 500 itself) in your total portfolio table.
Thank you.
Yes, I’m dubious too!
There’s definitely no holy grail. So I do think there must be something wrong with these results. I only wanted to point out that whatever it is, it’s not a “let’s just blindly over-optimize backtest data” error baked into the design (at least as far as I can determine), but is more likely an implementation issue, perhaps with the way the historical data is generated.
Good afternoon Mr Klein,
So according to the graph above the return per year is 141%, is that correct?
I don’t know. If Daniil wants to do some calculations, I’ll defer to him.
And actually, thinking about it more, the problem here is probably a version of the “wastepaper basket” problem. When you have a nice UI where you can pull sliders left and right, and see results if you do this, or do that; then it’s easy to find some set of parameters that produce nice results. (That’s the analogy to a “wastepaper basket” filled with 100 crumpled papers with ideas that didn’t work; then finally idea #101 just happens to be great! You publish idea #101 but never talk about ideas 1-100.)
The way around this is to use out-of-sample testing: I.E. decide on a model that probably will work, run it on days 1 -1000 of backtest data, see that it actually did work okay… and then run it on days 1001-1500 to confirm.
But the more times you try to find a winning strategy in days 1-1000, and the more degrees of freedom you have to play with (i.e. how many little sliders and stats can you play with), then the less likely that the out-of-sample test performance will correspond to the in-sample test.
Incidentally, the “Strategy Scoring Workbench” does allow users to test out-of-sample.
That’s what his table says in any case : C2-return hosted at ImgBB — ImgBB
I am not sure if he is talking about the entire test period (almost 5 years) or the return per year, that’s why I asked.
good point, i assumed it was annual return and then of course it is a joke. but if that is the total ROI after 5 years, things become more believable
Absolutely, getting a 141% return after 5 years is like making 19.24% a year, on a compound basis.
