The opinions expressed in these forums do not represent those of C2, and any discussion of profit/loss
is not indicative of future performance or success.
There is a substantial risk of loss in trading. You should therefore carefully consider
whether such trading is suitable for you in light of your financial condition. You should read,
understand, and consider the Risk Disclosure Statement that is provided by your broker
before you consider trading. Most people who trade lose money.
Say that I have developed an intraday strategy. And I found that it achieves a Sharpe ratio of 1.7 on SPY but only 0.5 on QQQ. What do you call this? Overfit?
I can further tune some parameters, so that the Sharpe of SPY goes to 1.1 and QQQ goes to 1.0. Is it now better or worse than before?
I think the reason the algo performed better on SPY is that sector rotation often causes mean reversion so that you can short at a high point and long at a low point. QQQ is one sector. QQQ goes one direction intraday and will not look back. The performance is related to the rotation and timing behavior Iâm trying to model after.
Monte Carlo is not going to help because the traits Iâm trying to identify only exist in real data.
What options do I have for validating this algo and calling it good?
Havenât got the minute bars to test yet. But according to my past experience, DIA is generally similar to SPY. IWM on the other hand may have a worse Sharpe than QQQ.
I donât have a technical implementation of walk-forward. But I noticed that the Sharpe stay relatively stable (except for Mar 2020, during which the return is high) in random one-year windows.
Itâs not machine learning, I couldnât really take out any rules. Parameters also havenât been optimized to the extremes. The histograms of daily low/highs have been stable over the years. Daily lows are more likely to happen in the first half of the day. Which is the clue for me to guess the entry. Only problem is that QQQ is more volatile and less trendy than SPY.
If your system is profitable then the Sharpe ratio is a rather meaningless metric, because it penalizes both the downside and, unfortunately, the upside volatility.
The Adjusted Sortino Ratio.is much better and will give you the real value of your system, try it.
Thanks for the idea. Sortino is good, although in my particular case, itâs not that different from Sharpe considering the strategy is in fact performing poorer in QQQ compared to SPY.
Never trust backtested results. Iâve seen hundreds (if not thousands) of systems blown up after they started to test it live claiming fabulous results.
If its never forward tested you will never know if it really works.
I figured âblow upsâ are mostly due to leverage. If some strategy seemingly provides stable returns at a very high leverage, there is usually some tail risk event causing it to blow up.
But I donât agree that âwalk forwardâ is the remedy here. In my experience, if I try hard enough, I can definitely overfit out-of-sample data. And life is short. You donât have endless time for testing. You certainly wonât run an algo thousands of years like a Monte Carlo test. Any strategy could work better in a certain market condition than others. I think itâs a judgment call and everyone is entitled to their own opinion.
One thing you can do is just do the past year as walk-forward, and see if the results are still good. Maybe use 6 years ago to 1 year ago as your data, and fiddle around with the model until it produces good results. The run the model from data from 1 year ago to now. Maybe also fit model to QQQ, and see how it performs on SPY.
But how much confidence do I have for 1 year walk forward results? 2023 feels drastically different from 2022. I have a hunch that the best strats in 2022 would not work in 2023, and vice versa. I feel like a mediocre return for both symbols is less overfitting.
I like to look at 1 year rolling returns, 1 year rolling profits pet trade and maximum number of weeks before system makes new highs to test for consistency and how likely I am to stick with it.
Many years help to form many different market conditions. A backtest from 1999-2020 had to go through many different market phases. Then a forward test from 2021-today, so that a backtest can be validated well.
Thank you for all your input. Although it seems that the only options available: backtest and walk forward are still pretty limited to give a full picture.
1 year walk forward test will reflect how someone who make the model on 6/30/2022 would have done the following year. â2023 feels drastically different from 2022â this is why people with good backtests fail when implementing system.
