I understand C2 provides real time testing of systems and this is the most reliable way to test a system, but it takes so much time to generate a good track record on C2. It would be nice if C2 could INDEPENDENTLY backtest 100% mechanical systems. For example, I would like to see different systems tested against 2008 data. The systems should also be tested for ‘peeking’ during backtesting (i.e. change a future price and see if this changes the present signal and backtesting results). Any suggestions on how this could be implemented? Maybe there could be a C2 ‘stamp of approval’ on independently backtested systems.
Hi Warren,
this would require that you send your system’s code to C2.
In a standardized language for some well known simulation software.
In the case of Wealth-Lab you could send a C# script. But because the way WL works it is nearly impossible to tell if your code peeks into the future. There are many very subtle ways to use future information. The developer of the script or even the developer of WL may not be aware of the loophole your script uses. (I went through the excercise several times)
Some more modern software designs (like RightEdge (RE)) prevent this. But it would still require that:
1) you implement your trading algorithm as a C# or VB script for RE
2) some (very good) expert reviews your code for "dirty tricks"
3) a knowledgable person runs a simulation
If I had to do this as a service it would be very expensive and cost a few years worth of vendor’s fees on C2…
… and I doubt if serious investors would trust my results …
in principle it’s possible, even easy, automationable.
The “problems” just only arise (IMO) because the developers
don’t want it, don’t like it.
Even C2, whose main “mission” is to compare the systems by their
performance doen’t seem to want it (to be done by others),
so there seems to be a rule that
statistics about performance of C2-systems is forbidden.
maybe it’s time for Collective3 to do the same thing
on increased historical data rather than actual data …
We need to find an honest, independant, ‘bonded’, reliable, truthful, un-bribable individual who will sign a ‘non-disclosure of system agreement’ who has all the standard trading development softwares (Wealth-Lab, TradeStation, etc.) and who knows how to use them.
If we could find such a person (or organization) then they could independantly test and certify historic backtesting.
Surly there must be an honest individual in the world?
The closest thing to what your describing is Future Truth.
However, I fail to see any benefit in this idea. Why? Because backtests are hypothetical in the most strict definition of the word as far as the CFTC and NFA are concerned. I, for one, would never think for a second to buy a system based on its backtested results.
This is not, however, an endorsement of C2’s methodology, which is also seriously flawed in that it tracks trades AFTER they have passed through the C2 gateway and cause unreal worldlike fills (read, slippage) on autotraders’ accounts.
The best method is to have obfuscated code on the broker’s server that sends developer’s signals as soon as they are triggered by the code (not developer) and then executed in subscriber’s accounts, without the C2 tollbooth. This produces realworld fills in subscribers accounts with realistic slippage, too.
a notary ?
The fact is that there are developers of Stock, Futures and Forex Systems (I do not subscribe to any Option Systems so I cannot speak of them) who understand the limitations of C2 (scalping systems, for example, don’t work here) and have nevertheless created some very good systems, suitable for the C2 platform, and have happy subscribers.
Maybe in the future somebody will come up with a better platform (it probably would require deep pockets), many have tried but I don’t think anybody at this point can beat C2.
Give credit where credit is due.
Quite so. If a system is sailing very close to the wind (e.g. requires execution within less than a second of signal generation) then any intermediate platform will be unsuitable. I agree with Karl that C2 does a very good job especially considering the sheer breadth of supported instruments, markets and brokers.
Independent backtesting would be very nice. Unfortunately it is impractical for many (if not most) systems as Rene correctly says. Supposing the system is written with your own stand-alone code instead of WeathLab or similar tool? Supposing it involves discretion, being semi-automatic or completely manual?
At best the independent tester would have to charge very high fees to give the kind of personal attention that would be required to study a vendor’s code or rule-set and then choosing the right data sets to run the tests on.
At worst the tester would only accept easy cases, such as medium frequency technical analysis based systems written for WealthLab in a subset of markets. Now the only independently tested systems would all be very similar, cutting out systems that are “different” which may actually turn out to be among the best ones by virtue of them being outside the mainstream crowd.
