Status: AutoTrade Problem - 2016/10/11

At around noon Eastern USA time, we had a technical problem. Our AutoTrade servers lost their connections to our main database. This meant that for a few minutes, our AutoTrade servers believed that broker accounts should have no open positions. Thus C2’s AutoSync process closed many account positions. When the database connections were re-established, these positions were re-opened.

I am very sorry that this happened. I know that it was deeply disturbing to our customers, and worse, that - in most cases - it meant that customers incurred slippage costs (i.e. the cost of closing and reopening positions).

Here is what we are doing in response to this problem.

  1. We are investigating why database connectivity was lost temporarily, and then re-established. We’ve been running our AutoTrading service for about six years, and we have never seen this happen before. We will find the cause of this problem and then try to prevent it from recurring.

  2. Most important: we will change our “AutoSync” logic to respond more appropriately to situations like this. In cases where an unanticipated loss of database connectivity occurs, AutoSync should be automatically be suspended until the database connection is re-established. This will prevent the closing/opening cycle from happening in cases that are similar to this.

As I mentioned, we have never before had a case where the database simply “disappeared” for a brief period and then re-appeared. We’re going to try to learn from this and make sure that similar incidents are handled in a more graceful way.

I apologize to all C2 AutoTraders for this problem. I know you rely on C2, and you expect that our technology should “just work” – no hassles, no problems, no excuses. That’s the standard I hold us to, as well. Today we failed, and I am deeply sorry for this.

I’m going to work with the team here at C2 to investigate what happened, and to make sure that nothing similar happens again in the future.

If you have questions about your specific account, and position status within it, please reach out to our help desk at help@collective2.com. I and others will try to respond quickly and answer your questions.

2 Likes

Postscript: I closed the previous thread to move the conversation to this thread.

It’s not the first time it happened, it’s happens more often than you will acknowledged and I sent you email couple of times. It happened again just a minute ago, (most of my positions closed and reopened again). It will be good if you guys fix the problem when user report you and not let these problem to get worse. When it effects every body. It’s happening on the day, when marketing is moving and people have lost money. Its very disappointing.

4 Likes

Agreed. What about those guys who are trading instruments with a wide bid ask spread, Outage twice today. They paid twice for entry and will pay again when the trade exited. Not good

1 Like

Seriously are you guys not making enough money to hire proper developers to fix issues? You realize you are probably like a lawsuit away from going completely under no matter all your legal word-smithing in your terms.

When will this happen? (Obviously it didn’t happen yet.)

Please keep us posted and send us the post mortem. [No, I don’t want a T-shirt. But we lost $$ today. Yes, I know: we did sign a boatload of legal crap. Still.]

Theres a fine line with signing a bunch of legal crap and negligence, in the latter mountains of said legal crap won’t be able to protect them.

1 Like

My subscribers and I lost nearly $10,000 in actual money today due to these errors.

How is C2 going to make this right?? There are millions of dollars trading on my systems and it is hard to continue given these issues.

5 Likes

We’ve installed a patch (as of 16:00 ET) that will prevent this specific problem from recurring.

We’re also working on a broader solution to issues of this general nature. (I.E. When we see a large number of accounts requiring re-synchronization, we will pause attempts until we can confirm that there is not some kind of exception).

Are you thinking of an indemnification of any sort like free subscription or refund? We have lost real money today twice and are frustrated

1 Like

My account lost the connection at 08:58 PM PST and again at 12:41 PM PST.
I had 17 open positions that were closed and then reopened today resulting in 51 trades.
On top of the commission charges, all of them lost money as a result of adverse pricing.

I know we do not live in a perfect world, but thank you for putting a plan in place to avoid this in the future.
This is my first experience with this problem.

Correction to my last post. The first connection was lost at 08:58 AM PST not PM.

Hi Matthew,

Are these problems by any chance caused by you taking bandwidth away from your core business to accommodate new ventures like the recent opening of a new mutual like fund? That would be a mistake. I also noticed several weeks ago that the updating of the system pages lagged far behind. On the other hand I appreciate that you try to raise funds with a private offering, apparently to improve the system.

I would also appreciate this. I pay (what is to me) a lot of money to C2 every month and after a similar close-all-open-all occurrence I experienced in May 2016, plus the two times today, have been stung by this significant cost three times now. I understand it is impossible for C2 to cover the losses of all subscribers, but as I received my payment reminder email today minutes after the market closed on this very, very bad day, I think I can speak on behalf of the affected subscriber base that we would appreciate a break on autotrading fees or some other gesture of goodwill.

3 Likes

I am not sure I follow this planned solution. If the connection to the DB is lost, the autotrade system must detect that instead of assuming that

and trying to fix this (non-existent, incorrectly categorized) problem.

Based on your description that you posted earlier, the problems today was exactly this. The autotrade system incorrectly assumed that a large number of positions should be closed. It did that and then correctly realized that the accounts are out sync (and re-entered the positions.)

I strongly recommend that you fix the problem instead of patching it.

Please keep us informed.

Thanks.

Perhaps I didn’t explain it clearly. We can circle up via email and I can explain the technical reasoning behind what I wrote.

Is there anything the AutoTrade subscribers need do to ensure the patch fix will become effective on Oct 12? Do we need to stop AutoTrade and reactivate it before the next trading day?

I experienced the issue twice on Oct 11 and had lost serious money…please ensure it won’t happen again.

What’s up now with the open positions? I got three emails telling me that AutoTrade has been disabled for a specific position. Is this true now, so do I have to care manually of this position or not?

As you pointed out, this is the bug, not the failing DB connection. If there is no connection to the database, then the server should not assume that the DB is empty. This is just plain wrong, and it’s a serious bug. We live in an asynchronous world.

So, we’ve lost money and it’s your fault. I agree that a partial refund for the AutoTrade fee is certainly appropriate. I had a similar problem a few weeks ago.

1 Like

Not to pile on too much… you can use auto-trade without auto sync. I used auto sync for about 5 minutes over the summer. I turned it on, and next thing I knew it had sold everything in my acct. Then it started trying to add the exact same stocks back in. Since I use MB Trading, MB wouldn’t allow the new trades to take place until the next morning. MB no longer free’s up buying power in real time.
I turned off auto-sync, and doubt I’ll ever use it again.
However the fix IMO, is to check the acct to see if it’s already in sync, or at least already has the same/correct numbers of shares, before emptying the acct. It certainly sounds like it treated everyone here as if it thought they had just enabled auto-sync every time it reconnected to the brokers.

There is no guarantee in here, especially in autotrade n sync. Like today, it happened 2 times. If you can do manual trade, that’s the best solution. Otherwise, may be just stop autotrade until next week because we don’t know what is gonna happen. I bet, they are still monitoring after they put a new logic in their system. It can work or may be not, for precautions purposes just put low scale or stop using autotrade. It’s better loss opportunity rather than loss money. FYI, if you stop autotrade means you exit all positions. For me, I’m considering to discontinue this service for autotrade because it’s not worth it they can not fix this similar issue, make a little money n loss a lot because of technical issue. Terrible issue.