Why does this happen so often? On such an important day in the market too!
Note the time below is central time.
2011-08-08 09:35:25:948 (2011-08-08 14:35 UTC) (Connection) Collective2 polling errors for 120 seconds. Reconnecting to Collective2
2011-08-08 09:35:25:973 (2011-08-08 14:35 UTC) (Connection) Disconnecting from Collective2
2011-08-08 09:35:26:968 (2011-08-08 14:35 UTC) (Connection) Collective2 Logout response error: The operation has timed-out… Trying again…
2011-08-08 09:35:34:054 (2011-08-08 14:35 UTC) Sending logon to Collective2 for [deleted]
2011-08-08 09:35:34:054 (2011-08-08 14:35 UTC) (Connection) Connecting to Collective2
2011-08-08 09:35:34:209 (2011-08-08 14:35 UTC) (Connection) Connected to Collective2 and account [deleted]
2011-08-08 09:36:34:212 (2011-08-08 14:36 UTC) (Connection) Collective2 Logon request error: Can’t reach the C2 server (The operation has timed-out.)
2011-08-08 09:41:37:937 (2011-08-08 14:41 UTC) (Connection) Collective2 polling errors for 120 seconds. Reconnecting to Collective2
2011-08-08 09:41:37:938 (2011-08-08 14:41 UTC) (Connection) Disconnecting from Collective2
2011-08-08 09:41:38:051 (2011-08-08 14:41 UTC) (Connection) Disconnected from Collective2
2011-08-08 09:41:45:055 (2011-08-08 14:41 UTC) Sending logon to Collective2 for [deleted]
2011-08-08 09:41:45:056 (2011-08-08 14:41 UTC) (Connection) Connecting to Collective2
…
(still happening up to now) 2:38 PM Eastern
Ahhh, no wonder, and I thought there was some connection issue with my NinjaTrader program to C2. I had to manually close out all my positions and called it a day. How often does this happen, Alex, or anyone???
Alex:
We had major problems with one of our two (but primary!) data providers today. I’m not sure if it was caused by the market meltdown (they couldn’t provide the data fast enough) or if it was unrelated.
I know this is not really your problem: you expect C2 just “to work” and it’s my problem to make sure it does. I do try to do whatever is needed to make C2 stable and able to handle increasing loads. We’ve invested a ton in our own internal infrastructure over the past few months, in both our data centers, and we finally had been enjoying the fruits of that investment (faster site, able to handle loads, etc.) when yesterday’s problems set us back a bit.
To be clear: the problems were transient yesterday, but far-reaching. They affected Gen1 servers, the web site, the Hypothetical Fill Engine, etc. I have contacted our data provider and I am hopeful that yesterday’s problems during the market decline will not be repeated today.
Part of this is out of my hands, though. I know this isn’t what you want to hear, but we are very dependent on timely delivery of market data by our vendors. This may be one area where C2 will probably need to beef up redundancy, though, and opt for secondary (or backup) vendors.
I have my fingers crossed that today will proceed more smoothly. I am really sorry for yesterday’s delay and outage issues.
Matthew
DOWN AGAIN TODAY! Unbelievable when we need C2 at this critical market juncture.
(times below central time)
2011-08-09 08:32:35:902 (2011-08-09 13:32 UTC) (Connection) Collective2 polling errors for 120 seconds. Reconnecting to Collective2
2011-08-09 08:32:35:905 (2011-08-09 13:32 UTC) (Connection) Disconnecting from Collective2
2011-08-09 08:32:36:901 (2011-08-09 13:32 UTC) (Connection) Collective2 Logout response error: The operation has timed-out… Trying again…
2011-08-09 08:32:43:918 (2011-08-09 13:32 UTC) Sending logon to Collective2 for [deleted]
2011-08-09 08:32:43:919 (2011-08-09 13:32 UTC) (Connection) Connecting to Collective2
2011-08-09 08:32:44:298 (2011-08-09 13:32 UTC) (Connection) Connected to Collective2 and account [deleted]
2011-08-09 08:32:53:178 (2011-08-09 13:32 UTC) (Connection) Collective2 Logon request error: Can’t reach the C2 server (The underlying connection was closed: An unexpected error occurred on a receive.)
Market sell no fill for 30 minutes.
Wipe out 30% of my account.
Hi Matt,
My futures platform have not received a data feed for the entire day.
I received a incorrect fill yesterday. Your support team correct the fill but the current price is saying "0". I send support an email this morning but nothing was done.
Please have your team check on this for me immediately.
[LINKSYSTEM_60272880]
Glen:
We’re receiving price = 0 from the exchange right now on the June 2012 YM contract. Hang in there - these things usually get corrected soon. Things have been nuts at all the data services. I’ll correct the erroneous equity chart as soon as we can get a good price on that contract.
Thanks in advance.
Regards,
[LINKSYSTEM_60272880]
I can’t tell you how disappointed I am with C2 about this. I run an automated system to submit the orders. At 9:32 (Chicago) my program sent a market order to buy wheat futures. The C2 site did not reply, and the API call received http message 504 indicating a timeout. My program assumed the order had not been placed and terminated for the day. Subsequently, I found that C2 had executed this market order 4 minutes later when the market had risen nearly 14 points! Not only did clients miss a very profitable trade, but they also sustained a large loss because the market fell again before I was able to close the order manually.
To add insult to injury, after submitting 2 problem tickets for the trade asking for the entry price in the trading record to be adjusted to the market price prevailing when the order was issued, I have had absolutely no response.
David Bennett [LINKSYSTEM_56830792]
Hi, David:
I’m sorry for what you’ve experienced. As I said in another post in this thread, due to the trading volumes during the recent market volatility, one of our data providers went down. This caused a cascading problem at C2 and brought down the site intermittently as well.
This affected many systems in different ways. In your case, your trade signals were accepted, but executed later than you entered them. This caused slippage in your customer accounts. Unfortunately, due to regulatory restrictions, I am required to post the actual fills achieved by your subscribers, even though these fills had slippage due to a technical issue on the site. This wasn’t your fault, nor what you intended, but I cannot change that, and for that I am sorry.
We try never to have downtime. The first days of this week were quite bad, with several (brief, but painful) periods of downtime during peak trading hours, due to the failure of our data provider. Again, as I mentioned in my previous post, we have beefed up a lot of our infrastructure recently – both hardware and software – but this was one area that we had never had problems with before, in 8 years of running this site. This experience has taught me that this is a possible point of failure on the site, and now we are moving forward on a project to add some level of redundancy here as well.
Regarding current status:
Yesterday, after a brief period of slow responsiveness during the stock market open, performance seemed to stabilize, and we handled the rest of the day without incident, even during the end-of-day peak.
So, I’m hopeful that today will be the first “perfect” day (from an infrastructure operations point of view) all week. We shall see.
Finally, I’m sorry I didn’t respond to you personally in a timely way. Sometimes, when I get swamped, common courtesy goes overboard, and for that I apologize.
Matthew
