Matthew,
During the last trading hour TradeBullet was constantly disconnecting and re-connecting to C2 and even now it happens sporadically. As a result two positions of RD30 Day Trading were not closed out even though on C2 they show as closed.
In addition, for two systems on OEC some positions are not in sync.
What is the problem this time?
Karl
I’ve just learned that our data center’s main DNS server went down this afternoon, and internal requests, originating within C2, timed out. We’ve switched to the backup DNS server.
But I’m perplexed how this could affect your OEC positions, Karl. Could you please email me the details – specifically, what you believe is out of sync, and for which system – and me and Francis will look into it ASAP.
Matthew,
Apperently the backup DNS server is no working properly either. TB gets constantly disconnected and re-connected. BTW, I am running the latest TB version, build 761.
Karl
Matthew
Same here! I am having this same problem. Turned a good winning position into almost break even :-([LINKSYSTEM_37780234]
Ug. Not a good day for C2 infrastructure.
Okay, there was a process overloading a database server, and so I have shut it down and now everything seems to be more responsive. I apologize for this.
Matthew
Looking much better now. Thanks a lot.
Matthew - given we are betting real money and depend on C2 infrastructure to be highly available; I am hoping you are putting in place some rigorous production monitoring software to alert you when such problems occur. Without high level of availability - we are just bound to lose money left right and center.
Thx
Unfortunately, for me, the connection C2 from TB was still on and off during the night. And TB missed several signlas.
Matthew,
From 10:11AM till 10:45AM today I experienced the same issue again.
Please advise what is done to prevent this from happening again.
Thx, John
PS: Anybody else?
We did have a problem today, starting at 10:03 AM, lasting until 10:20 ish. We found the source of the problem, and then – to be cautious – we shut down and brought back online the gen1 servers to make sure the issue had not caused any corruption of unrelated processes, and thus any further problems. We’re back online now, and things appear stable.
John, I try to be as straight as I can be when it comes to discussing infrastructure problems. This hasn’t been a great week for us, obviously, but we are working through issues and resolving them. The indirect cause of many of these problems was the addition of a new server last week, designed to take load off of our older, overburdened servers. What should have been a helpful development turned into a problem, however, as we had unexpected trouble trying to make the new server to play nicely with the old ones.
I wish I could tell you this won’t happen again, but I cannot yet do so with certainty. I believe we found the cause of this morning’s problem, and I implemented a patch to prevent it from recurring. I am guardedly optimistic that we will enjoy stable and reliable infrastructure performance going forward.
Matthew
