I want to update people about last night’s Gen3 AutoTrade down time. We were forced to take our database offline between 1:23 AM ET and 3:30 AM ET on Sunday night / Monday morning. We did this in order to repair our backup database which became corrupted last night. (For those of you who are not technology people: We always run with two databases in tandem - a master and a backup, in different geographically-located data centers, so that we can switch from one to the other in case our primary data center goes down or in case our primary database fails).
But our backup database failed on a Sunday as trading started and we couldn’t repair it without downtime in the master database. That meant we we had to make a decision: whether to run Gen3 AutoTrade service for the entire week with no backup (and risk problems in the unlikely chance of its failure), or whether to choose some limited downtime in order to fix the backup database. We chose the limited downtime since it was a Sunday night when trading volume is low. So all Gen3 AutoTrade service was offline between 1:23 AM and 3:30 AM. When we restored service at 3:30, AutoSync kicked in and re-synchronized all customer broker accounts with the C2 trading systems they are subscribed to.
I am sorry for this trouble. We try hard to avoid downtime, but we made the decision to have a limited service interruption last night in order to insure that we could run the service this week with a backup database in place so we have a margin of safety in case our primary data center goes down.
- Francis Gingras
Trade Integration LLC
(the guys who run the Gen3 AutoTrade service)
In the future, for critical matters like this, you really should send a blast email to all C2 users ahead of time. This will serve the purpose of preventing any panic and to allowing the users to manage their accounts while they are unattended by Gen3.
Ok. Thank you for taking care of the issue. Now I want to know what is done to take care of the customer who suffer losses due to the trade integration downtime?
Thanks,
Julio