Last night, the main database crashed.
It appears that some recently added trading systems were lost. We have a replication server, which is designed to act as a hot backup… but of course it is offline at this moment, about to be shipped to a geographically remote facility (so that we can be prepared in case our main data center has some sort of disaster or attack).
It may be possible to reconstruct the lost systems from the binary server logs, using the replication server. I will know whether this is possible, and perhaps even succeed, within a few hours. All I can ask is that you try your best to stay patient while I work on this course of action. I will keep you informed here of progress.
Mathew you are a busy man this morning. Target 50 went long and covered its short on the open. Not showing yet. No hurry. Thanks
I can access the site through my browser, but TradeBullet says it is "unable to connect to remote server"
Let me take care of that. Hang on
seems to work fine now, thanks!
Given the presumed severity of the problem, I believe you did a good job in dealing with it (or probably more important, with previous preventive measures). From the post of ST I infer that the essential funtions of the site were repaired within one hour. I only noticed the problem via some minor forum malfunction.
As always, Matthew was on top of the problem and fixed it–given how severe it was–quickly. I also appreciated the notification box at the top of the page.
Thanks, guys. We were lucky to have set up some basic replication in the weeks before. As I mentioned earlier, I am also in the process of setting up multiple colocation facilities, in different geographic areas, in case one facility should suffer catastrophic failure. This should be complete within a few weeks.
Charts for some systems (including mine - Bounce Systems) have not updated based on Friday’s market and account action.
Yes. All charts should get caught up during the course of today (Monday). If midnight Eastern time rolls around and your chart is out of date, email me. Once again, sorry for the trouble.
