This morning, we had an extended outage of our web server or several hours. The server became unresponsive early in the morning EST and was finally recovered by 9:00AM EST. The issue is related to an ongoing issue we are facing with the web server that causes it to become unresponsive. We have worked to narrow down the issue, and felt it was addressed in an update last week, but it appears we were wrong. We had hope this issue was behind us, so we let our guard down a bit. Our overseas team was on holiday today, so that compounded the issue by not having anyone available to restore services as quickly as normal. That is our fault entirely.
We are still working to find out any outstanding issues with our server machine. It is a bit complicated because the server is a virtualized instance of a Windows Server running on Amazon’s EC2 platform- more commonly referred to as "the cloud". Working in this environment introduces its own challenges when trying to fix issues like his, but we are doing our best. If we can't track the issue in the next week, we will begin building a new server instance using a more recent version of the OS from scratch. We may have to go this route anyway. The issue with this is that our site has a large number of custom server components, so they have o be carefully configured. Swapping the servers out could cause issues if we don't have everything setup perfectly.
Please rest assured we are working hard to resolve the issues and we sincerely apologize for any inconvenience.