Sorry for the delay in notifying you guys that it’s back up — we’ve been busy with other pressing matters.
Our site was down on Friday, October 2 from around 12pm EST to about 9pm EST. What happened? We host our server with the folks at www.mediatemple.com. They’ve got some really unique products that provide “burstable” architectures. Basically, instead of your own server, you get a “virtual” server that is part of a cluster of servers, and when traffic increases, you get more resources. When it decreases, you get fewer.
In our case, we had a burst of database usage and so they performed a temporary upgrade to our database to provide a higher level of service, then notified us of the change via email. Because this temporary upgrade requires a migration of the database from one area of the server cluster to another, there is a brief interruption of 5-10 minutes. Once usage drops back down, there is another migration back to the “normal” level of service. Since we wanted to insure that our web site users wouldn’t experience any future interruptions, we elected to make the upgrade permanent on the spot.
Unfortunately, because the server was already in the “migration” process, this additional request crashed the migration process, and the server got hung — requiring the intervention of Media Temple, which took 5-6 hours as they had other requests in front of ours.
Fortunately, no data was lost, emails worked normally during the outage, and the server is operating 100% normally at this time. Sorry for the downtime, folks — it’s unlikely to happen again!
-Micah