Regarding the Past Couple Days
Dear Photoblog.com Members,
At 4:55pm Saturday evening, the datacenter in which Photoblog's servers are located experienced a transformer explosion that knocked down three walls of the electrical equipment room. It immediately took down 9000 servers, including ours, and left 7500 customers' servers without power. Luckily, the servers were well protected and there was no data lost at all. While the datacenter is a world class datacenter with countless measures to prevent outages, what happened on Saturday was much more serious than datacenters are typically prepared for (the fire department would not even let them start the backup generators until it was deemed safe).
What was worse was that communication between the datacenter company and it's customers (including us) was very poor for the first couple days, and they were not able to provide us with a clear estimation as to when we would have power again. Since our main server was powerless, there was no way we could access the databases to contact our members regarding this situation. A possible way we could have informed the Photoblog community about what happened was if we changed the Photoblog.com domain name to temporarily point to another server or website. However, this process can take up to a day to fully propagate to everyone on the Internet, and would have guaranteed a significant amount of downtime while the process was carried out. Since we felt there was a chance power would be restored soon, we chose to avoid the redirection issues at a cost of no communication for as long as it took for the server power to be restored, since this would allow all of you to immediately use Photoblog again when power is restored.
Sadly, the time it took to get power back to the server was much longer than we had hoped, and the datacenter company repeatedly gave us short term estimates that never happened. We're truly sorry for the lack of communication on our part during this whole crisis, and wish we could have done more to communicate with all of you with the information we were given.
Here are a few links to news articles regarding the datacenter disaster:
Explosion At ThePlanet Datacenter Drops 9000 Servers
Houston hosting provider The Planet chronicles weekend power outage and recovery process
Fire at The Planet takes down thousands of websites
The full chain of events can be viewed on the status updates page set up by the datacenter:
H1 Data Center Updates
While Photoblog is back online and running smoothly after the major outage, there is still a slight chance of intermittent downtime for the next week or so, while the datacenter is working on getting everything permanently fixed. Any downtime from here forward should be minimal, but we'll keep you informed. Right now, we are told that there will be a downtime of about 4 hours in the middle of the night when traffic is lowest either this weekend or next weekend. This final downtime will ensure a permanent fix and should be the last (but hopefully only) downtime experienced.
We just received good news regarding the location of our servers in the datacenter. Turns out we probably won't be experiencing any more downtime!
Thank you all for your patience and kindness towards us while we were waiting for more information regarding what happened, and while we were waiting for power to be restored. We hope to continue to provide the best free service photoblogging service available on the Internet, and we're thankful you're all along for the ride.
Sincerely,
Michael Zhang
Photoblog.com
