Somehow the “bad week” of last week has extended into a 10 day marathon of problems. Hopefully the DoS attack of this morning will put a close to an extraordinary number of unrelated and unforeseen problems that we’ve gone through this week. In a nutshell, we’ve seen equipment failures at our data center, fiber outages, routing problems, service attacks, and all the normal day to day problems thrown in for good measure.
While it may appear that we “point fingers” at our data center partners, they have been outstanding in managing issues that at first glance appear to be completely unmanageable. See their letter to customers for more on what went on last week, and just a bit of what they did to resolve it. We have seen shipments of additional equipment, revised emergency plans, additional routing, and plans for a better information and alert system for everyone using the center.
We realize that all the explanations in the world don’t keep your business running. We’re sorry to see a couple of our customers go, and we understand their decisions. We honestly feel that we are seeing real results, and will continue to see improvements to the systems we rely on to bring you the service we provide.
I’ve had more than a couple of people ask why we don’t just move. It took 2 months of planning and a month and a half of execution to get the servers into the facility without noticeable downtime, and to be honest, by the time we had planned the move, we’d be leaving one of the most prepared centers around, because once they’ve fixed all the problems, and put solutions, upgrades, backups to the backups, etc. in place, things tend to be smooth for a long time.
The Smile team hasn’t been as available for support as we normally are, and we apologize. The friendly emails, and encouragement are particularly helpful when the chips are down, thank you to those who sent them along. We take your service very seriously, and many of us take it very personally. We all have been working with our providers to make sure problems are resolved as quickly as possible.
If you have any questions, please send them to us at our support or administration email addresses.
— Scott
