We just wanted to post a quick update regarding the status of our primary email server. At this point in time the server is back online, however it is operating at a reduced capacity:
Issue:
Corrupt filesystem on one of the mirrored back up drives, which forced a reboot of the server, prevented the server from continuing boot sequence, and caused a disruption in service.
Resolution:
In order to fully restore service, the filesystem integrity check of the corrupt drive needs to successfully run and repair local errors on the disk. This is currently running, and we are hopeful that it will complete soon so that we can restore service to our remaining customers.
Workaround:
In order to offer some form of relief to all of our customers without access, we’ve booted the server without the corrupt drive (this means that any customer data residing on the corrupt drive won’t be accessible until the filesystem check completes). This will at least allow webmail, managemail, and email services to function for about half of our customers.
If you are still unable to access your email at this time, please be patient and continue to wait while we work on restoring your service. We understand the magnitude of the inconvenience this has caused many of you, and we will do whatever we can to make this up to you.
Thank you again for your understanding, and we hope to have everything working soon.
Smile Support Staff
UPDATE – 3:22AM: We have recovered the corrupt data from the storage array that went down yesterday, and setup a new storage array which is currently in the process of copying the data to the new location. This is about 70GB of data (comprised of 2 million+ files), so it is going to take a couple of hours to successfully transfer all of the data. However, the good news is that the server will remain online during this time and as soon as your mail is copied over from the old array it will start working.
If you are still without access, or are receiving errors relating to your username/password it is very likely that your data hasn’t finished copying over. So we ask that you continue to have patience and can assure you that things will be operational over the course of the next few hours.
Thank you,
Smile Support Staff
