October 31, 2013 in Server Status by admin | Comments Off
10:45am - we were alerted by our monitoring system that our primary web hosting server (v2.smileglobal.com) was unresponsive and unreachable.
11:00am - we were connected to the internal network and remote management interface of the server. We follow our standard procedure to identify the cause of the outage and were able to discover the the nature of the problem.
11:10am - The immediate solution called for a quick power-cycle of the machine, followed by a modification to a scheduled maintenance script.
11:20am - The server was back online and fully operational.
We apologize for the inconvenience and appreciate your understanding during the brief outage earlier this hour.
We’ve been notified of a problem with a recent set of definition updates for our anti-malware/anti-virus system that has inadvertently caused some messages to be rejected by our filtering servers.
You may have experienced this issue or been informed by an outside party that your message was rejected, with an error similar to or the same as the following:
Your email was rejected because it contains the MBL_349876.UNOFFICIAL
We’ve taken the steps necessary to resolve the issue and ensure that mail is once again flowing through to our customers. This was an upstream issue caused by a bad set of definitions provided by our anti-malware/anti-virus distributor. We apologize for the inconvenience and will keep you informed of any further status changes or problems, should they occur.
Smile Support Team
We experienced several connectivity issues shortly after midnight today (June 27th, 2013), and began immediately working on evaluating the situation and working towards a solution.
By 8am we were able to have the server back online, along with our websites and all the customer sites that were hosted on the machine. We are still reviewing the logs on the server to determine the exact nature of the outage, and don’t have a definitive diagnosis at this point – however, early indicators are suggesting that it was a combination of a Distributed-denial-of-service (or DDOS) attack which started this whole thing off, and then a problem with the RAID array in the server itself, which prevented the machine from completing its boot sequence.
Thank you for your patience, and we are truly sorry for any inconvenience this may have caused you.
Smile Support Staff
We recently made a few changes to our Spam filtering service…
We have been notified by several customers that some mail has been incorrectly identified (false-positive) as spam or unwanted email. We are addressing those issues as soon as we are informed of the problem. If you a noticing legitimate email being marked as spam, please check out the following Knowledgebase article.
You may have noticed some of your emails were tagged with the following Subject: ***SPAM*** – [Smile Global Gateway Filtering Service] -
and then the original email subject was displayed after the tagged portion.
Unfortunately, one thing that we overlooked was that the length of the tagged subject was quite long and made it difficult to see what the actual message subject was (especially on mobile devices and webmail). This has been changed back to our standard: [SPAM] - message.
Our solution was to add a custom message header that is added to the source of each message received by our Gateway Filtering Service. Here is an example:
X-Spam-Technical-Details: [Smile Global Gateway Filtering Service] – Support information:
This message was scanned by: gateway1.smileglobal.com and received a score of: -95.9
This system will flag a message as spam if the score is greater than 5.
If you feel this was in error, please visit http://smileglobal.com/sender-rejected