Incident Report: Double trouble..

Late Sunday night at 21:31 CET we were alarmed by lost connections. Symptoms seemed to point to a networking issue in the datacenter. Our datacenter provider was alerted, we sent our own emergency team onsite to assist in resolving the issue, and to deal with any fallout.

A faulty firewall was recovered, and mail started flowing again at around 00:10. Our ops team is now investigating why the firewalls didn’t fail over as they are configured to do in such situation.

At about 04:15, long after the system had recovered from the firewall issue, one system started to throw disk errors, which was leading to some users not being able to access the data in their folders. As this was a separate issue, the symptoms were only discovered at around 06:45, as it was assumed to be recovery issues from the previous outage. As soon as the disk error issue was realized, it was fixed and recovered. Connection to data was available again for all users at 07:10.

Where as we are still investigating the root cause for both of these issues, we can state, that no data was at risk at any time during the events.

We apologize for any inconvenience this has caused.