Incident Report: Network Interruption

At 06:00 UTC on Wednesday May 26th Kolab Now fell silent. All connections were dropped. What happened?

The internet provider for Kolab Now (https://nts.ch/) had planned changes to some configuration in the datacenter on that Wednesday morning. These changes should not have had any impact on the Kolab Now connections or systems, and since it would have no impact, the Kolab Now staff were not made aware of any such changes.

The changes were executed starting at 06:00 UTC. As the Kolab Now staff make use of our own services, it was discovered right away that there was a problem and that we had no internet.

We immediately contacted NTS. They informed us about the ongoing configuration change, and investigated right away. Around 06:15 UTC they informed us that they had located the faulty configuration and that they were going to fix it. At 06:30 UTC we could confirm the internet connectivity was fully restored. All systems showed available right away, and operations could announce that all was OK.

The cause of the issue was a human error outside of our control. The incident lasted less than 30 minutes. Where as we are fully aware of the criticality of email for our users, we do what we can to avoid unplanned downtime. We had conversations with our provider regarding the incident and they assure us of the future stability in delivery.

We apologize for any inconvenience that this may have caused.