At 10:23 UTC this morning, Wednesday January 29th, our environment experienced a catastrophic storage failure. The time to resolution for this underlying problem was approximately 80 minutes, and full service was restored approximately 60 minutes thereafter — 12:48 UTC.
Posts Tagged ‘Incident Report’
Incident Report: Cascading Performance Problems
From last Sunday afternoon onward, up to Monday evening and throughout the Monday night, performance problems have deteriorated the Kolab Now service up to and including services becoming unavailable.
Incident Report: Thursday, 20:25 – Friday 01:15 UTC
Last night, a failure in the storage layer caused most of our services to be unavailable. In the week before, we replaced a failed hard drive. In the week before that, a so-called Virtual Fabric Adapter failed, causing a hypervisor to shut itself off. Since the most recent incident was the more serious downtime, that’s what we’ll start our reporting on.
Incident Report: DNSSEC record expired
On Saturday morning (CET) the DNSSEC records expired on one of our DNS servers. This caused a group of customers to have troubles logging in and connecting to Kolab Now services. The record has been renewed and all customers should have access (at 22:49 CET – Please read below).
Incident Report: Hypervisor Failure
This weekend, at approximately 12:00 UTC on Sunday, an issue on one of the hypervisors went by unnoticed for too long, and was finally resolved in the morning of Monday. This post explains what happened, why it happened, and what we’re going to be doing to address the situation.
Incident Report: Backend Down
Earlier this morning, at 04:38 UTC, one out of the twenty-two IMAP backends in production stopped serving its mail spool, showing Input/Output errors on its disk. Our Standard Operating Procedure is to examine log files, flush vm caches, stop the virtual machine, and start it back up again. This occurred at 05:48 UTC. The IMAP backend in question did not come back up cleanly.
