Author Archive for Mads Petersen

On Friday January 21 at 15:48 UTC  users observed that the main page https://kolabnow.com/ became unaccessible and reviled an error: 503 – Server unavailable. The webclient (https://kolabnow.com/apps) was still available, but with seriously degraded performance – to the point where the service would give up and reveal the ‘Maintenance’ page.

Also IMAP, SMTP, ActiveSync, and $DAV was impacted by the failing frontends.

The service was unavailable for about 50 minutes, during which time all mails, in and out, were queued. When the service again was available, those mail queues first had to be delivered before new mail could be sent or received. This caused users to experience a delay in the traffic.

> Continue Reading

On Monday morning at 03:22 CET the kolab Now $DAV services stopped working.

While resolving a performance issue for $DAV connections, we moved around  servers from an old infrastructure to a new infrastructure, and with that the rotation of logs. Unfortunately a tiny bit were missed while configuring the new servers, which caused the log rotation attempting to reload httpd (while that wasn’t needed). In turn, reloading httpd didn’t succeed and timed out, and the new DAV servers were left in a maintenance state;  unavailable.

This happened at ~ 03:22 CET. As these are new DAV servers, they were only included in the standard monitoring, and no one got notified about the problem until it was discovered at 07:47 CET. The configuration was promptly corrected and the new DAV servers came back online right away (and was added to the monitoring notifications). At this time everything should be available and working as expected.

We apologize for any inconvenience that this may have caused.

On Tuesday 2021-10-05, between 10:20 UTC and 10:40 UTC, a network issue kicked our firewalls off the grid, and Kolab Now was down. Our Operations team was on the case right away and could correct the issue instantly. As a result the downtime was very limited.

When the dust had settled, it turned out that one of our hypervisors had stopped one of its redundant storage blocks, as it was unavailable for too long, and a group ~ 20 users were unable to see their mails in the mail folders. Again our Operations team was reacting fast and could restore the connections to the redundant storage. Everything is back up and running for everyone now, and no data was in danger at any point through out the incident.

If you are one of the few users who had the problem after the down time, or if you experienced troubles during the outage around lunch time, then we truly apologize for the inconvenience that it caused.

If you have any questions or concerns in this context, please contact support.

On Monday night, the 7th of June 2021 @ 18:29 UTC, a process on one of our many backend servers was taking a lot of memory; faster than it could release it again, which made the server run out of memory and stop responding. The server were serving IMAP for a limited group of users, who in turn were impacted by the incident

As the server was heavily monitored, alarms were going off with the staff, but was little noticed, as it happened in a low traffic period, and as the systems are build to fix such situations themselves. The systems in this class usually just restore and keep working. This time however, the server did not come up in a timely manner, and the mentioned users saw their mailboxes freeze and their mail become unavailable.

The staff realized that something was not as it should be and went on to manually restore the situation. At 20:27 UTC the server was back up and running, and all mailboxes were available again.

No data was in danger, and mail delivery to the impacted mailboxes continued during the incident.

We apologize for any inconvenience that this incident may have caused.

At 06:00 UTC on Wednesday May 26th Kolab Now fell silent. All connections were dropped. What happened?

> Continue Reading