Author Archive for Jeroen van Meeuwen

Incident report: Misconfigured traffic shaping..

Posted on: November 6th, 2026 by

After yesterdays smooth upgrade of our uplinks bandwidth, we have suffered an outage today, November 6th, between 13:34 and 14:03 UTC.

Kolab Now employs traffic control (among other measures) to ensure no few parties can eat all of the pie. While the pie was of one size before the upgrade, after the upgrade the pie is much larger. This means there was a need to change the traffic control (alongside other measures) and that’s where this incident started.

While Traffic Control is a scheduling discipline also known as traffic shaping, other measures at Kolab Now include Hybrid IDS/IPS, rate-limitation policies and high availability proxy. Not all of them are to be treated equally across a set of firewall layers (and the DMZs in between), but transparent proxying allows for proper analysis of the connection further inside the infrastructure by logic best delegated to the proper solution.

To illustrate; before you get to send a new email, you traverse 3 firewalls. The first is an external firewall, and relatively stupid. The second is an internal firewall, and somewhat smart. The third is a load-balancer (through which we achieve high-availability) that can apply policy. Now you’re connected to what we consider a public-facing backend application. The external, internal and load-balancer firewalls are all redundant.

The incident thus unfolded as follows:

  • an increase in available bandwidth to/from the world is purchased and applied (very smoothly)
  • we seek to alter the mitigations we’ve had in place to ensure everyone had their fair slice of the pie
  • a mistake is made because the stack is intricate and complex
  • we recover in ~30 minutes

The symptoms we’ve reviewed thus-far during the actual outage include not dysfunction (of the stack), but an inability to properly control and oust abusers before they get in (y)our way.

What/how/where/when and in combination with what, a mistake exactly caused the outage remains to be determined, because like you, we also wish to learn more about it. We will spend some extra time on attribution, because that would be fun, wouldn’t it?

We apologize for any inconvenience that you may have experienced.

Incident Report: Various Types of Failure Symptoms

Posted on: March 3rd, 2023 by

On Friday March 3rd, 2023, from approximately 01:00 UTC to approximately 14:40 UTC, numerous users of our service experienced a variety of issues.

A complete listing of the symptoms experienced is infeasible at this time, but logins will have failed to the web-client interface, the cockpit interface, and people will have not received emails in time.

> Continue Reading

Incident Report: Storage Outage at Kolab Now

Posted on: August 25th, 2021 by

On Monday, August 23rd 2021 at around 04:00 UTC, Kolab Now suffered a catastrophic outage related to its storage. This post will outline what happened, when it happened, and what is to happen.

> Continue Reading

Our Statistics for 2020

Posted on: January 15th, 2021 by

Our terms of service state there’s basically no way for anyone to get any access to your data without us also being able to talk about the fact it happened.

Like the last time we declared having made a full eclipse around the sun at an arbitrary date, here’s our statistics for 2020.

> Continue Reading

The Future of Kolab Now: Part I

Posted on: July 7th, 2020 by

This is an introduction to the Kolab Now roadmap, and what the future is going to bring. I’ll give you a high-level overview of our future plans.

It’ll constitute the first in a series of blog posts, and different people will speak in more detail about individual topics.

> Continue Reading

How Kolab Now Deals with SARS-Cov-2 (aka. Covid-19, Corona Virus)

Posted on: April 16th, 2020 by

There’s been four distinct aspects of the current pandemic we’ve needed to deal with, and I’ll attempt to state where they come from, and what the consequences are or have been or will be.

> Continue Reading

Stay Safe at Home with Kolab Now

Posted on: March 16th, 2020 by

Authorities and experts alike tell us physically staying at home is the safest thing to do for you and yours, and everyone else. Our experts want you to enjoy the same safety on the web.

We have decided to spread the burden and not the virus, and do the right thing by not opportunistically profiting off of our collective misery. We now provide a 100% discount to all new signups for as long as we can manage, or for as long as the crisis continues, whichever comes first. When things cool down, we’ll remove the discount.

> Continue Reading

Incident Report: Storage Failure

Posted on: January 29th, 2020 by

At 10:23 UTC this morning, Wednesday January 29th, our environment experienced a catastrophic storage failure. The time to resolution for this underlying problem was approximately 80 minutes, and full service was restored approximately 60 minutes thereafter — 12:48 UTC.

> Continue Reading

Our Annual Certificate Refresh

Posted on: January 16th, 2020 by

Our annual certificate refresh is upon us.

> Continue Reading

Our statistics for 2019

Posted on: January 3rd, 2020 by

Our terms of service state there’s basically no way for anyone to get any access to your data without us also being able to talk about the fact it happened, and further down nested in our legal framework outline do we have a list of 3 general types under which individual requests could be filed.

Like the last time we declared having made a full eclipse around the sun at an arbitrary date, here’s our statistics for 2019.

> Continue Reading