After yesterdays smooth upgrade of our uplinks bandwidth, we have suffered an outage today, November 6th, between 13:34 and 14:03 UTC.
Kolab Now employs traffic control (among other measures) to ensure no few parties can eat all of the pie. While the pie was of one size before the upgrade, after the upgrade the pie is much larger. This means there was a need to change the traffic control (alongside other measures) and that’s where this incident started.
While Traffic Control is a scheduling discipline also known as traffic shaping, other measures at Kolab Now include Hybrid IDS/IPS, rate-limitation policies and high availability proxy. Not all of them are to be treated equally across a set of firewall layers (and the DMZs in between), but transparent proxying allows for proper analysis of the connection further inside the infrastructure by logic best delegated to the proper solution.
To illustrate; before you get to send a new email, you traverse 3 firewalls. The first is an external firewall, and relatively stupid. The second is an internal firewall, and somewhat smart. The third is a load-balancer (through which we achieve high-availability) that can apply policy. Now you’re connected to what we consider a public-facing backend application. The external, internal and load-balancer firewalls are all redundant.
The incident thus unfolded as follows:
- an increase in available bandwidth to/from the world is purchased and applied (very smoothly)
- we seek to alter the mitigations we’ve had in place to ensure everyone had their fair slice of the pie
- a mistake is made because the stack is intricate and complex
- we recover in ~30 minutes
The symptoms we’ve reviewed thus-far during the actual outage include not dysfunction (of the stack), but an inability to properly control and oust abusers before they get in (y)our way.
What/how/where/when and in combination with what, a mistake exactly caused the outage remains to be determined, because like you, we also wish to learn more about it. We will spend some extra time on attribution, because that would be fun, wouldn’t it?
We apologize for any inconvenience that you may have experienced.

