Announcing Service Windows: Reboot Weekend

As may have already been brought to your attention, some software mitigation is needed for vulnerabilities dubbed “Meltdown” (CVE-2017-5754) and “Spectre” (CVE-2017-5753, CVE-2017-5715).

If I were to summarize the issue with these vulnerabilities then in principle they would, when successfully exploited, allow reading memory that doesn’t belong to the process, the user or even the same operating system instance. In just that way, the Kolab Now infrastructure isn’t impacted.

However, we’re still going to need to patch this out. The only way we can is by updating software and rebooting systems, and this will happen during the weekend of Saturday January 13th and Sunday January 14th.

The Kolab Now deployment runs on a single-tenant infrastructure, meaning that no third parties have any privileges, not even POSIX shell user privileges. This is an important distinction to draw, in that it reduces the attack surface to local users only. In contrast, if a Kolab deployment runs in one of those public cloud services, then third-party access to guests is outside anyone’s control.

However, security isn’t security if you just polish the surface — so let us review what the impact might be, provided a set of circumstances are assumed to fall in to place;

In principle, anything that can execute anything — regardless of the privileges with which it happens, and regardless of whether it happens on a guest or a physical piece of hardware — could potentially get access to sections of memory that contain privileged information. Such memory could contain the in-memory buffer caches, query caches, application memory, private keys, etc.

With our entire software stack being Free Software, we can ascertain (to a reasonable level of confidence) no piece of software currently installed exploits the Meltdown nor Spectre vulnerabilities, so the attack surface is reduced to uploading code and getting it to execute somehow. Our software is typically not vulnerable to such vectors, and nor are our systems (Security Enhanced Linux helps). But we can’t stop at assuming there will be no such vulnerability anywhere in the current software stack and ignore an underlying vulnerability. We need to ensure no future vulnerability on one level exploits an unpatched, passed security vulnerability lower down in the stack.

So, here’s what we’re going to need to do; Each of the systems we operate is going to need to get updated and rebooted. Most critically, this involves a few single points of failure — hypervisors, IMAP backends. For the most part., each of our users should not notice anything more then a few minutes of service interruption.

5 thoughts on “Announcing Service Windows: Reboot Weekend

  1. Others have reported there may be some performance hits depending on the type of workload being run, I’d be interested in hearing your real world experience post patch.

  2. Realistically, you ought to be able to ‘cycle’ infrastructure in and out of production, so that whilst the production infrastructure is updating, the backup infrastructure is running in production, and the backup infrastructure updating whilst the production infrastructure remains in production.

    This does beg the question, do you have backup infrastruture, as it should be said – politely, that Kolab Now has had one too many downtimes more than I wuold like.

    1. This is what will happen for 95% of the infrastructure — take nodes out of rotation and cycle them, or fall back on to backup infrastructure while primary gets updated — and you won’t be noticing a thing.

  3. Why isn’t it possible for infrastructure to be cycled in and out of production, such that the backup infrastructure is ‘cycled’ into production whilst the production is being updated, and vice versa – or does this perhaps expose a fundamental flaw of Kolab Now, in that it does not have backup infrastructure, which would perhaps suitably explain all the downtime which has occured.

Comments are closed.