Incident report: Authentication overload

Today at ~14:45 one of the imap frontends failed in a way that was not detected by the load-balancer, which kept it in rotation and resulted in widespread but intermittend errors across services including Webmail, DAV, ActiveSync and IMAP.

Initial investigation suggests that this failure has been triggered by malicious activity (a large amount of authentication attempts in a short amount of time), which while unsuccessful triggered this failure which unfortunately did not recover automatically.

Services are now restored and further investigation is ongoing in how to make our systems more resilient to the same kind of failure going forward.

We apologize for the inconvenience caused.