On Monday, August 23rd 2021 at around 04:00 UTC, Kolab Now suffered a catastrophic outage related to its storage. This post will outline what happened, when it happened, and what is to happen.
Announcing Service Window: Improving Authentication
After much work we are now ready to switch over to a new authentication system based on laravel passport. The switch over will happen on Saturday, August 21st starting at 09:00 UTC. Short periods of downtime might occur, but we will of course keep it to a minimum.
Users can follow the status of the update via the Kolab Ops Twitter account. They can keep working and collaborating online as usual. Should you have any issues in this or any other context, then please contact Kolab Now Support.
(re-)Introducing Bank payments..
On Saturday, 31st of July 2021 we added a few features to Kolab Now during a service window. We already informed you about two of the features that were added; the user controlled Sender Preferred Framework (SPF) for group admin users, and the user controlled opt out of the greylist for users. What we failed to update you on was another added ‘new’ payment option: Bank transfers.
Announcing Service Window: Updates to Kolab..
This Saturday, 31st of July 2021 at 08:00 UTC, the Kolab Now operations team will perform an update of the Kolab dashboard software on the Kolab Now platform.
Announcing Service Window: Updates to Kolab..
This Saturday, 19th of June 2021 at 09:00 UTC, the Kolab Now operations team will perform an update of the Kolab software on the Kolab Now platform. The update will add a few features for our support and admin personnel, which are hidden from normal users. Of course we also take the opportunity to fix a line of minor defects with the same update.
The service window is expected to last for no more than 1 hour, ending on Saturday, 19th of June 2021, at 10:00 UTC.
Incident Report: IMAP backend server out of memory
On Monday night, the 7th of June 2021 @ 18:29 UTC, a process on one of our many backend servers was taking a lot of memory; faster than it could release it again, which made the server run out of memory and stop responding. The server were serving IMAP for a limited group of users, who in turn were impacted by the incident
As the server was heavily monitored, alarms were going off with the staff, but was little noticed, as it happened in a low traffic period, and as the systems are build to fix such situations themselves. The systems in this class usually just restore and keep working. This time however, the server did not come up in a timely manner, and the mentioned users saw their mailboxes freeze and their mail become unavailable.
The staff realized that something was not as it should be and went on to manually restore the situation. At 20:27 UTC the server was back up and running, and all mailboxes were available again.
No data was in danger, and mail delivery to the impacted mailboxes continued during the incident.
We apologize for any inconvenience that this incident may have caused.
Announcing Service Window: Update of firmware on Storage
This Saturday, 5th of June 2021 at 09:00 UTC, the Kolab Now operations team is performing an update of the firmware on the storage canisters in the datacenter. As Kolab Now is running the operations on high class professional IBM servers under great warranty (see the knowledge base article), the update will be performed under advice from the IBM technical support crew.
As the storage system is 100% redundant, we expect that there will be no interruption or downtime – planned or unplanned – in connection to this change. Users can follow the status of the change via the Kolab Ops Twitter account. They can keep working and collaborating online as usual.
Should you have any issues in this or any other context, then please contact Kolab Now Support
UPDATE 2021-05-05 @ 11:00 UTC: The update has happened, and as expected, there was no downtime. All systems are back up and running as they should be.
Incident Report: Network Interruption
At 06:00 UTC on Wednesday May 26th Kolab Now fell silent. All connections were dropped. What happened?
Kolab Meet beta – The story so far..
Earlier this year we wrote here on this blog about our new added feature: ‘Kolab Meet’ The voice & video conferencing tool. We explained how this tool was still in beta, but that all users of kolab Now (with full groupware subscriptions) could gain access to the tool, and how it was used.
Now it has been a while, and we have made a few updates to the tool. It is still in beta, but we have added features and fixed some defects and corrected some inconveniences. here are a few comments on experiences with the latest iteration of Kolab Meet (listed in no particular order):
Incident Report: Lock down of firewalls
At 10:10 UTC this morning, Wednesday April 28th, parts of our environment were getting updated. The updates included Security Enhanced Linux configuration on one layer of our firewalls. Unfortunately this new configuration locked up these firewalls and all traffic was blocked for a group of users.
The problem was confirmed corrected at 10:36 UTC.
No data was in any danger of being compromised during the incident.
We apologize for the inconvenience.
