Author Archive for Mads Petersen

ActiveSync broken by Windows update..

Posted on: February 8th, 2024 by

Since last weekend, Support received a few reports from users who saw problems with Outlook ActiveSync synchronization towards Kolab Now. No emails were synchronized to their inbox, and the content of the Outlook Inbox would not match the inbox in the Kolab Webclient.

After some investigation it was realized, that Microsoft had provided (and automatically deployed) a new update on February 1st, and that this update caused the faulty behavior. The new version was: 2401 (various build numbers).

Rolling back Outlook resolved the issue for all users and brought back the normal behavior, and outlook would again synchronize email to the inbox.

A short version of the downgrade procedure is:

  • Find the version to downgrade to at the Microsoft site.
  • Disable automatic updates via Start -> Settings -> Windows Updates
  • Close down Outlook.
  • Open an admin shell.
    • Run: C:\Program Files\Common Files\microsoft shared\ClickToRun\OfficeC2RClient.exe /update user updatetoversion=16.0.17029.20108
    • (change 16.0.17029.20108 if you want to roll back to an even older version).
  • Start Outlook again.

Please seek further information about rollback of outlook versions at the Microsoft support site.

2024-02-23 – Update: We had a few users asking when we will have a fix for this issue. To that we must refer to the Microsoft support article about the issue, which can be found here. In short they lightly suggest, that the patch will be released on 1st of April (We do not think that this is an April Fool’s joke ).

 

Incident report: One external submission server overwhelmed by spam flood..

Posted on: January 8th, 2024 by

On the 2024-01-07 a spammer made a large flow on one of the external submission servers. The server stopped the spam mails, and saved them to a separate holding queue to make room for other users.

It took a while for the reporting to get to the operations team, but as soon as the issue was known it was swiftly resolved at ~19:00. However, meanwhile the server ran out of space, and some users (who hit that server) would have seen that the send and receive activities failed.

We apologize for the inconvenience that this issue has caused, and will focus on improving the reporting to also cover this specific issue.

Cross-site Scripting (XSS) issue in the Kolab Webclient?

Posted on: December 5th, 2023 by

Lately we got a lot of questions at Support about a possible vulnerability in Roundcube, the Kolab Now webclient. The listing in question is  CVE-2023-47272 which upstream in the Roundcube development has been handled and patched.

To see the version of Roundcube running as the Kolab now webclient, login and press the ‘About’ button in the left side of the screen. This will reveal that Kolab Now is running:

Roundcube Webmail 1.5.6.3-30.1

This tells, that the fix was already installed, and that the Kolab Now webclient is not impacted by the vulnerability.

We are thankful for having an observant and active user base.

 

Incident report: Service not restarting automatically on failure

Posted on: September 19th, 2023 by

In the very late hours (UTC) of 2023-09-18 Some users experienced that they could not receive mails, or do administrative activities – like making payments or creating new users. When they tried they were given an error message: ‘Internal Server error’. This lasted until the morning (UTC) of 2023-09-19.

The issue was caused by the in-memory data store (Redis) going out of memory.

> Continue Reading

Incident report: “Stability!”

Posted on: September 1st, 2023 by

Welcome back!

We were finally able to bring our blog back. During the past 2 weeks we had a long lasting incident. We did what we could to make sure that users could use their primary services during our work on the systems, but we know that there was unplanned downtime. As the blog was a victim of the situation, we couldn’t inform users as well as we wanted (we really do not like X (‘formerly known as Twitter’)), so here is finally the incident report from the beginning to the end.

> Continue Reading

Service window postponed – Due diligence..

Posted on: June 23rd, 2023 by

Last week we announced a service window that was supposed to start and be in session about now. However, following due diligence we went through the list of activities last night and found that we had overseen an important step in the plan. The operations that are going to happen are on the backend servers, are quite sensitive, and we would like to avoid any problems. We therefore suspend the current service window and hereby declare it for the coming Friday:

Friday 2023-06-30 at 08:00 UTC until 13:00 UTC.

> Continue Reading

Announcing Service Window Friday 23rd of June: The everlasting race to get rid of old infrastructure..

Posted on: June 16th, 2023 by

The Kolab Now Staff is going to update parts of the platform to get rid of even more old infrastructure, and to ensure the best possible solution for users. Therefore we declare a service window on:

Friday 2023-06-23 at 08:00 UTC until 13:00 UTC.

> Continue Reading

Announcing Service Window: Caching infrastructure..

Posted on: April 17th, 2023 by

Lately we have had reports from users of the Kolab webclient who saw a variety of issues such as: login issues, some data entries being hidden and showing up again and folder creations failing. As this was seriously affecting the reliability of the webclient, we put the deployment of new features on hold to investigate the root cause of those issues.

> Continue Reading

Update on the ongoing performance issue

Posted on: February 2nd, 2023 by

During the last 2 days Kolab Now has experienced a serious performance degradation issue. Users see that logins to- and operations in the webclient is taking a long time, and users connecting via desktop or mobile clients might not be able to perform synchronizations.

In the end of Wednesday it was thought that the root cause was found, and many users saw better performance for a while, but Thursday morning the performance dropped back to unacceptable levels.

> Continue Reading

Announcing Service Window: More replacement of old infrastructure..

Posted on: January 20th, 2023 by

This Friday’s service window was being executed well, and while assisting in replacing some old infrastructure, it opened opportunities to proceed down the same track. With that in mind, the Kolab Now operations team will perform more maintenance on the infrastructure of the Kolab Now platform on Friday, 27th of January 2023 at 10:00 UTC.  The update will move more of backend services to newer infrastructure. No new features visible to end users will be enabled during this maintenance.

The service window is expected to last for no more than 4 hours, ending on Friday, 27th of January 2023, at 13:00 UTC.

> Continue Reading