Update on the ongoing performance issue
During the last 2 days Kolab Now has experienced a serious performance degradation issue. Users see that logins to- and operations in the webclient is taking a long time, and users connecting via desktop or mobile clients might not be able to perform synchronizations.
In the end of Wednesday it was thought that the root cause was found, and many users saw better performance for a while, but Thursday morning the performance dropped back to unacceptable levels.
The issue is a sharp and unexpected increase in usage of a certian part of the backend systems as a follow of Last Friday’s migration of mailboxes on to the new infrastructure. At this time the Kolab Now Operations and systems engineering teams are working to increase the resources for that part of the systems. As this work is urgent, users might experience even worse performance and timeouts while teh work is ongoing.
We are aware that our services are business critical for many of our users, and we do what we can to get the performance back to acceptable levels within as short time as possible. We regret the inconvenience and troubles that this is causing.
No data is in danger at any point during this work. The issue is with performance. Operations are taking a long time (and in worst case timing out).
Updates to this blogpost will follow as new information becomes available.
[Update 2023-02-02@17:54 CET]: As work is proceeding, systems are getting moved around and migrated. This makes the accessibility of teh systems even slower.
[Update 2023-02-02@18:10 CET]: At this time all IMAP backends are getting restarted to pick up new configuration. This will break all IMAP connections and logins to the webclient.
[Update 2023-02-02@22:01 CET]: A version of a specific service tool on the new environment (mupdate) has shown to act up under pressure. The tool was downgraded, which brought the authentication services back to normal. The downgrade however lead to some slip stream issues. Many services had to be restarted. This caused the service to be unstable and unavailable for about 60% of users.
At this time the webclient and email is working and has an acceptable performance, but there are still issues (as mentioned above) which Operations are investigating and fixing one by one as we type this. The most visible issue is, that groupware folders (like ‘Calendar’, ‘Files’, Contacts, etc) are sporadically visible to users of the webclient at some logins. Users are encouraged to just leave those folders alone. The issue will be investigated and fixed as fast as it is possible.
[Update 2023-02-03@06:21 CET]: This morning the performance of the webclient and IMAP front and backends are back to normal. We will keep monitoring the environment very closely. This should be the last update to this post. The incident report will follow later; some lines of events still needs some clarification.