Posts Tagged ‘Incident Report’

Emergency Service window: Restart of internal nameservers..

Posted on: July 23rd, 2026 by

As a follow up on the previous mentioned internal DNS outage, a vulnerability was found in the existing configuration. To mitigate, and deploy an update, the operations team need to restart the internal DNS servers. This will happen:

Friday 24th of July, 2026 – 07:00 UTC.

The operation will last for a very short period (minutes). Most systems have a cache for DNS, and it is expected that very few users will notice any issues. Never the less, if you have the possibility, please limit your large operations between 06:55 UTC and 07:10 that day.

We will of course update this post afterwards, when the operation is done and everything is back to normal.

Tags: , ,

Incident report: Nameserver issues..

Posted on: July 23rd, 2026 by

This morning, on Thursday July 23, 2026, at about 06:30 UTC, a problem was triggering the internal nameservers in our infrastructure to stop servicing. This resulted in a domino effect of other internal and external services failing and falling over.

Users would have seen that the Kolab Now web page, this blog and the knowledge base was all available, but the access to data (such as the Webmail client and IMAP) was giving errors like “Proxy error – service not available” or “Access to storage server failed”..

The issue was discovered right away, and the activity state was restored. We are now investigating what caused the original interruption of the nameservers.

At this time, all systems are up and running again. The incident lasted about an hour.

We apologize for any interruption or inconvenience that this may have caused.

 

Update 2026-07-23 @ 15:02; To mitigate the issue that was showing up this morning, the operations team need to restart the internal nameservers. Please check this blog post about the mitigation.

Tags: ,

SSO or not SSO – That is the question..

Posted on: March 31st, 2026 by

Before the migration to the new Kolab Now environment, users had to give their login credentials twice to get into the Kolab webmail client. First at https://kolabnow.com/login and later after pressing the Webmail button. You could also make use of the direct URL https://kolabnow.com/apps – in all instances the credentials would be validated against the same LDAP internally.

Many users found that a cumbersome procedure, and the number of support tickets suggesting to do a more modern authentication was high. The Kolab team took a little time to evaluate such solutions.

SSO was finally implemented in the latest new environment. With the new SSO solution, the Webmail client can use an OAuth token instead of credentials, further improving security.

The new SSO authentication system is created and run by the Kolab Now team using the standard OIDC protocol. It is only the cockpit and the Webmail client authenticating against this SSO. No 3’rd party provider is involved. Access from external clients is separate from the SSO authentication.

We had expected from the previous user feedback, that this would be received well. However, it turned out, that many users were not found of, or for one or the other reason unable to use, SSO.

Hence we have made a change:

> Continue Reading

Tags: , , , , , , , ,

Post Mortem on a migration

Posted on: March 28th, 2026 by

The dust is settling..

The migration happened on Tuesday, and was for a large part successful. No data was harmed during the event. All data was migrated. Kolab Now is now running on hardware that has guaranteed warranty for at least the next 5 years, and on a software platform that is making it easier for us to develop and maintain.

We did however run into a very unfortunate situation. A few very urgent matters caused our support mailbox to be quite full, and on top of that, the support ticket form at ‘https://kolabnow.com/support’ was out of order for a few hours. This resulted in response times much longer than we like them to be.

A few issues impacted users:

> Continue Reading

Tags: , , , , , , , , , , , ,

Migration postponed for 24 hours..

Posted on: March 23rd, 2026 by

As you can read in the update of the previous post on this blog, we ran into some unforeseen DNS issues during the migration, and were able to stop the actual migration in time. We found what we believe is the cause of the issues, and made changes to the configurations that we need to test.

At  the time of writing, we have postpone the migration until:

Tuesday March 24, 2026 @ 13:00 CET.

As it was the plan today, at 13:00 CET we will switch over the database and the main DNS.

This will cause downtime that can last up to 20 – 30 minutes.

We sincerely apologize for the inconvenience

Update 2026-03-24 @ 13:00 CET: The DNS issues from yesterday is under control. We are doing it. The migration is in progress.

Update 2026-03-24 @ 14:20 CET: The database has been migrated to the new environment. we are now updating the DNS. When DNS is updated, we can just wait for the global update. depending on where you are on the globe, this can take anywhere from a few minutes to 72 hours (just to mention the extremes). 

Update 2026-03-24 @ 15:30 CET: The migration has finished and all data is live on the new environment. We however hit a few issues; one certificate issue, one webclient defect, and DNS is slow to propagate in some areas (we know from user feedback). These issues should be resolved by now. At this time we are investigating some delays, but mail is flowing and getting delivered. Most of the issues are coming down to “restarting a browser or a Thunderbird”, but we resolve the tickets in Support as they come in. If you have issues that you do not read about here, then please contact Support.

 

Tags: , , ,

Incident report: Spam attack.. [Updated]

Posted on: March 6th, 2026 by

On Tuesday 2026-03-03, a number of user accounts on Kolab Now was used to send out large amounts of spam. The spammers didn’t get to send out much mail. Most of it was stopped in the Kolab Now exit filtering, but the spam rating of the mails going out unfortunately was high and caused Microsoft online services, specifically outlook.com, hotmail.com, to add two of the Kolab Now exit IP addresses to their block lists. This caused emails sent to recipients on these services to bounce with messages like:

550 5.7.1 Unfortunately, messages from [212.103.80.154] weren’t sent.

Please contact your Internet service provider since part of their network is on our block list (S3150).

The spammers were identified and stopped, but the damage was done.

> Continue Reading

Tags: , , , ,

Incident report: IMAP client service temporarily unstable..

Posted on: November 20th, 2025 by

Last night at 21:50 UTC a large number of client connections hit Kolab Now, which resulted in the IMAP client servers (‘imap.kolabnow.com’) running out of memory. Staff was noted about the issue at 06:00 UTC, and immediately raised the amount of memory for these servers, which brought the issue to an end.

During these ~8 hours users of IMAP clients had issues with unstable connections; some might even have been completely unable to connect.

Mail transport was not impacted. all mails were streaming in and out as they should, and also the webclient (‘https://kolabnow.com/apps’) worked well during the period.

The issue was the unusually large number of incoming client connections. To mitigate the situation and avoid similar issues, the staff has enlarged the memory size on the servers in question, and are now working to improve the monitoring of the service.

We regret the incident and apologize for any inconvenience that this may have caused.

 

Tags: ,

Incident report: 50% of ActiveSync connections failed..

Posted on: June 17th, 2025 by

On Sunday 2025-06-15 one of the Kolab now ActiveSync servers were getting overloaded and got stuck on failed jobs. In turn, about 50% of the ActiveSync jobs were failing – leading to some ActiveSync users loosing the connections to Outlook or to their mobile devices.

> Continue Reading

Tags: ,

Incident report: Some mails to Microsoft online services was getting blocked..

Posted on: May 7th, 2025 by

Yesterday we dealt with a spammer/phisher who specifically targeted the Microsoft outlook.com service (and it’s affiliates).  One of the Kolab Now MX servers was listed on the Microsoft throttle list (S3150). This meant that some users saw, that mails sent to recipients at ‘@outlook.com’, ‘@live.com’, ‘@hotmail.com’, and other Microsoft online services was bounced back.

> Continue Reading

Tags: , , ,

Incident report: Some mails to Microsoft online services was getting blocked..

Posted on: February 13th, 2025 by

This afternoon earlier today one of the Kolab Now MX servers was listed on the Microsoft block list. This means that some users might have seen, that mails sent to recipients at ‘@outlook.com’, ‘@live.com’, ‘@hotmail.com’, and other Microsoft online services was bounced back with the message that looks something like this:

This is the mail system at host mx.kolabnow.com. 
I'm sorry to have to inform you that your message could not 
be delivered to one or more recipients. It's attached below. 
For further assistance, please send mail to postmaster. 
If you do so, please include this problem report. You can 
delete your own text from the attached returned message. 
The mail system <some-email@outlook.com>: host  
outlook-com.olc.protection.outlook.com[x.x.x.x] said: 550 5.7.1 
Unfortunately, messages from [y.y.y.y] weren't sent. Please contact 
your Internet service provider since part of their network is on our block 
list (S3150). You can also refer your provider to 
http://mail.live.com/mail/troubleshooting.aspx#errors. [Name=Protocol 
Filter Agent][AGT=PFA][MxId=<some long number>] 
[SG2PEPF03345FBECA.apcprd05.prod.outlook.com 2025-02-13T<timestamp>Z 
<another long number>] (in reply to MAIL FROM command)

Although the listing was fast discovered, Microsoft was contacted and the listing is reversed as soon as it is possible, it took a while. At this time emails should be delivered to the Microsoft online services.

A few users has misinterpreted the symptoms with error messages from missing the DKIM changes made on Monday (please read this blog post from December 2024 and the follow ups). If you are a group manager, then please make sure that you have the new DKIM related CNAMES added to your DNS zone.

If you have any questions or concerns in this context, then please contact support.

Tags: , , , , , , ,