Incident report: Some mails to Microsoft online services was getting blocked..

Posted on: February 13th, 2025 by

This afternoon earlier today one of the Kolab Now MX servers was listed on the Microsoft block list. This means that some users might have seen, that mails sent to recipients at ‘@outlook.com’, ‘@live.com’, ‘@hotmail.com’, and other Microsoft online services was bounced back with the message that looks something like this:

This is the mail system at host mx.kolabnow.com. 
I'm sorry to have to inform you that your message could not 
be delivered to one or more recipients. It's attached below. 
For further assistance, please send mail to postmaster. 
If you do so, please include this problem report. You can 
delete your own text from the attached returned message. 
The mail system <some-email@outlook.com>: host  
outlook-com.olc.protection.outlook.com[x.x.x.x] said: 550 5.7.1 
Unfortunately, messages from [y.y.y.y] weren't sent. Please contact 
your Internet service provider since part of their network is on our block 
list (S3150). You can also refer your provider to 
http://mail.live.com/mail/troubleshooting.aspx#errors. [Name=Protocol 
Filter Agent][AGT=PFA][MxId=<some long number>] 
[SG2PEPF03345FBECA.apcprd05.prod.outlook.com 2025-02-13T<timestamp>Z 
<another long number>] (in reply to MAIL FROM command)

Although the listing was fast discovered, Microsoft was contacted and the listing is reversed as soon as it is possible, it took a while. At this time emails should be delivered to the Microsoft online services.

A few users has misinterpreted the symptoms with error messages from missing the DKIM changes made on Monday (please read this blog post from December 2024 and the follow ups). If you are a group manager, then please make sure that you have the new DKIM related CNAMES added to your DNS zone.

If you have any questions or concerns in this context, then please contact support.

Tags: , , , , , , ,

Service Window: Change to DKIM-Signatures – domain alignment

Posted on: February 5th, 2025 by

On December 20’th 2024 we announced a change to the DKIM configuration on Kolab Now. The announcement described actions needed for users with private domains. We recommended that group managers (the owners of private domains) set the following CNAMEs (both of them) in the DNS of their private domain:

dkim1._domainkey CNAME dkim1._domainkey.kolabnow.com.
dkim2._domainkey CNAME dkim2._domainkey.kolabnow.com.

We also said in the announcement, that we would provide the actual update top the system in the end of January. Unfortunately the snow in the Swiss alps was so wonderful in the end of January, that our techies got a bit delayed.

However, it has now come so far that we announce a service window on:

Monday February 10’th, 2025 @ 08:00 UTC.  

The Service window will last for an hour, within which users might see a small bump in the performance.

If you are the owner of a private domain and you have not yet added the CNAME records to your domain, then your outgoing emails might be refused by the recipient servers after the change.

We will keep you up to date with the progress of the work via updates to this post.

 

– 2025-02-10 @ 07:59 UTC: The service window is now open and the work is in progress. As written, you should not see any big impact.

– 2025-02-10 @ 08:45 UTC: The change has been implemented, and the service window is done. The new domain alignment is tested and seems to be working as expected. If you have any problems with your private domain, then please contact support.

Tags: , ,

Service Window: Update of Roundcube to v 1.6

Posted on: January 20th, 2025 by

The Kolab Now development team has been working on updating Roundcube to version 1.6 on the platform for a while, and we are finally ready to push it. Therefore we hereby announce a service window for:

Wednesday 2025-01-22 @ 08:00 UTC (09:00 CEST)

The service window will last for an hour. Users can keep working, but there may be brief webmail interruptions during the period.

Users using the old ‘Larry’ skin might experience some short-comings and problems with that skin. Users with such issues can switch to use one of the two Kolab Now skins (Settings -> Preferences -> User interface -> Interface skin).

If you are one such user and you really want to make use of the old ‘Larry’ skin, then please contact support.

In fact, Please contact support with any questions or concerns in this context.

Action required for Group managers!: Change to DKIM-Signatures – domain alignment – Errata

Posted on: December 24th, 2024 by

Due to the perils of simultaneous wordpress editing a crucial error snuck into the previous blogpost:

The newly required DNS entries are:


dkim1._domainkey CNAME dkim1._domainkey.kolabnow.com.
dkim2._domainkey CNAME dkim2._domainkey.kolabnow.com.

and NOT as the previous blogpost claimed


dkim1 CNAME dkim1.kolabnow.com.
dkim2 CNAME dkim2.kolabnow.com.

The previous blogpost has been corrected meanwhile.

Apologies, and merry Christmas!

Action required for Group managers!: Change to DKIM-Signatures – domain alignment

Posted on: December 20th, 2024 by

Lately we have seen a few emails not being delivered to third parties and bounced emails with messages about failing DKIM signatures.

DKIM is a mechanism that allows a receiving party of emails to determine whether an email has indeed been sent by the party that is claimed to be the sender, thus protecting against forged sender email addresses. Kolab Now implemented DKIM signatures a long time ago, but so far we have always used the kolabnow.com domain as the sender domain, when sending an email from a custom domain. An example signature header would look like this (please note the ‘d= tag’):

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h=
content-type:content-type:content-transfer-encoding:message-id
:subject:subject:from:from:date:date:mime-version:received
:received:received; s=dkim20240523; t=1734354313; x=1736168714;
bh=mBUfOmuiUe6nDmAiAsHAHqpD0F+Gd9nJUF5Z5spFd8I=; b=bVuQog18XlAx
+YG8FhYOSvrHhdAyr2PUb/24fINK1zlqDGQS56ULJp87ogvG0NBK7G4dNG94Nhnc
GIOtTwZX5+NDpOFcQ6hldkxU7thO1734fWHA6kL8CXKWZ35IWnyyf7/DAp1rPIhe
wUM9td8SwP+/SOibhOOLPKf4Zz9I3qygVvnzMBMFXb0bTQbpV45ASLk0RsG8Q+jP
RBFlRboeqE5mCEgrg3q0i3ip2bGkhqAGzUTmqi0ckTvXltm+nCFpVSKlRy+lgrXY
PQyaK97xt3pUHX9sdcJFHyIDldU/cSWCcTsrQobk5J0UPj8Dlh2RIma/06K9EEcl
Bx27XRIK4Q==

This used to be fine paired with our DMARC policy recommendation, but recently some parties in the email ecosystem have become more stringent, often ignoring the DMARC policy, and rejecting email that is not domain aligned.
Going forward, we are planning to adjust our DKIM-Signature so that it will use your sender domain for alignment. This means, that for a user ‘doe@kolab.org’ the signature would look something like this:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolab.org; h=
content-type:content-type:content-transfer-encoding:message-id
:subject:subject:from:from:date:date:mime-version:received
:received:received; s=dkim20240523; t=1734354313; x=1736168714;
bh=mBUfOmuiUe6nDmAiAsHAHqpD0F+Gd9nJUF5Z5spFd8I=; b=bVuQog18XlAx
+YG8FhYOSvrHhdAyr2PUb/24fINK1zlqDGQS56ULJp87ogvG0NBK7G4dNG94Nhnc
GIOtTwZX5+NDpOFcQ6hldkxU7thO1734fWHA6kL8CXKWZ35IWnyyf7/DAp1rPIhe
wUM9td8SwP+/SOibhOOLPKf4Zz9I3qygVvnzMBMFXb0bTQbpV45ASLk0RsG8Q+jP
RBFlRboeqE5mCEgrg3q0i3ip2bGkhqAGzUTmqi0ckTvXltm+nCFpVSKlRy+lgrXY
PQyaK97xt3pUHX9sdcJFHyIDldU/cSWCcTsrQobk5J0UPj8Dlh2RIma/06K9EEcl
Bx27XRIK4Q==

and so ensuring that all outgoing emails from this sender are domain aligned. However, this will require that the DKIM key is available on your domain in DNS. We recommend that group managers (the owners of private
domains) set the following CNAMEs (both of them) in the DNS of their private domain:

dkim1._domainkey CNAME dkim1._domainkey.kolabnow.com.
dkim2._domainkey CNAME dkim2._domainkey.kolabnow.com.

This will delegate the actual DKIM public key to be managed by the kolabnow.com domain, who in turn will align the key with the sending domain as mentioned above.

We will enable domain-aligned signatures in the end of January 2025, at which point DKIM validation will fail if these above (CNAME) records are not set.

Please keep an eye on this blog for news and updates. We hope this will improve email deliverability.

 

PS: Thank you to the users who reported the issue, and delivered content for our investigations. You know who you are.

Tags: , ,

Incident report: Spam filtering overflow filling up disk..

Posted on: December 19th, 2024 by

On Wednesday  2024-12-18 early evening (~19:00 UTC) a spammer attempted to use a Kolab Now account for sending out large amounts of spam. The Kolab Now exit spam filter was sorting out the spam and redirecting it, as it was supposed to do, and none of the spam was sent out. The spammer was however stubborn and kept up the sending, which subsequently was filling up a disk and hence blocking traffic. Due to ongoing maintenance on the monitoring, the full disk was unfortunately not discovered until Thursday morning, when the problem was immediately corrected, and queued mails were again flowing in both directions.

The problem caused a group of users (about 30%) to be unable to receive mail, and sent mail was queued until the space was again freed up and spooling was possible. No mail should have been lost during the incident.

The missing monitoring has been put back into action, and the Kolab Now Engineering team is evaluating changes that will prevent the situation from repeating.

We apologize for any inconvenience that this incident may have caused.

Tags: ,

Become Kolab ‘Now’: Christmas Referral is ON

Posted on: December 9th, 2024 by

 

This year we have something special for you!

During the last weeks some of our developers have been partly busy with creating something new and fresh: our first Referral Program – Christmas edition (Hooray).

Simply invite your beloved ones to become part of Kolab ‘Now’ by using your personal Christmas Referral Code, earn CHF 10 once your person/persons topped up their wallet with CHF 10 – offering 20% discount for the first 3 months of usage.*

Step by step:

  1. Navigate to your wallet
  2. Find your personal Christmas Referral Code (link)/ QR-Code
  3. Copy that link and send it out to your friends, offering them 20% discount for the first 3 months of usage
  4. For every successful sign-up with a topped up wallet of min. CHF 10, you earn CHF 10 (Kolab ‘Now’ will add this to your wallet)

In case you are still looking for a nice gesture during this festive season Kolab Christmas Referral is ON.

We wish everyone lovely pre-Christmas days.

 

*The Christmas Referral Program will be active until 31.01.2025

Incident: No internal email delivery..

Posted on: November 25th, 2024 by

During the investigation of another issue on the Kolab Now front-end servers, email delivery was broken on one (of many) frontend server. This caused delivered emails to be queued on an internal MX server for a group of users. When the issue was discovered, it was quickly resolved, and emails were again distributed to the user inboxes.

The queuing of emails started at  2024-11-22@21:56 UTC and lasted until 2024-11-23@15:02. Emails were delayed – not dropped. No emails were lost. All emails have been delivered at this time.

Monitoring has been put in place detecting this and similar delivery issues going forward.

We apologize for any inconvenience that this may have caused.

 

 

Tags: ,

Incident: Service outage

Posted on: November 6th, 2024 by

Kolabnow is currently experiencing a networking infrastructure interruption. We apologize for the inconvenience while we investigate the issue.
We will update this blogpost as soon as more information is available.

2024-11-06 @ 05:44 UTC: This issue was triggered by one of our hypervisors spontaneously rebooting. Most services have been restored, the Operations team is working through remaining issues. Users can login and use the facilities.
2024-11-06 @ 07:17 UTC: The incident has been resolved.

Tags:

Incident: DATABASE ERROR!

Posted on: October 29th, 2024 by

A database issue has just presented itself, and our operations team is investigating to find the cause and fix it.

Users who try to make use of the webclient will get the message:

DATABASE ERROR!
Unable to connect to the database!
Please contact your server-administrator.

Operations have a good lead on the issue, and we expect everything to be back online shortly.

You can follow the situation here on this blog.

2024-10-29 @ 10:22 UTC: The root cause of the issue has been identified to be a problem with a synch routine in the database cluster. The Operations team is working to get synchronization back in order. Meanwhile the login and use of the webclient is back. Users can login and use the facilities.

Please keep an eye on this blog, as there might be slipstream performance issues. The synchronization use a lot of resources and will most probably slow down the systems while running. 

 

 

 

Tags: ,