Author Archive for Mads Petersen

Action required for Group managers!: Change to DKIM-Signatures – domain alignment

Posted on: December 20th, 2024 by

Lately we have seen a few emails not being delivered to third parties and bounced emails with messages about failing DKIM signatures.

DKIM is a mechanism that allows a receiving party of emails to determine whether an email has indeed been sent by the party that is claimed to be the sender, thus protecting against forged sender email addresses. Kolab Now implemented DKIM signatures a long time ago, but so far we have always used the kolabnow.com domain as the sender domain, when sending an email from a custom domain. An example signature header would look like this (please note the ‘d= tag’):

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h=
content-type:content-type:content-transfer-encoding:message-id
:subject:subject:from:from:date:date:mime-version:received
:received:received; s=dkim20240523; t=1734354313; x=1736168714;
bh=mBUfOmuiUe6nDmAiAsHAHqpD0F+Gd9nJUF5Z5spFd8I=; b=bVuQog18XlAx
+YG8FhYOSvrHhdAyr2PUb/24fINK1zlqDGQS56ULJp87ogvG0NBK7G4dNG94Nhnc
GIOtTwZX5+NDpOFcQ6hldkxU7thO1734fWHA6kL8CXKWZ35IWnyyf7/DAp1rPIhe
wUM9td8SwP+/SOibhOOLPKf4Zz9I3qygVvnzMBMFXb0bTQbpV45ASLk0RsG8Q+jP
RBFlRboeqE5mCEgrg3q0i3ip2bGkhqAGzUTmqi0ckTvXltm+nCFpVSKlRy+lgrXY
PQyaK97xt3pUHX9sdcJFHyIDldU/cSWCcTsrQobk5J0UPj8Dlh2RIma/06K9EEcl
Bx27XRIK4Q==

This used to be fine paired with our DMARC policy recommendation, but recently some parties in the email ecosystem have become more stringent, often ignoring the DMARC policy, and rejecting email that is not domain aligned.
Going forward, we are planning to adjust our DKIM-Signature so that it will use your sender domain for allignment. This means, that for a user ‘doe@kolab.org’ the signature would look something like this:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolab.org; h=
content-type:content-type:content-transfer-encoding:message-id
:subject:subject:from:from:date:date:mime-version:received
:received:received; s=dkim20240523; t=1734354313; x=1736168714;
bh=mBUfOmuiUe6nDmAiAsHAHqpD0F+Gd9nJUF5Z5spFd8I=; b=bVuQog18XlAx
+YG8FhYOSvrHhdAyr2PUb/24fINK1zlqDGQS56ULJp87ogvG0NBK7G4dNG94Nhnc
GIOtTwZX5+NDpOFcQ6hldkxU7thO1734fWHA6kL8CXKWZ35IWnyyf7/DAp1rPIhe
wUM9td8SwP+/SOibhOOLPKf4Zz9I3qygVvnzMBMFXb0bTQbpV45ASLk0RsG8Q+jP
RBFlRboeqE5mCEgrg3q0i3ip2bGkhqAGzUTmqi0ckTvXltm+nCFpVSKlRy+lgrXY
PQyaK97xt3pUHX9sdcJFHyIDldU/cSWCcTsrQobk5J0UPj8Dlh2RIma/06K9EEcl
Bx27XRIK4Q==

and so ensuring that all outgoing emails from this sender are domain aligned. However, this will require that the DKIM key is available on your domain in DNS. We recommend that group managers (the owners of private
domains) set the following CNAMEs (both of them) in the DNS of their private domain:

dkim1 CNAME dkim1.kolabnow.com.
dkim2 CNAME dkim2.kolabnow.com.

This will delegate the actual DKIM public key to be managed by the kolabnow.com domain, who in turn will align the key with the sending domain as mentioned above.

We will enable domain-aligned signatures in the end of January 2025, at which point DKIM validation will fail if these above (CNAME) records are not set.

Please keep an eye on this blog for news and updates. We hope this will improve email deliverability.

 

PS: Thank you to the users who reported the issue, and delivered content for our investigations. You know who you are.

Tags: , ,

Incident report: Spam filtering overflow filling up disk..

Posted on: December 19th, 2024 by

On Wednesday  2024-12-18 early evening (~19:00 UTC) a spammer attempted to use a Kolab Now account for sending out large amounts of spam. The Kolab Now exit spam filter was sorting out the spam and redirecting it, as it was supposed to do, and none of the spam was sent out. The spammer was however stubborn and kept up the sending, which subsequently was filling up a disk and hence blocking traffic. Due to ongoing maintenance on the monitoring, the full disk was unfortunately not discovered until Thursday morning, when the problem was immediately corrected, and queued mails were again flowing in both directions.

The problem caused a group of users (about 30%) to be unable to receive mail, and sent mail was queued until the space was again freed up and spooling was possible. No mail should have been lost during the incident.

The missing monitoring has been put back into action, and the Kolab Now Engineering team is evaluating changes that will prevent the situation from repeating.

We apologize for any inconvenience that this incident may have caused.

Tags: ,

Incident: No internal email delivery..

Posted on: November 25th, 2024 by

During the investigation of another issue on the Kolab Now front-end servers, email delivery was broken on one (of many) frontend server. This caused delivered emails to be queued on an internal MX server for a group of users. When the issue was discovered, it was quickly resolved, and emails were again distributed to the user inboxes.

The queuing of emails started at  2024-11-22@21:56 UTC and lasted until 2024-11-23@15:02. Emails were delayed – not dropped. No emails were lost. All emails have been delivered at this time.

Monitoring has been put in place detecting this and similar delivery issues going forward.

We apologize for any inconvenience that this may have caused.

 

 

Tags: ,

Incident: DATABASE ERROR!

Posted on: October 29th, 2024 by

A database issue has just presented itself, and our operations team is investigating to find the cause and fix it.

Users who try to make use of the webclient will get the message:

DATABASE ERROR!
Unable to connect to the database!
Please contact your server-administrator.

Operations have a good lead on the issue, and we expect everything to be back online shortly.

You can follow the situation here on this blog.

2024-10-29 @ 10:22 UTC: The root cause of the issue has been identified to be a problem with a synch routine in the database cluster. The Operations team is working to get synchronization back in order. Meanwhile the login and use of the webclient is back. Users can login and use the facilities.

Please keep an eye on this blog, as there might be slipstream performance issues. The synchronization use a lot of resources and will most probably slow down the systems while running. 

 

 

 

Tags: ,

Incident: The dashboard is down..

Posted on: October 25th, 2024 by

This morning many users has reported that they can not login to the dashboard of Kolab Now.

We have seen the issue, and the operations team is working to resolve the problem.

Meanwhile you can reach your webclient directly through: https://kolabnow.com/apps and IMAP and ActiveSync clients should be working as well.

We will update this post as soon as we know more about the issue and what to expect.

2024-10-25 @ 06:34 UTC: The direct cause of the problem was identified and mitigated. One database growing over it’s expected size. More investigation needed to identify the root cause, but all services should now be available for all users.

Service Window: Datacenter maintenance

Posted on: August 22nd, 2024 by

The internet provider for Kolab Now, NTS, planned changes to some configurations of the access routing in the Bern datacenters. Therefore we hereby declare a service window on this coming:

                        Tuesday, 2024-08-27 @ 18:00 UTC until 22:00 UTC

Users will experience a 5 minute outage at some point during this window. Unfortunately we are unable to narrow down a more close time slot for the 5 minute outage. We will however update this note with any update that we get from the provider.

Should you have any questions or concerns in this context, then please contact Support.

 

 

2024-08-27 @ 20:34 UTC – Update: Network access has been interrupted since 19:30 UTC and we are in contact with NTS to re-establish access as soon as possible. 

 

2024-08-27 @ 20:48 UTC – Update: Network connection was re-established and Kolab Now is back up and running. Mails arriving during the downtime was delayed – not deleted. No mails was lost.

An update with more details will be posted here as soon as we know the details. We regret any inconvenience that this extra downtime has caused.

Tags: , ,

Incident report: Spool overflow filling up disk..

Posted on: July 2nd, 2024 by

On Monday 2024-07-01 early evening (CEST) a spammer attempted to use a Kolab Now account for sending out large amounts of spam. The Kolab Now exit spam filter was sorting out the spam and redirecting it, as it was supposed to do, and none of the spam was sent out. The spammer was however stubborn and kept up the sending, which subsequently was filling up a disk and hence blocking traffic. Due to ongoing maintenance on the monitoring, the full disk was unfortunately not discovered until Tuesday morning, when the problem was immediately corrected, and queued mails were again flowing in both directions.

The problem caused a group of users (about 30%) to be unable to receive mail, and sent mail was queued until the space was again freed up and spooling was possible. No mail should have been lost during the incident.

The missing monitoring has been put back into action, and the Kolab Now Engineering team is evaluating changes that will prevent the situation from repeating.

We apologize for any inconvenience that this incident may have caused.

Tags: ,

Urgent Service Window: More updates to frontend servers

Posted on: May 27th, 2024 by

A list of needed updates are ready to be deployed at Kolab Now – Some of these updates are of some urgency. Therefore we hereby announce an Urgent service window for:

Wednesday 2024-05-29 @ 07:00 UTC (09:00 CEST)

The service window will last for an hour.

Users can keep working, but at some point during the one hour service window, the webclient and the dashboard will be unavailable for a few minutes. Users will be logged out and will need to login again.

This blogpost will be updated with status of the work during the service window.

Please contact support with any questions or concerns in this context.

 

2024-05-29 @ 07:40 UTC – The work is done and the results are tested. All activities were successfull. If you have any issues please contact Support.

Tags: ,

Service Window: Update of front end servers

Posted on: April 4th, 2024 by

A list of needed updates are ready to be deployed at Kolab Now – Some of these updates are of some urgency. Therefore we hereby announce a service window for:

Monday 2024 – 04 – 08 @ 07:00 UTC (09:00 CEST)

The service window will last for an hour. Users can keep working, but they will experience a short hickup outage at some point during that period.

One important fix that will be deployed during this service window is:

  • Roundcube: Update of calendar invitation cause the message “Failed to import event”, and the event is not updated. This has been resolved in the update.

Please contact support with any questions or concerns in this context.

 

2024-04-08 @ 07:32 UTC – Update: The updates has been executed and the service window is done. 

Tags: ,

ActiveSync broken by Windows update..

Posted on: February 8th, 2024 by

Since last weekend, Support received a few reports from users who saw problems with Outlook ActiveSync synchronization towards Kolab Now. No emails were synchronized to their inbox, and the content of the Outlook Inbox would not match the inbox in the Kolab Webclient.

After some investigation it was realized, that Microsoft had provided (and automatically deployed) a new update on February 1st, and that this update caused the faulty behavior. The new version was: 2401 (various build numbers).

Rolling back Outlook resolved the issue for all users and brought back the normal behavior, and outlook would again synchronize email to the inbox.

A short version of the downgrade procedure is:

  • Find the version to downgrade to at the Microsoft site.
  • Disable automatic updates via Start -> Settings -> Windows Updates
  • Close down Outlook.
  • Open an admin shell.
    • Run: C:\Program Files\Common Files\microsoft shared\ClickToRun\OfficeC2RClient.exe /update user updatetoversion=16.0.17029.20108
    • (change 16.0.17029.20108 if you want to roll back to an even older version).
  • Start Outlook again.

Please seek further information about rollback of outlook versions at the Microsoft support site.

2024-02-23 – Update: We had a few users asking when we will have a fix for this issue. To that we must refer to the Microsoft support article about the issue, which can be found here. In short they lightly suggest, that the patch will be released on 1st of April (We do not think that this is an April Fool’s joke ).

 

Tags: