Bitcoin Payment Problems

In recent times, Bitcoin payments have suffered three major blows;

  • Our payment provider now requires customer’s wallets to use a Bitcoin Payment Protocol, while some wallets are simply not compatible,
  • Transaction confirmation times have increased, and then increased some more, both inconveniencing our users and increasing the underpay/overpay problem,
  • Transaction fees have gone through the roof, making the use of Bitcoin less and less attractive, especially for smaller transactions.

Obviously, this is causing some of our customers some grief. Some of the customers that choose to want to use Bitcoin payments exclusively will actually use profanity in expressing their frustration over these changes, but our support staff does not deserve that. I, for one, will defend my staff and cause those customers to find themselves on the losing end of this “conversation”.

Continue Reading “Bitcoin Payment Problems”

Incidents and Recurring Tasks

In the past, we’ve had several occurrences of DNSSEC signatures on DNS zones expiring — partly by not using our own product to the fullest extent of its capabilities. Embarrassing, if you ask me, but it’s more like a misappropriation of the features we did use, where we maybe should have used another feature better suited to our processes and collective work-flows.

So let me explain how we use Kolab’s features to battle our recurring task to refresh signatures, and why and how this is a task that requires manual intervention.

Continue Reading “Incidents and Recurring Tasks”

Announcing Service Windows: Reboot Weekend

As may have already been brought to your attention, some software mitigation is needed for vulnerabilities dubbed “Meltdown” (CVE-2017-5754) and “Spectre” (CVE-2017-5753, CVE-2017-5715).

If I were to summarize the issue with these vulnerabilities then in principle they would, when successfully exploited, allow reading memory that doesn’t belong to the process, the user or even the same operating system instance. In just that way, the Kolab Now infrastructure isn’t impacted.

However, we’re still going to need to patch this out. The only way we can is by updating software and rebooting systems, and this will happen during the weekend of Saturday January 13th and Sunday January 14th.

Continue Reading “Announcing Service Windows: Reboot Weekend”

Annual SSL Certificate Refresh

This is a reminder that our annual certificate renewal period is coming up soon. Usually, we have our certificates issued some time in December, and certificate issuers allow for a grace period up to some time in January. We’re now in that grace period, so our certificates are going to be renewed and cycled throughout our infrastructure.

Continue Reading “Annual SSL Certificate Refresh”

Recursion Violation: A 2nd 2nd Factor

A second factor protects your account, but to such extent that if you loose the one device you are normally using, you will have lost access to your account. Hence, we’re going recursive and get you a second second factor (22FA, 2FA^2).

Here’s the process;

  1. Create a first second factor and call it your Secondary.
  2. Take a picture or screenshot of the QR code, so that you can print it and store it offline, some place safe.
  3. Continue with enabling the first, but secondary, second factor and type the validation code so you know everything works.
  4. Create a new second factor, and confirm the high-security with your Secondary.
  5. Call the new factor Primary; this will be the one you keep on your phone, and use in your day-to-day.
  6. In order for this token to be scanned, you’ll have to remove the Secondary from your device.
  7. Continue with the process of getting your primary second factor on to your device.
  8. Log out.
  9. Loose your phone.

Help! Now what?

  1. Buy a new phone.
  2. Add back your secondary second factor using the piece of paper stored safely offline.
  3. Log back in using this backup TOTP-based second factor.
  4. Go to your settings and remove the primary token; you’ll need to confirm it using your secondary.
  5. Add a new factor and call it New Primary.
  6. Remove the secondary from your phone.
  7. Continue with adding back a primary token and enjoy your new token!

Simple, right? Nothing to it. Too easy.

Incident Report: Backend Down

Earlier this morning, at 04:38 UTC, one out of the twenty-two IMAP backends in production stopped serving its mail spool, showing Input/Output errors on its disk. Our Standard Operating Procedure is to examine log files, flush vm caches, stop the virtual machine, and start it back up again. This occurred at 05:48 UTC. The IMAP backend in question did not come back up cleanly.

Continue Reading “Incident Report: Backend Down”

Pushing Support to our Enterprise Customer Support Platform

We are preparing a transition from our current platform underpinning the email address — based on OTRS — to a more up-to-date, feature-rich environment based on Phabricator.

For various reasons, we much anticipate this change. Not in the least we’ll have real-time notifications about new tickets, and user responses, and a few chat rooms — it’ll allow us to sort our workload better.

Continue Reading “Pushing Support to our Enterprise Customer Support Platform”