Incidents and Recurring Tasks

In the past, we’ve had several occurrences of DNSSEC signatures on DNS zones expiring — partly by not using our own product to the fullest extent of its capabilities. Embarrassing, if you ask me, but it’s more like a misappropriation of the features we did use, where we maybe should have used another feature better suited to our processes and collective work-flows.

So let me explain how we use Kolab’s features to battle our recurring task to refresh signatures, and why and how this is a task that requires manual intervention.

First, let me outline the topology; we use a hidden master, some “authoritative masters” that are actually slaves, and the RCodeZero AnyCast DNS network (that are also slaves but actually authoritative masters). This article isn’t about DNS topologies though.

Naturally, the keys used to sign zones and zone records do not exist on any infrastructure that the public can get to (in other words, no systems facing the old Interwebs know anything about the keys — they just know about the records and the signatures).

In this topology, we could choose to let the hidden master A) sign and re-sign zones automatically, and B) do so on its own locally defined schedule. We have previously examined our options in this space and decided against automatic signing. We have since re-considered, but only too briefly and no definite result came out of those discussions.

So, our signatures are valid for 30 days. If a zone such as kolabnow.com isn’t signed again within 30 days, the signatures would expire. We had scheduled this signing to happen through a serial bump every two weeks. A calendar event was created, and our system administrators were instructed to obey the event reminder. This didn’t work.

The problem is that this recurring “event”, which was actually a “task”, would naturally return to fall on the shoulders of a designated person — the group of system administrators would expect the one sysadmin that most regularly executed this and other tasks to execute future occurrences of same task, each time. It might as well not have been a shared event at all. Most if not all would not turn on reminders for the shared calendar and/or dismiss the reminder.

In rethinking our approach we decided to go with “Recurring Tasks” rather than a “Recurring Event (actually a Task)”. Furthermore we have added the management overseeing the team so that there’s another level of assurance the task is executed, and on time. It should be noted this Task is not in some shared folder — the organizer and the assignees each have their own local, private copy.

One of the improvements is the fact that the task needs to be completed, or marked as completed, by each individual assignee — i.e. all system administrators, and its management. This means each system administrator individually needs to confirm the task having been completed in order to get to the next occurrence, or have perpetually overdue tasks on their schedule. The same consideration goes for the team’s management.

As an incidental side-effect, this also allows management to plan slightly better. As a result, our process now requires at least one system administrator to perform the task, and management to confirm it has indeed happened. The trigger for the organizer and each assignee to check and double-check is the notification he/she receives when someone marks the task occurrence as completed.

Explore recurring tasks for yourself, and be sure to let us know what you think!