Solutions over Playbooks

Just a quick thought I had the last couple of days. I have been part of an on-call rotation for more than a few years now. The hardest part was getting over the "this is how we do things" and break out of the bad habits. Most critically, once you get over them, never fall back. In this case I got used to "playbooks". A playbook is a set of instructions that will guide you (the person on-call) on how to navigate or investigate an issue. Nothing too wrong with this until they become how you act on an alert during your rotation.

Not too long ago my coworker (hey Nick!) surfaced this exact thought during an off-site. That resonated with me as we drifted into playbook-land rather than "how do I make this alert go away forever?". So my invite was to stop delegating this responsibility to a playbook that tells you "oh when this happens you usually have to go through these steps and that will make it go away". Instead we should work towards making that alert go away forever. What is the solution that will not wake me up at night for a similar issue? How can I stop using the playbook every time I see this alert?

This is why I enjoy new people coming onboard and calling out all the weird processes the team has in place. When you find yourself without an answer to explain the why, it's time to take the chance to change the process. Better yet, every once in a while, take a step back and look at your system as a whole. Which alerts are recurring more often? How do we stop that? Is there a permanent fix for this, if not why? Why do we keep following this playbook instead of getting to the bottom of it? How much time are we wasting investigating the same thing over and over again?

Thoughts

Playbooks should guide the high level process but not how to make an alert go away. Drive the initiative that brings a solution, make that alert stop from ever coming up again rather than replaying the same steps over and over. Take a look at your system and start from your most frequent alerts. Stop wasting your time (and your team's) having everyone follow a playbook that should be replaced by a permanent solution.