Runbook

What is a runbook?

A runbook is a step-by-step procedure for operating a specific part of a system. It answers exactly one question - "how do I do X on this service?" - where X might be routine (rotate a certificate, restart a worker, replay a queue) or urgent (recover from a bad deploy, cut a hotfix, revoke a leaked token). It exists so the person on-call at 3am does not have to reason from first principles.

The best runbooks share three traits: they are short (one screen if possible), current (updated the moment reality drifts), and executable where possible (a script or pipeline, not a wall of prose).

Why do runbooks matter?

Because incidents happen to the person least equipped to think clearly about them. Whoever is paged has partial context, adrenaline, and a ticking clock. A runbook flattens that:

• It reduces MTTR. The recovery steps are already written; the responder executes rather than invents.
• It democratizes on-call. Junior engineers can safely handle scenarios they have never seen, because the senior who wrote the runbook is - via the document - "in the room" with them.
• It captures institutional memory. The 2am fix a senior engineer figured out becomes a repeatable operation instead of tribal knowledge that leaves when they do.
• It exposes automation candidates. Any runbook step you execute more than three times is a candidate for a script; any script you run more than three times is a candidate for a pipeline.

What belongs in a good runbook?

At minimum:

1. What this runbook is for - one sentence, in plain English. If it takes three, the runbook is doing too much; split it.
2. When to use it - the observable trigger ("queue depth exceeds 10k", "error rate above 2% for 5 minutes"), not a feeling.
3. Prerequisites - permissions, secrets, and tools the operator needs before starting.
4. The steps - numbered, imperative ("Run foo", not "You might want to run foo"). Include the expected output so the operator knows what "good" looks like.
5. Verification - how to confirm the fix worked, with the exact metric, log line, or command.
6. Escalation - who to page if the runbook fails, and at which step.
7. A "last updated" line - a runbook without a date is a runbook nobody trusts.

Prose runbooks vs executable runbooks

The industry has been quietly moving from wiki-page runbooks to executable ones - pipeline files, scripts, and jobs a human triggers with one click.

• Prose runbooks are cheap to write but rot silently. Nothing tells you that the curl command in step 3 stopped working two releases ago.
• Executable runbooks run in CI/CD. When the service changes, the runbook changes in the same pull request. It is tested. It is reviewed. It leaves logs. It has authorship and history.

The pragmatic middle: keep judgment steps as prose (customer comms, "should we page the CTO?"), and put every mechanical step into code that a human triggers.

How do popular tools handle runbooks?

Runbook tooling breaks into two camps: incident-response platforms and CI/CD systems repurposed for operations.

• PagerDuty Rundeck (formerly Rundeck). The reference implementation for operational runbook automation. Deep on-call integration, rich RBAC, node-based execution, and a decade of enterprise polish. If your runbooks live inside the incident lifecycle (paged → runbook → post-mortem), Rundeck is the better fit and it is not close.
• Ansible / AWX / Ansible Automation Platform. Playbooks are the industry standard for configuration and one-shot ops tasks. Excellent when the operation is fleet-shaped ("do X on 400 hosts") and inventory-driven.
• GitHub Actions and GitLab CI. Widely used as makeshift runbooks: workflow_dispatch and manual pipelines let anyone trigger a documented, audited procedure from the UI. Not purpose-built for ops, but "the runbook lives next to the code" is a genuine advantage.
• Jenkins. Still ubiquitous as an operator's remote control - a parameterized Jenkins job remains a very effective runbook, especially in environments that already run Jenkins.
• Buddy. Buddy pipelines with trigger_mode: MANUAL work well as executable runbooks when the operation is deployment-shaped - restart a sandbox, republish an artifact, promote a version, rotate a route. It is one recommended option specifically because the same tool that ships the service also operates it, so there is no second runbook system to keep in sync with the deploy tool.

Rule of thumb. If the runbook is about incidents and on-call, use a dedicated runbook platform such as Rundeck. If the runbook is about the service you already deploy, keep it next to the pipeline that deploys it.

Example

Here is a real executable runbook as a Buddy pipeline - manually triggered by the on-call engineer to safely restart a production API sandbox after a lock-up:

# .buddy/pipeline.yml - runbook: restart the production API
# No events are defined, so this pipeline only runs when a human clicks "Run" - never automatically.
- pipeline: "runbook-restart-prod-api"
  refs:
  - "refs/heads/main"
  actions:
  - action: "Drain traffic"
    type: "BUILD"
    docker_image_name: "library/alpine"
    commands: |-
      apk add --no-cache curl
      curl -sf -X POST https://api.example.com/admin/drain

  - action: "Restart sandbox"
    type: "SANDBOX_MANAGE"
    operation: "APP_RESTART"
    targets:
    - "prod-api"

  - action: "Smoke-test /healthz"
    type: "BUILD"
    docker_image_name: "library/alpine"
    commands: |-
      apk add --no-cache curl
      for i in 1 2 3 4 5; do curl -sf https://api.example.com/healthz && exit 0; sleep 5; done; exit 1

  - action: "Notify #ops"
    type: "SLACK"
    channel: "ops"
    integration: "slack"
    content: "prod-api restarted by ${BUDDY_EXECUTION_STARTED_BY} - run ${BUDDY_EXECUTION_ID}"

The runbook is: a file in the repo, versioned, code-reviewed, one click to run, and logged. The prose parts - "when to invoke this, who to escalate to if the smoke test fails" - stay in a short RUNBOOK.md next to it. That split, prose for judgment and code for mechanics, is what a modern operational runbook looks like in 2026.