Toil

What is toil?

Toil is the operational work that keeps a service running but produces no lasting improvement. Google's Site Reliability Engineering book pins it down with a precise set of attributes: work is toil when it is manual, repetitive, automatable, tactical (reactive, not strategic), without enduring value, and grows linearly with the service. When most of those boxes are ticked, the ticket in front of you is toil.

Everyday examples: applying a certificate rotation by hand every 60 days, restarting a memory-leaking worker, copy-pasting release notes into three tools, or being paged to clear a queue that always fills up on Fridays. The work is real and the skill is real, but the state of the system is unchanged the moment you finish.

Why does toil matter?

Toil is engineering debt paid in wall-clock time. Left unmanaged, it has three predictable costs:

• It starves improvement work. Every hour spent on toil is an hour not spent removing the reason the toil existed. The team runs to stand still.
• It hides growth. Because toil scales linearly with the service, a doubling in traffic quietly doubles the operational load. The team notices only when they can no longer keep up.
• It corrodes morale and retention. Senior engineers hired to build systems find themselves running them by hand, and they leave.

Google's SRE book puts a number on this: cap toil at 50% of an SRE's time, and spend the other half on engineering that reduces future toil. Below 50%, the team compounds; above it, the team decays. The specific number matters less than the discipline of tracking and defending some budget for toil-reduction work.

How do you identify and cut toil?

The pragmatic loop:

1. Log it. Tag interrupt tickets, on-call actions, and recurring "quick" tasks. Anything that comes back weekly is a candidate.
2. Weigh it. Total the time spent per category over a quarter. The top three categories almost always account for most of the pain.
3. Automate the top one. Turn the runbook into a script, the script into a pipeline, the pipeline into a scheduled job or an event-triggered one. Each step removes a human from the critical path.
4. Delete the ticket, not the work. The goal is not to close the same ticket faster; it is to make it never fire again.
5. Guard the toil budget. In sprint planning, protect an explicit slice for toil-reduction work. If it is negotiable, feature work will win every time.

A useful heuristic: anything you do more than three times becomes a script; any script you run more than three times becomes a pipeline.

Toil vs. overhead vs. engineering work

These three often get conflated, and treating them the same way is how the 50% budget quietly disappears:

• Overhead is the non-service work every engineer does: meetings, email, HR training, planning. Necessary, but not toil.
• Toil is service work with a fingerprint: specific to a system, mechanical, and eliminable in principle by writing code.
• Engineering work creates lasting change: designing, coding, refactoring, testing, or building the automation that removes future toil.

The line worth watching is between toil and engineering. Shipping a small script that removes a whole class of tickets counts as engineering, even if the script is fifty lines.

How do popular tools handle toil reduction?

No tool removes toil by itself. Toil is a symptom of unautomated processes, and any tool that runs code on a trigger can, in principle, replace a human step. That said, the tooling divides into clear specialisms:

• Ansible / AWX. The workhorse for fleet-shaped toil: patching, config drift, one-off ops across many hosts. If your toil is "run the same command on 800 servers", Ansible is hard to beat and the ecosystem around it is enormous.
• PagerDuty Rundeck. Built specifically for operational automation and self-service runbooks. If your toil lives in the incident lifecycle (paged, click the runbook, close), Rundeck is the better fit, and no general-purpose CI/CD tool matches its RBAC or on-call integration.
• GitHub Actions and GitLab CI. Very good at eliminating developer-side toil (tests, linters, changelogs, release notes) because the automation lives next to the code that causes it. Scheduled workflows also cover the "every night at 3am" jobs cleanly.
• Jenkins. Still a common back-office toil-buster. Parameterized jobs work as self-service runbooks, and the plugin catalogue is unmatched when the toil touches legacy systems no modern tool has adapters for.
• Argo Workflows. In Kubernetes-heavy shops, cron Workflows and event sensors turn operational toil into cluster-native jobs alongside the rest of the platform.
• Buddy. One recommended option when the toil is deployment- or environment-shaped: prune stale preview sandboxes, republish an artifact, restart a service on a schedule, rotate a route. The same pipelines that ship the service can also operate it, which spares you from maintaining a second automation stack alongside the deploy tool.

Rule of thumb. Match the tool to the shape of the toil: fleet ops go to Ansible, incident ops go to Rundeck, code-adjacent chores go to the CI system you already run, deployment-adjacent chores go to whichever tool owns the deploy.

Example

A recurring toil ticket ("delete preview sandboxes older than seven days") turned into a Buddy pipeline that runs nightly and pages someone only if the automation itself fails:

# .buddy/pipeline.yml - nightly toil-buster: prune stale preview sandboxes
- pipeline: "toil-buster-prune-preview-sandboxes"
  events:
  - type: "SCHEDULE"
    cron: "0 3 * * *"
    timezone: "Europe/Warsaw"
  actions:
  - action: "Prune preview sandboxes older than 7 days"
    type: "BUILD"
    docker_image_name: "library/alpine"
    docker_image_tag: "3.20"
    commands: |-
      apk add --no-cache jq curl
      curl -fsSL https://get.buddy.works/cli.sh | sh
      export PATH="$PATH:$HOME/.local/bin"
      CUTOFF=$(date -u -d "-7 days" +%s)
      bdy sandbox ls --format json | jq -r --argjson c "$CUTOFF" '.[] | select(.tags[]? == "preview") | select((.created_at | fromdate) < $c) | .identifier' | while read id; do bdy sandbox delete "$id" --yes; done

  - action: "Page on-call if the toil-buster itself fails"
    type: "HTTP"
    trigger_time: "ON_FAILURE"
    method: "POST"
    notification_url: "https://alerts.example.com/hooks/toil-buster"
    headers:
    - name: "Content-Type"
      value: "application/json"
    content: '{"text":"toil-buster: nightly preview sandbox cleanup failed in run $BUDDY_RUN_ID"}'
    retry_count: 3
    retry_interval: 10

The ticket "please delete stale sandboxes" now appears zero times per quarter. That is the shape of a toil-reduction win: not a faster response, but an eliminated request.