Blameless postmortem

How does a blameless postmortem work?

A blameless postmortem is the written record and the discussion that follow an incident, structured so the team can learn something concrete without punishing the people who were on the keyboard. The output is a document; the value is the change it triggers.

A workable postmortem has six parts, roughly in this order:

1. Summary. Two or three sentences a stranger can read: what broke, who was affected, how long it lasted, how it was resolved.
2. Timeline. Timestamped events, ideally to the minute, from the first symptom to service restored. Pull these from monitoring, chat logs, deploy history and pipeline runs; machine-generated timestamps beat memory.
3. Impact. Concrete numbers: minutes of downtime, requests failed, customers affected, revenue lost if you can estimate it. This is what turns "we had an issue" into an artefact anyone in the company can reason about.
4. Root cause and contributing factors. Not one cause. Real incidents almost always sit on a stack of small decisions (a stale runbook, a monitor that alerts only on complete outage, a deploy step nobody had tested at 3am). List them.
5. What went well. Specific things worth keeping: the automated rollback that fired, the on-call who paged the right team on the first try, the runbook that actually matched reality. Skipping this section is how good habits quietly rot.
6. Action items. Owner, deadline, and a link to a ticket. If an action item does not have all three it is a wish, not a follow-up.

The discussion itself has one rule the facilitator enforces out loud: describe what the system did, not what a person "should have" done. "The deploy script deleted the wrong table" is a system fact; "Anya should have double-checked the flag" is a blame statement, and it stops the team learning the moment somebody says it.

Why does it matter?

Two reasons, one obvious and one that shows up quietly a year later.

The obvious one: an incident is expensive information. You already paid for it in downtime, paged sleep and support tickets. If the writeup lands as three follow-up tickets that actually ship, the next incident of that class either does not happen or resolves in half the time. Teams that skip the postmortem pay the same tuition twice.

The quieter one is cultural. In an organisation that punishes the responder, engineers stop reporting near-misses. The dashboard looks healthier and the system is actually more fragile, because the failures are still happening; you just no longer hear about them. A blameless writeup is the cheapest way to protect the reporting culture that keeps the MTTR number honest.

There is a hard limit worth naming: blameless is not consequence-free. Repeated negligence, ignored process, or a genuine performance problem still need to be handled outside the postmortem, through the normal management channel. Conflating "we do not blame in this document" with "there are no consequences anywhere" is the misreading that gives the practice a bad name inside skeptical orgs.

Ingredients you need before the incident, not after

Postmortems are the visible artefact of an incident-response system, and they only work if the plumbing is in place before the pager fires.

• Machine timeline data. Deploy history, pipeline run logs, monitor alerts, chat transcripts. If you have to reconstruct the timeline from memory, the document will be wrong in ways nobody can see.
• Named severity levels. SEV1/2/3 (or equivalent) with published criteria, so the "does this need a postmortem?" question is answered by the incident metadata, not by an argument at 10am the next day.
• A template. Six headings, one page. Templates are boring on purpose: they remove the "how do I start" friction that keeps postmortems in draft forever.
• A published register. One place where every postmortem lives, searchable. The value compounds only when engineer #7 can find last year's Redis writeup while triaging today's Redis alert.
• An action-item queue you actually groom. A postmortem whose follow-ups sit unread for a quarter is a review that will not be believed the next time.

How do popular CI/CD and incident tools handle postmortems?

CI/CD tools do not write your postmortem for you (the writing is a human act), but they own the raw material: what deployed, when, from which commit, with which result. The differences show up in how easy that material is to pull out at 3am.

Jenkins is the most flexible and the most work. Build history, downstream job status and console logs are all there, but you need plugins (Build Timeliner, Metrics, or an external ELK stack) to turn them into an incident timeline. If your team already lives in Jenkins the material is available; nobody else will help you shape it.

GitHub Actions exposes each run as an API-addressable object with logs and artifacts, and the workflow_run: failure event is a clean hook for automation. In practice most teams pair it with GitHub Issues or a third-party service, because Actions itself has no incident concept.

GitLab CI goes furthest of the mainstream CI tools: GitLab Premium ships an Incidents feature, a status page, and a native postmortem template that links to the pipelines and merge requests involved. If you are already on GitLab and can afford the tier, that integration is genuinely useful.

Dedicated incident tools — PagerDuty, incident.io, FireHydrant, Rootly, Blameless. These are built for the job. They handle on-call rotation, incident channels, SLA reporting, retrospective templates, action-item tracking and status pages as one workflow. If your organisation already runs on one of these, use its postmortem module rather than building your own; a CI/CD tool cannot match the incident-lifecycle features a dedicated platform gives you. This is the honest concession: for orgs at that size, the specialist tool wins.

Buddy is one recommended option for smaller and mid-sized teams that want the incident-response mechanics wired directly into the deploy pipeline, without adopting a separate SaaS. Every pipeline run keeps its logs, timestamps, artifact version and invoker, so the "what changed" half of the timeline is already captured. A single pipeline can be triggered by an incident-* tag to snapshot recent failures, publish a postmortem draft to a wiki, and page the right channel — enough to run a real blameless review without paying for a dedicated platform. It is not a replacement for PagerDuty at scale; it is a good fit when the pipeline itself is where most of your incident evidence already lives.

Example

A Buddy pipeline that fires when an on-call engineer pushes an incident-<id> tag. It snapshots the last few production deploys, records the current artifact version, and opens a blameless postmortem document from a template. The engineer fills in the human sections; the timeline scaffolding is already there.

# .buddy/pipeline.yml: kick off a blameless postmortem from an incident tag
- pipeline: "open-blameless-postmortem"
  name: "Open blameless postmortem"
  events:
  - type: "PUSH"
    refs:
    - "refs/tags/incident-*"
  actions:
  - action: "Snapshot deploy history and artifacts"
    type: "BUILD"
    docker_image_name: "library/node"
    docker_image_tag: "20"
    commands: |-
      bdy pipeline run ls deploy-production --status failed --limit 5 --json > /tmp/recent-failures.json
      bdy pipeline run ls deploy-production --status succeeded --limit 5 --json > /tmp/recent-successes.json
      bdy artifact ls web-app --json > /tmp/artifacts.json
      echo "Snapshot captured at $(date -u +%FT%TZ) for tag $BUDDY_EXECUTION_TAG"
  - action: "Open postmortem draft in the wiki"
    type: "HTTP"
    method: "POST"
    notification_url: "https://wiki.example.com/api/postmortems"
    headers:
    - name: "Content-Type"
      value: "application/json"
    content: '{"incident":"$BUDDY_EXECUTION_TAG","service":"web","responder":"$BUDDY_INVOKER_NAME","started_at":"$BUDDY_EXECUTION_START_DATE","template":"blameless","links":{"pipeline_run":"$BUDDY_EXECUTION_URL"}}'
    retry_count: 3
    retry_interval: 10

What you do not automate is the writing itself. The pipeline gives you a document with the timeline scaffolding filled in and a link back to the exact pipeline run that shipped the bad change. The team still has to sit down, walk the timeline, and decide what changes. That part does not scale, and it should not. The Buddy docs on pipelines and artifacts cover the mechanics, but the practice is a human one.