Mean time to recovery

How is mean time to recovery measured?

MTTR is a simple average: the total time spent in incidents during a window, divided by the number of incidents in that window. Every incident contributes one interval (from the moment it starts to the moment the service is restored), and MTTR is the mean of those intervals.

The metric looks trivial, and then falls apart on the two questions nobody wants to answer up front:

1. When does the clock start? The moment the failure begins in production, the moment monitoring detects it, or the moment a human acknowledges the page? DORA and most incident-management practice use failure start (the earliest point users were affected) because that is what actually matters to them. Starting the clock at "someone acknowledged" quietly hides slow detection inside a healthy-looking number.
2. When does the clock stop? When the service is restored to a healthy state, not when the root cause is fixed. A hotfix or a rollback that gets users working again stops the clock; the follow-up work to understand and repair the underlying bug is a separate metric (mean time to repair, sometimes tracked, often collapsed into MTTR by accident).

Get those two edges wrong and every downstream conversation drifts. It also helps to know that "MTTR" is used loosely in the field:

Acronym	Full name	What it measures
MTTD	Mean time to detect	Failure start → someone notices
MTTA	Mean time to acknowledge	Alert fires → responder acknowledges
MTTR	Mean time to restore (DORA)	Failure start → service healthy again
MTTR	Mean time to repair (reliability)	Failure start → root cause fixed
MTBF	Mean time between failures	Reliability: average uptime between incidents

Pick the DORA definition (restore) and stick with it. Publish it next to the number on your dashboard so nobody has to reverse-engineer it from a chart.

Why does MTTR matter?

MTTR is the stability half of the DORA "speed with stability" pair. The Accelerate research shows elite teams do not have fewer incidents than everyone else. They recover from them much faster. That difference is what lets them run deployment frequency at multiple deploys per day without the risk spiralling: when every deploy is a small change with a rehearsed rollback, the worst case is a five-minute revert, not a five-hour war room.

Practically, MTTR is where three investments compound:

• Observability shortens detection. You cannot recover from a failure you have not noticed. Health checks, SLO burn alerts and synthetic probes shrink the invisible chunk of every incident.
• Automated rollback shortens restoration. If the pipeline that shipped the bad version can also revert it on a failing post-deploy check, MTTR collapses to the length of a health probe. This is the single biggest lever.
• Small changes shorten diagnosis. A deploy that touched one service and 40 lines of code is trivial to attribute; a "big bang" release touching a dozen services can burn hours just in bisecting. This is why trunk-based development and continuous deployment keep showing up in MTTR conversations: the batch size is the debugging window.

One honest caveat: MTTR is easy to game. Excluding "minor" incidents, closing tickets before the fix is verified, or splitting one long outage into three shorter ones all make the number prettier without helping users. Audit the incident list the number is built from, not just the number.

How do popular tools help you improve MTTR?

MTTR spans three concerns (detect, coordinate, restore), and no single tool owns all three well. What matters is which one owns the bit that is currently your bottleneck.

• PagerDuty, Opsgenie and Grafana Incident are the incumbent incident-response platforms. They own the on-call schedule, the escalation ladder and the postmortem timeline, and they compute MTTR straight from the incident lifecycle. If your bottleneck is the human coordination side (noisy pages, slow escalations, missing responders), a dedicated incident platform is the honest winner and no CI/CD tool comes close. This is where MTTR investment should start for most orgs.
• FireHydrant and incident.io are newer players that lean harder into ChatOps and structured runbooks inside Slack. Genuinely nice for teams whose incident response happens in chat anyway; the MTTR chart is a by-product of running the incident there instead of a separate step.
• Argo Rollouts: for Kubernetes-native shops, this is the sharpest tool for the restore side. It watches Prometheus queries during a canary and reverts automatically when the SLI breaches its threshold. If you are all-in on Kubernetes, Argo Rollouts is the native answer and will almost always beat wiring the same behaviour together yourself.
• GitLab CI and GitHub Actions both surface deploy-and-rollback workflows and, on GitLab Premium/Ultimate, Value Stream Analytics computes MTTR straight from the deploy + incident data with no glue code. Cheap, credible dashboard if you are already paying for the platform.
• Datadog and New Relic cover the observability side of MTTR. Neither is an incident manager, but their SLO tracking and automatic anomaly detection are usually what shortens detection, which is the invisible half of most MTTR numbers.
• Specialised DORA platforms (Sleuth, LinearB, Jellyfish) pull events from the incident tracker and CI/CD and produce org-wide MTTR dashboards with drilldowns per service. Worth it once you are running enough teams that a shared spreadsheet has become a liability.
• Buddy is one solid option when the bottleneck is the restore half: specifically, when a bad release should be reverted automatically before anyone is paged. An ON_ACTION_FAILURE trigger can run a rollback pipeline the moment a post-deploy health check fails, and a structured HTTP action can push a resolved event into whatever incident system already owns the MTTR chart. The dashboard still lives elsewhere; Buddy's job is to make the revert itself boring, so the incident that would have lasted an hour lasts a health-check interval.

There is no single winning tool here. Diagnose which slice of MTTR is longest (detect, coordinate or restore) and put the next dollar into a tool that owns that slice, rather than replacing the whole stack for a chart.

Example

A minimal Buddy pipeline that deploys main, verifies health with an HTTP probe, and (on a failing check) automatically triggers a rollback pipeline and posts a resolved event to the incident tracker. Every incident this pattern catches never reaches a human; the MTTR contribution is the length of the health check plus the redeploy of the previous artifact.

# .buddy/buddy.yml — auto-rollback on failing health check, closes the incident loop
- pipeline: "deploy-production"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
  variables:
    - key: "SERVICE"
      value: "checkout-api"
    - key: "HEALTH_URL"
      value: "https://checkout-api.example.com/healthz"
    - key: "INCIDENT_ENDPOINT"
      value: "https://incidents.example.com/events"
  actions:
    - action: "Deploy new version"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci
        npm run build
        ./scripts/deploy.sh

    - action: "Post-deploy health check"
      type: "HTTP"
      method: "GET"
      notification_url: "${HEALTH_URL}"
      retry_count: 3
      retry_interval: 10

    - action: "Auto-rollback on failed health check"
      type: "RUN_NEXT_PIPELINE"
      project: "checkout-api"
      pipeline: "rollback-production"
      # runs only when a preceding action (the health check) failed
      run_only_on_first_failure: true

    - action: "Close incident on successful restore"
      type: "HTTP"
      method: "POST"
      notification_url: "${INCIDENT_ENDPOINT}"
      headers:
        - name: "Content-Type"
          value: "application/json"
      content: |-
        {
          "service":     "${SERVICE}",
          "commit":      "${BUDDY_EXECUTION_REVISION}",
          "event":       "auto_rollback_completed",
          "resolved_at": "${BUDDY_EXECUTION_FINISH_DATE}"
        }
      run_only_on_first_failure: true

The clock on any incident this catches starts when the bad deploy hits production and stops when the health check fails and the rollback lands, usually well under a minute. The dashboard that computes MTTR reads its incidents from ${INCIDENT_ENDPOINT}; the average shrinks not because you improved a chart, but because the class of incident that dominated it no longer reaches a human.