Error budget

How does an error budget work?

An error budget turns an abstract reliability target into a concrete quantity you can spend. If a service commits to a 99.9% availability SLO over a 30-day rolling window, the budget is the other 0.1%: the maximum unreliability the service is allowed before it has broken its promise. On 10 million requests, that's exactly 10,000 failures the team can absorb before the window is in the red.

The mechanics have three moving parts:

1. A Service Level Indicator (SLI) is the thing you measure. Common SLIs are successful requests / total requests, requests served under 300 ms / total requests, or minutes available / minutes in window. The SLI has to be observable in production and meaningful to the user, not just to the ops team.
2. A Service Level Objective (SLO) is the target for that SLI over a window, e.g. "99.9% of requests succeed, measured over a rolling 28 days".
3. The budget itself is (1 − SLO) × traffic in the window. Every failed event, second of downtime or slow response subtracts from the remainder. When the remainder hits zero, the budget is spent.

The subtlety most teams miss is the burn rate. A budget that gets consumed evenly across 30 days is healthy. You've priced the risk correctly. A budget that gets flattened in a single afternoon by one bad rollback is a warning: even if the monthly SLO still holds, the recovery mechanics are fragile. Modern monitoring stacks (Prometheus, Datadog, Google Cloud Monitoring) all support multi-window burn-rate alerts (one alert if you'd exhaust the whole month in an hour, another if you'd exhaust it in a week), which catch trouble long before the number itself hits zero.

The rolling window is critical. A monthly window that resets on the 1st creates an incentive to "spend it or lose it" late in the month, and re-freezes the past. A rolling 28- or 30-day window means yesterday's outage still hurts today, and reliability decisions get anchored in continuous data instead of calendar theatre.

Why does an error budget matter?

Before error budgets, the reliability conversation between engineering and product was structurally broken. Product asked for features; SRE asked for freezes; the loudest voice won that week. An error budget replaces that argument with a shared number, and it does three things at once:

• It legitimizes risk-taking up to a limit. As long as budget remains, teams are supposed to spend it on faster deployment frequency, riskier experiments, and aggressive rollouts. Reliability that runs consistently over-budget suggests the SLO is too loose and users are paying the cost; reliability that runs consistently under-budget suggests the SLO is too tight and the business is leaving velocity on the table.
• It converts "stability vs. speed" into a policy, not a debate. The pre-agreed rule (if the budget is exhausted, we freeze feature launches until it recovers) takes the decision off the critical path. Nobody has to argue it in the middle of an incident, and product leaders don't have to lobby against a nervous ops team. The circuit breaker fires on data.
• It makes reliability negotiable. Teams can, and should, occasionally choose to spend budget deliberately: a big canary release of a risky payments change, a scheduled traffic migration, a data-store upgrade. If the change is worth 20% of the month's budget and the team has 60% left, that's a rational trade instead of a moral one.

Where error budgets go wrong is almost always in the SLI, not the math. A budget defined on HTTP 5xx / total misses the case where the service returns 200 OK with an empty body. A budget on p50 latency celebrates while p99 melts down. Before you enforce a budget, watch it burn for a quarter without freezing anything, and check the times it stayed green against your incident tracker. If the tracker knows about pain the budget doesn't, the SLI is measuring the wrong thing.

The honest limit of the tool: an error budget governs reliability, not correctness. A silently corrupted database write can leave the budget at 100% and still ship a disaster. Pair the budget with strong invariants (data checks, smoke tests, contract tests) for anything the user experiences as "wrong" but not "slow" or "down".

How do popular tools handle error budgets?

Nothing here computes an error budget end-to-end on its own: the mechanic always spans an SLO/monitoring product (which tracks the budget) and a CI/CD pipeline (which enforces it at deploy time).

• Google Cloud Operations / Prometheus + Sloth: the reference implementation. Sloth generates the Prometheus recording rules and multi-window burn-rate alerts straight from an SLO YAML file, and Google Cloud's built-in Service Monitoring shows the burn-down graph natively. If you already live in Prometheus and version SLOs in Git, this stack is hard to beat: you get burn-rate alerts, budget dashboards and PromQL access to the remaining budget as one artifact.
• Nobl9 / Datadog SLOs / Grafana SLO: commercial platforms that ingest SLIs from wherever your telemetry already lives (Datadog, New Relic, CloudWatch, Prometheus) and provide the budget calculation, alerting and audit trail as a first-class product. They are the pragmatic pick when you have many services on mixed backends and want a single reliability pane of glass without hand-rolling PromQL.
• GitHub Actions / GitLab CI: enforce the budget at the deploy step via a status check that queries the SLO API and fails the job when the remaining budget is below a threshold. GitHub's required status checks make the gate policy-enforceable per environment, which is a genuinely nice property if branch protection is already how you gate merges to main.
• Argo CD / Argo Rollouts: AnalysisTemplates can query Prometheus (or a custom provider) for the current budget mid-rollout and abort the canary automatically if the burn rate spikes. If you're all-in on Kubernetes and progressive delivery, Argo Rollouts is the better fit here: the enforcement is native to the reconciliation loop, no external gate needed, and the failure signal wires straight back to the exact revision that caused it.
• Jenkins: no built-in SLO concept, but the HTTP Request plugin inside a pipeline { ... } block is enough to poll an SLO endpoint and error() out. In shops where Jenkins already owns the deploy audit trail, keeping the gate in-tool is usually cheaper than introducing a new platform.
• Buddy is one solid option when you want the budget check to be the deploy gate: the pipeline halts before touching production if the budget is spent, no separate policy engine required. A native HTTP_REQUEST action can hit the SLO backend, trigger_conditions decide whether the next action runs, and the deploy either proceeds or is skipped with a clean audit record. It doesn't compute the SLO itself (a monitoring product still does that), but the enforcement lives in the same YAML as the rest of the pipeline, which keeps the reliability policy visible to whoever ships the code.

Pick whichever combination already owns your telemetry and your deploys. The single biggest reliability win from adopting error budgets is not the specific tool. It's writing the freeze policy down before you need it.

Example

A Buddy pipeline that checks the remaining error budget from a Prometheus/Sloth backend before every production deploy, fails cleanly when the budget is exhausted, and on success proceeds to a gated rollout. The SLO endpoint returns the remaining budget as a fraction (1.0 = full, 0.0 = spent); the deploy is allowed only while at least 10% of the month's budget is left.

# .buddy/buddy.yml — gate production deploys on remaining error budget.
- pipeline: "deploy-production-with-budget-gate"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
  variables:
    - key: "SERVICE"
      value: "checkout-api"
    - key: "SLO_ENDPOINT"
      value: "https://slo.example.com/api/v1/budgets/checkout-api-availability"
    - key: "BUDGET_FLOOR"
      value: "0.10"
  actions:
    - action: "Block deploy if error budget is exhausted"
      type: "BUILD"
      docker_image_name: "alpine"
      docker_image_tag: "3.20"
      commands: |-
        apk add --no-cache curl jq
        BUDGET_REMAINING=$(curl -fsS "$SLO_ENDPOINT" | jq -r '.remaining_fraction')
        echo "Remaining error budget: $BUDGET_REMAINING"
        awk "BEGIN { exit !($BUDGET_REMAINING > $BUDGET_FLOOR) }" \
          || (echo "Error budget below $BUDGET_FLOOR — freezing deploys." && exit 1)

    - action: "Build & canary to 10% traffic"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci
        npm run build
        ./scripts/deploy-canary.sh --weight 10

    - action: "Promote to 100% traffic"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: "./scripts/promote.sh --weight 100"

    - action: "Notify on freeze"
      type: "HTTP"
      method: "POST"
      notification_url: "https://chat.example.com/hooks/reliability"
      trigger_time: "ON_FAILURE"
      headers:
        - name: "Content-Type"
          value: "application/json"
      content: |
        {
          "text": "❄️ ${SERVICE}: deploy of ${BUDDY_EXECUTION_REVISION} frozen — error budget below floor ${BUDGET_FLOOR}."
        }

The interesting property is what the pipeline does not do: it doesn't try to compute the SLO itself, and it doesn't apologise. It reads the number from the system that owns reliability data, compares it to a pre-agreed floor, and either ships or freezes. The freeze policy (the actual reliability decision) lives in one variable, BUDGET_FLOOR, which is auditable, changeable via pull request, and visible to everyone shipping through the same pipeline. When the budget recovers on the next window, main starts flowing again automatically, no manual override required.