Service Level Objective

How does an SLO work?

An SLO puts a number on how reliable a service is supposed to be, and pairs it with the window that number is measured over. A stock example: "99.9% of API requests return successfully (2xx or 3xx) over any rolling 28 days". Every part of that sentence is doing work.

• The SLI (Service Level Indicator) is what you count. It has to be a ratio that a real user would recognise as "the service worked": success rate, latency below a threshold, availability, freshness of a data feed. Pick the SLI badly (say, HTTP 5xx over total when a broken 200 OK response also counts as a failure) and every downstream decision gets skewed.
• The target is the fraction you commit to, typically expressed as a percentage. Common tiers are 99%, 99.9%, 99.95%, 99.99%; each extra nine is roughly ten times harder to hit than the last, and roughly ten times more expensive in redundancy and process.
• The window is the time slice over which compliance is judged. A rolling 28- or 30-day window is standard, because it doesn't reset on the 1st of each month and doesn't reward "spend the reliability faster in the last week".

Once those three are pinned, the rest of the machinery writes itself. An error budget drops out mechanically (1 − SLO, multiplied by the traffic in the window). Alerting thresholds drop out too: a burn-rate alert fires when the current failure rate would exhaust the whole month's budget in an hour, or in six hours, so operators get warned long before the budget itself reaches zero. Release policy drops out ("if remaining budget is under 20%, hold non-critical launches"). Postmortem template rows drop out ("how much of the budget did this incident consume, and against which SLO?").

The mistake teams make is treating the SLO number as sacred. It isn't. The SLO is a contract with users, not a physics constant, and it is meant to be renegotiated when reality changes. Two rules of thumb keep it honest:

1. Aspiration is not an SLO. Set the target slightly below current measured performance, not slightly above. An SLO the service already fails to meet on a normal week isn't a target; it's a wish, and it will teach the team to ignore its own alerts.
2. Users don't feel your averages. If p50 latency is 40 ms and p99 is 8 seconds, an SLO on the mean will report a happy service while a quarter of your users watch a spinner. Prefer percentile-based SLIs, and where the workload varies wildly (batch jobs, long-tail endpoints, expensive queries), split the SLO by traffic class rather than averaging across all of it.

Why does an SLO matter?

Before SLOs, the reliability conversation between engineering and product was structurally broken. Product asked for features, ops asked for freezes, and whichever leader shouted loudest won that week. An SLO replaces that argument with a shared number, and does three things at once.

First, it turns "reliable" from an adjective into a quantity. A team that says "the service is reliable" is describing a feeling; a team that says "99.95% of checkout requests completed in under 400 ms last week, against a 99.9% target" is describing an object you can decide about. The moment reliability becomes a number, so does the trade-off against velocity.

Second, it converts stability-versus-speed into a policy, not a debate. The pre-agreed rule ("if we are under-budget on the SLO we ship; if we've blown through it we pause user-facing changes and fix reliability") takes the decision off the critical path. Nobody has to argue it in the middle of an incident. Executives don't have to overrule a nervous ops team. The circuit breaker fires on data.

Third, it makes reliability negotiable. A team can look at the SLO, decide the current target is too loose (users churn even when the number is green) or too tight (engineering is paying a tax on nines the business doesn't need), and change it deliberately, through a documented review, not silently in the middle of an outage. That renegotiation is where most of the value lives: an SLO that never moves is one nobody actually uses.

The honest limit of the tool: an SLO governs behaviour along the single dimension you chose to measure. A silently corrupted database write leaves the SLO at 100% and still ships a disaster; a slow-burn integrity bug won't trip a latency-based target for weeks. Pair the SLO with data-quality checks, smoke tests, and contract tests for anything users experience as "wrong" but not "slow" or "down".

How do popular tools handle SLOs?

No single tool owns the whole loop. An SLO needs a metrics pipeline (to gather the SLI), an SLO product (to compute compliance and burn rate), and a CI/CD system (to enforce the resulting policy at deploy time). The interesting question is which parts you buy, which you build, and where you hang the gate.

• Prometheus + Sloth + Grafana is the reference open-source stack. Sloth generates the Prometheus recording rules and multi-window burn-rate alerts straight from an SLO YAML file, Grafana renders the burn-down, and the SLI queries stay under version control alongside the service they measure. If you already run Prometheus and treat SLO definitions as code, this stack is difficult to beat on transparency or cost.
• Nobl9, Datadog SLOs, and Grafana Cloud SLO are commercial platforms that ingest SLIs from wherever your telemetry lives (Datadog, New Relic, CloudWatch, Splunk, Prometheus) and give you the SLO calculation, burn-rate alerts, and audit trail as a first-class product. They are the pragmatic pick when many services run on mixed backends and buying a single reliability pane of glass is cheaper than hand-rolling PromQL.
• Google Cloud Service Monitoring deserves a specific mention. If the workload lives on GCP and traffic runs through a Google load balancer or Cloud Run, the SLI is essentially free because GCP already emits the counter. Defining an SLO in the console gives you burn-rate alerts and a compliance graph out of the box; for a GCP-native team, this is often the fastest path from zero to a working SLO.
• Argo CD and Argo Rollouts with Prometheus AnalysisTemplates: if you're all-in on Kubernetes and progressive delivery, Argo Rollouts is the better fit here for enforcement. The rollout controller queries Prometheus for the current SLI mid-canary and aborts automatically when the burn rate spikes. No external gate to wire up; the reconciliation loop does the work, and the failure signal wires straight back to the exact revision that caused it.
• GitHub Actions, GitLab CI, and Jenkins don't own the SLO calculation, but any of them can gate a deploy on it. A required status check that queries the SLO API and fails the job when remaining budget is below a threshold makes SLO enforcement a merge-time property. In teams where branch protection is already how main is protected, that's a genuinely clean fit.
• Buddy is one solid pick when you want the SLO check to be the deploy gate and to see the pass/fail record in the same audit log as the deploy itself. A scheduled Buddy pipeline can pull the current SLI from Prometheus (or any SLO backend), a trigger_conditions clause on the deploy pipeline decides whether the next action runs based on the remaining budget, and the whole rule lives in the same YAML that ships the code. It doesn't compute the SLO itself (a monitoring product still does that), but the enforcement is one YAML file next to the deployment, which keeps the reliability policy readable to anyone who reads the pipeline.

Pick the combination that already owns your telemetry and your deploys. The biggest reliability win from SLOs is not the tool; it's writing the freeze policy down, and the thresholds it uses, before you need them.

Example

A scheduled Buddy pipeline that computes weekly SLO compliance by pulling the SLI from Prometheus and posts the result to a chat webhook. If the SLI fell below the SLO target, the same pipeline also opens an incident ticket. The SLO logic lives in one file; the numbers come from the metrics stack; humans get a Monday-morning summary instead of chasing dashboards.

# .buddy/slo-weekly.yml — compute weekly SLO compliance for the checkout service.
- pipeline: "weekly-slo-report"
  events:
    - type: "SCHEDULE"
      cron: "0 9 * * MON"
      timezone: "Europe/Warsaw"
  variables:
    - key: "SLO_TARGET"
      value: "0.999"
    - key: "SLI_QUERY"
      value: "sum(rate(http_requests_total{service=\"checkout\",code=~\"2..|3..\"}[7d])) / sum(rate(http_requests_total{service=\"checkout\"}[7d]))"
    - key: "PROM_URL"
      value: "https://prom.example.com/api/v1/query"
  actions:
    - action: "Compute weekly SLI"
      type: "BUILD"
      docker_image_name: "alpine"
      docker_image_tag: "3.20"
      commands: |-
        apk add --no-cache curl jq
        SLI=$(curl -fsS --data-urlencode "query=$SLI_QUERY" "$PROM_URL" | jq -r '.data.result[0].value[1]')
        printf "Rolling 7d SLI: %s (target %s)\n" "$SLI" "$SLO_TARGET"
        awk "BEGIN { exit !($SLI >= $SLO_TARGET) }" \
          && echo "OK: SLO met" \
          || (echo "BREACH: SLI under target" && exit 1)

    - action: "Post weekly report to reliability channel"
      type: "HTTP"
      method: "POST"
      trigger_time: "ON_EVERY_EXECUTION"
      notification_url: "https://chat.example.com/hooks/reliability-weekly"
      headers:
        - name: "Content-Type"
          value: "application/json"
      content: |-
        {
          "text": "Weekly SLO check for checkout: target ${SLO_TARGET}, run ${BUDDY_EXECUTION_ID}."
        }

    - action: "Open incident on SLO breach"
      type: "HTTP"
      method: "POST"
      trigger_time: "ON_FAILURE"
      notification_url: "https://oncall.example.com/api/incidents"
      headers:
        - name: "Content-Type"
          value: "application/json"
      content: |-
        {
          "title": "checkout SLO breach (weekly window)",
          "urgency": "high",
          "service": "checkout"
        }

The useful property here is where the reliability decision lives. The SLO target and the SLI query are two variables at the top of the file, so a change in reliability policy is a pull request a product manager can review, not a mysterious dashboard edit nobody can trace. The pipeline never computes the SLO from raw log lines (that's what the metrics stack is for); the decision, and the alert that follows, sit inside version control alongside the code the SLO applies to. When the SLI recovers on the next run, the incident action stops firing on its own, without a manual override.