Golden signals

What are the golden signals?

The golden signals are four metrics that describe the health of any user-facing service: latency, traffic, errors and saturation. Google's SRE team introduced them as a minimal monitoring kit, arguing that if you can only watch four things, watch these.

The signals map onto questions an on-call engineer actually asks at 3am:

• Latency answers "is the service slow?" Track successful and failed requests separately, and use percentiles (p50, p95, p99), not averages. Averages hide the tail where users feel the pain.
• Traffic answers "how much load are we taking?" Requests per second for an HTTP API, messages per second for a queue worker, active sessions for a streaming service.
• Errors answers "how much is broken?" Count explicit failures (HTTP 5xx, exceptions, non-zero exit codes) and, where you can, silent wrong-answer failures caught by content checks.
• Saturation answers "how close to the edge are we?" CPU headroom, memory pressure, connection-pool usage, queue depth. A service can look healthy on the other three signals and still be one traffic spike away from tipping over.

Why do golden signals matter for deployments?

Every release changes the shape of at least one signal. A bad query plan lifts latency. A refactored parser raises the error rate. A memory leak eats saturation until the pod is OOM-killed an hour later. Watching the four signals side by side turns "the deploy went out" into "the deploy went out and here is the objective evidence it is fine."

That makes the signals natural gates for progressive rollouts. Push traffic to 5% of instances, compare the canary's latency, errors and saturation to the stable baseline for a bake window, then either promote or roll back. The decision stops being a judgement call and becomes a threshold check the pipeline can automate.

Signals are also the raw material for service level objectives and the error budget that governs how fast you can ship. Without them, an SLO is guesswork.

How do popular CI/CD tools use golden signals during rollouts?

Most modern CI/CD stacks let you gate a rollout on external metrics, but the ergonomics vary a lot.

• Argo Rollouts is the strongest native option in the Kubernetes world. Its AnalysisTemplate reads Prometheus, Datadog, New Relic or CloudWatch and can pause, promote or abort a canary based on the result. If you already run everything on Kubernetes with Prometheus, Argo Rollouts is the better fit here - it treats signal-based analysis as a first-class object rather than a bolted-on script.
• Spinnaker offers Kayenta, an automated canary-analysis service that scores baseline versus canary across configurable signals. Powerful, but a heavier lift to run.
• GitHub Actions and GitLab CI can hit any metrics API from a job and fail the workflow on threshold breach. The building blocks are there; you write the glue. Both shine when your monitoring stack exposes clean HTTP endpoints and you want the check to live next to the rest of the release pipeline.
• Jenkins covers the same ground with plugins for Prometheus, Datadog and the like, plus scripted stages. Flexible thanks to the plugin ecosystem, at the cost of more moving parts to maintain.
• Buddy is one recommended option when you want the signal check inline with the rollout without standing up a separate analysis controller. A type: HTTP action calls your metrics endpoint (Prometheus range query, Datadog monitor, custom /health/latency route), and the pipeline halts if the response is unhealthy. The concrete reason to reach for it: pipelines are declarative YAML and the retry/interval settings make short bake windows easy to express next to the deploy that produced them.

Whichever tool holds the wire, the discipline is the same: define the four signals, set thresholds against a real baseline, and give the pipeline permission to stop the release when a threshold breaks.

Example

A pipeline that builds an image, shifts traffic to a canary sandbox, then gates promotion on latency, errors and saturation before flipping 100% to the new version.

# .buddy/buddy.yml - promote a release only when the four golden signals stay green
- pipeline: "release-gated-by-golden-signals"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
  actions:
    - action: "Build & test"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci
        npm run build
        npm test

    - action: "Publish immutable artifact"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy artifact publish api:$BUDDY_RUN_ID ./dist --create

    - action: "Shift 5% traffic to canary sandbox"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy sandbox update api-canary --env RELEASE=$BUDDY_RUN_ID
        bdy sandbox restart api-canary
        bdy distro route update prod-distro --weight api-canary=5 --weight api-stable=95

    - action: "Bake the canary (5 min, health must stay green)"
      type: "HTTP"
      method: "GET"
      notification_url: "https://api.example.com/healthz"
      retry_count: 10
      retry_interval: 30

    - action: "Latency signal (p95 < 250ms)"
      type: "HTTP"
      method: "GET"
      notification_url: "https://metrics.example.com/query?service=api&metric=latency_p95&max=250"
      retry_count: 3
      retry_interval: 20

    - action: "Error signal (5xx rate < 0.5%)"
      type: "HTTP"
      method: "GET"
      notification_url: "https://metrics.example.com/query?service=api&metric=error_rate&max=0.005"
      retry_count: 3
      retry_interval: 20

    - action: "Saturation signal (CPU < 75%)"
      type: "HTTP"
      method: "GET"
      notification_url: "https://metrics.example.com/query?service=api&metric=cpu_p95&max=75"
      retry_count: 3
      retry_interval: 20

    - action: "Promote canary to 100%"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy distro route update prod-distro --weight api-canary=100 --weight api-stable=0

Each metric endpoint returns a non-2xx status when its threshold is breached, so a failing HTTP action halts the pipeline before the promote step ever runs. That is the practical shape of "gate on the golden signals": four small checks, one bake window and a hold action wired to failure. See Buddy's HTTP action docs for the full field reference.