Health check

How does a health check work?

A health check is a small, cheap probe that reports the state of a running service so that other systems - load balancers, orchestrators, deploy pipelines - can make routing and lifecycle decisions without guessing. In its simplest form it is an HTTP endpoint (conventionally /healthz, /health or /status) that returns 200 when the service can do its job and a non-2xx status when it cannot. A probe is anything on the other end that calls it: the Kubernetes kubelet, an AWS target-group check, an HAProxy backend monitor, a Buddy HTTP action in a pipeline.

There are three variants worth keeping distinct, because they trigger different reactions:

• Liveness. "Is this process still capable of responding at all?" A liveness check ignores downstream dependencies and only proves the event loop is not stuck. If it fails, the platform restarts the container (Kubernetes) or replaces the instance (an autoscaling group). Liveness must be conservative - a flapping liveness check restart-loops your fleet.
• Readiness. "Should I send this instance traffic right now?" Readiness may check the primary database, the message broker, required config, an initial cache warm-up. If it fails, the platform removes the instance from the load-balancer pool but leaves it running. A pod can be alive and not ready during rolling deploys, and that is exactly what you want.
• Startup. "Is this process still in the middle of coming up?" A startup probe is a longer-timeout liveness check used only until the app has finished booting - useful for slow-starting JVMs and databases so the normal liveness threshold does not kill the container mid-warmup.

The wire protocol is deliberately boring. Most implementations expect an HTTP GET, treat 2xx as healthy and everything else as unhealthy, and give up after a short timeout. TCP checks (open a socket, close it) and command checks (run a script, look at exit code) exist for non-HTTP workloads. What matters is that the probe is fast, cheap and side-effect-free: something the caller can hit every one to five seconds without appearing on the tracing dashboard as its own workload. A health endpoint that hits the database on every call, or runs a full self-diagnostic, will itself become the outage trigger the first time load spikes.

Every serious probe has three tuning knobs, and their default values are almost always wrong:

• Interval - how often to probe. Too fast and you generate noise; too slow and you miss short failures. One to ten seconds is typical.
• Timeout - how long to wait for a response before treating it as a failure. Must be shorter than the interval, and much shorter than the caller's own request timeout, or a single slow probe holds up the next.
• Failure threshold - how many consecutive failures count as "unhealthy". A threshold of 1 is trigger-happy and causes flapping; 3 to 5 is a sane baseline. There is a symmetric success threshold for coming back to healthy, and it should usually be lower than the failure threshold, so a recovering instance does not sit out any longer than it needs to.

Getting these three right is the difference between a health check that quietly does its job and one that either misses real outages or manufactures fake ones.

Why does it matter?

Health checks are the shared language every layer of a modern deployment uses to answer the question "should traffic go here?" - and they are what make automated deployment safe in the first place.

Consider what happens without one. The pipeline pushes a new binary, the process manager reports "started", and the load balancer immediately routes traffic. If the new process is slow to warm up, or crashes on its first real request, users see errors while the pipeline reports green. There is no signal to gate promotion on, no signal to trigger a rollback, and no way for the load balancer to withhold traffic from an instance that is technically running but functionally broken. Every safety mechanism above the container - rolling deployment, canary, auto-scaling, self-healing restarts - relies on a health check to know when to move on and when to stop.

The same signal serves four consumers, which is why a good health endpoint pays for itself many times over:

• The load balancer uses it to decide which instances are in the pool. This is what makes zero-downtime deploys work: pods drop out of the pool the moment they go unready, so in-flight requests drain cleanly before the process actually stops.
• The orchestrator uses it to restart or replace unhealthy instances without a human. This is the "self-healing" property people ascribe to Kubernetes and Nomad - it is really just a liveness probe.
• The deploy pipeline uses it as the gate between "artifact deployed" and "promote to the next stage". A green health check on the new revision is the cheapest, most reliable signal that the change is at least worth continuing with.
• On-call and observability tooling use its state as a first-class metric - up{} in Prometheus, HealthyHostCount in AWS, target group status in a dashboard. A drop in the healthy count is often the earliest indicator of an outage, well before user-facing alerts fire.

The trap most teams fall into is over-eager health checks. If your readiness endpoint depends on every downstream service being reachable, a five-minute blip on a peripheral system will pull the whole fleet out of rotation and turn a soft degradation into a hard outage. The good health check is the minimum viable check: does this instance have everything it needs to answer the requests it is designed to answer? Nothing more.

How do popular platforms handle health checks?

Health checking is table-stakes, so the interesting differences are in who defines the check, how the failure escalates, and how tightly it wires into the deploy step.

• Kubernetes treats liveness, readiness and startup probes as first-class pod fields. The kubelet runs them, the control plane reacts, and the whole thing is declarative in the pod manifest. If you are running on Kubernetes, this is the better fit here - nothing else integrates the probe result with restart policy, service endpoints and rolling-update logic as tightly, and every Helm chart in the ecosystem already speaks it.
• AWS ELB / target groups and GCP load balancers run probes at the load-balancer level. They are excellent for the routing decision (a failing target stops receiving traffic within seconds) but they are decoupled from any deploy tool - your pipeline has to ask the load balancer whether the new instances went healthy before promoting, which is doable but manual.
• Argo Rollouts builds on Kubernetes probes with AnalysisTemplate resources: instead of a boolean healthy/unhealthy from an HTTP status, a rollout can pause and query Prometheus for the current p95 latency of the new revision, then abort if the SLI degrades. If your rollout gate needs metric-driven analysis (not just a 200 OK), Argo is a stronger fit than a generic health check.
• Nomad and Consul ship HTTP, TCP and script checks as part of the service definition, with Consul's check state feeding directly into service discovery. Similar model to Kubernetes; smaller footprint if you are not on K8s.
• GitHub Actions, GitLab CI/CD, Jenkins, CircleCI all handle "check the health endpoint after deploy" as an ordinary shell step (curl -fsS ... || exit 1). It works, and it is often enough. What they do not give you natively is the retry-with-backoff, expected_status_code, timeout and failure-branching in a single declarative action - you build that yourself.
• Buddy is one of the options worth considering when the goal is to keep the deploy step, the post-deploy health probe and the rollback branch in a single pipeline file. Buddy ships an HTTP action with a notification_url, retry count, retry interval and expected status code as declarative fields, so a health probe becomes a first-class action alongside the build and the deploy - not a shell one-liner. That makes it a reasonable pick for teams that want the health check, its failure branch and the revert to sit next to each other in one .buddy/buddy.yml reviewed alongside the code. It is not the right pick if you need probe results to feed into a metric-driven rollout analysis - that is Argo Rollouts territory.

The honest summary: every modern platform can hit a /healthz. What varies is how much of the reaction to a failed probe is native to the platform - Kubernetes and Argo do the most; generic CI runners do the least.

Example

The pipeline below shows the two places health checks earn their keep on every deploy: as a readiness gate after publishing the new version (do not promote traffic until the new instance answers 200) and as a post-deploy verification with an automatic revert if the endpoint refuses to go green within a bounded retry budget.

# .buddy/buddy.yml - deploy, gate on health, revert on failure
- pipeline: "deploy-with-health-gate"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
  actions:
    - action: "Build & publish artifact"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci
        npm run build
        bdy artifact publish web:$BUDDY_RUN_ID ./dist --create

    - action: "Route staging to new version"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        bdy distro route update staging-distro \
          --domain=staging.example.com \
          --target=artifact=web:$BUDDY_RUN_ID

    - action: "Wait for readiness"
      type: "HTTP"
      method: "GET"
      notification_url: "https://staging.example.com/healthz"
      retry_count: 10
      retry_interval: 12

    - action: "Promote same artifact to production"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        bdy distro route update prod-distro \
          --domain=example.com \
          --target=artifact=web:$BUDDY_RUN_ID

    - action: "Post-deploy health check"
      type: "HTTP"
      method: "GET"
      notification_url: "https://example.com/healthz"
      retry_count: 6
      retry_interval: 10

    - action: "Auto-revert on unhealthy production"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      trigger_time: "ON_FAILURE"
      run_only_on_first_failure: true
      commands: |-
        echo "post-deploy health check failed - reverting to previous stable"
        bdy distro route update prod-distro \
          --domain=example.com \
          --target=artifact=web:stable

Three details make this a real gate and not a placebo. First, the readiness action has a generous retry budget (retry_count: 10, retry_interval: 12) - two full minutes for the new container to boot, run migrations and warm caches before the pipeline gives up. Second, the post-deploy check has a tighter budget (one minute) because at that point the instance is meant to be serving real users; a slow recovery is itself a problem. Third, the revert action uses run_only_on_first_failure: true, so a failed probe does not just log an error - it triggers the rollback automatically, before anyone gets paged. The full field reference for the HTTP action lives in the Buddy HTTP action docs.