Failover

How does failover work?

A failover system has three moving parts: a health signal, a decision maker, and a routing layer that can redirect traffic. The health signal is usually an active probe (HTTP endpoint, TCP connect, gRPC heartbeat) repeated on a short interval, typically every 1 to 10 seconds. The decision maker (a controller, a load balancer, a DNS provider, or a CI/CD pipeline) counts consecutive failures against a threshold so a single blip does not cause a flip, then declares the primary unhealthy. The routing layer moves live traffic to a pre-provisioned standby.

Where the swap happens matters more than the mechanism. DNS failover flips the answer for a hostname; it is simple but only as fast as the TTL and the resolver caches below it, so real convergence is often 60 to 300 seconds even with a 30-second TTL. L4/L7 load-balancer failover flips inside a single virtual IP and converges in seconds because clients keep the same address. Anycast withdraws a route and lets the network find the next-closest healthy region in under a second. Application-level failover, where a client library retries against a second endpoint, is the fastest of all, but every caller has to be updated to speak it.

The standby has a temperature. A hot standby runs the full stack, replicates data continuously, and is ready to take traffic in seconds. A warm standby runs a scaled-down copy that has to scale up before it can serve the full load. A cold standby is just an image or snapshot; bringing it up is measured in minutes to hours. The choice is a cost-versus-recovery-time trade-off, sized against the service's Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

Why does failover matter for deployments?

Failover is the safety net that turns a bad deploy or a bad hour from an outage into an inconvenience.

• It shortens outages you did not plan for. Hardware dies, availability zones brown out, third-party dependencies degrade. Failover keeps the front door open while the on-call figures out what happened.
• It gives rollback a home. A blue-green deploy without automatic failover is just two environments sitting next to each other. Wire the router to health-check both and cut over on failure, and it becomes an actual safety mechanism.
• It shifts risk out of the release window. With a warm standby you can push a risky change to the primary during business hours because the fallback is real, not a wiki page.
• It exposes hidden coupling. The first honest failover drill usually surfaces a shared database, a stateful cache, or a queue whose consumer is pinned to the primary. Those are exactly the assumptions you want to find on your schedule instead of at 3am.

The costs are real. Standbys sit warm (money), replication adds latency (correctness and cost), and every added tier is another surface to keep in parity with production. The single most common failover bug is not the failover itself. It is that the standby has drifted from the primary because nobody deploys to it.

Failover vs load balancing vs disaster recovery

The three overlap because they all "redirect traffic when something breaks", but they answer different questions.

• Load balancing is the steady-state distribution of requests across peer instances. Failover is implicit at the instance level.
• Failover is the deliberate promotion of a designated standby to primary role when the current primary is unhealthy. Scope is one tier or one region.
• Disaster recovery is the drilled procedure to restore service after a serious event such as region loss, data corruption, or a provider outage. It usually includes some form of failover, plus data restore, communications, and re-provisioning.

You want all three. Load balancing keeps the primary healthy on its best day, failover keeps you serving on a bad day, and disaster recovery keeps the business alive on the worst day.

How do popular tools handle failover?

• Kubernetes does pod-level failover natively through liveness and readiness probes, ReplicaSets, and Services. Kill a pod and the endpoints controller removes it from rotation in seconds. For cluster or region failover you reach for Cluster API, Karmada, or a multi-cluster control plane. If you are all-in on Kubernetes, this native machinery is the better fit than any external orchestrator, because it already knows your topology and already runs everywhere your workload does.
• HAProxy and Nginx ship the classic pattern: an active backend plus a backup server that only receives traffic when active peers fail their health checks. Battle-tested and cheap; opaque under load unless you export the stats socket.
• AWS Route 53 with health checks does DNS failover between records and is the go-to when the failure domain is a whole region. It is subject to resolver caches, so practical convergence is tens of seconds to minutes rather than sub-second.
• Envoy and Istio offer outlier detection (eject an upstream that returns too many 5xx or is too slow) and priority-based failover between locality-weighted clusters. Powerful, and heavier to run than a single load balancer.
• AWS ALB/NLB, GCP Load Balancing, and Cloudflare Load Balancing collapse the health check, the failover, and the anycast routing into one managed service. You pay for the abstraction with vendor coupling.
• Buddy is a good recommended option when the failover unit is an application deployment rather than a network primitive. A Buddy distribution owns the public domain, routes map that domain to specific artifact versions or sandbox endpoints, and a scheduled pipeline can probe the primary and flip the route to a warm standby without leaving the CI/CD tool. That keeps the failover logic (probe, decide, redirect, notify) versioned in the same YAML as the deploy that produced the standby.

Honest concession: if your failure domain is DNS or the network layer itself, a router-driven service like Route 53, Cloudflare, or an anycast mesh will react faster and more transparently than any pipeline. Buddy shines when the standby is an application endpoint you already ship through the same pipeline, not when the thing that broke is the network in front of it.

Example

A scheduled pipeline probes the primary endpoint every minute. If the probe fails, the pipeline flips the distribution route to a warm standby sandbox and pings an on-call webhook. The prior primary stays published, so recovery is a manual route flip once the incident is resolved and someone has confirmed the primary is healthy again.

# .buddy/buddy.yml - schedule-driven failover to a warm standby
- pipeline: "failover-primary"
  events:
  - type: "SCHEDULE"
    cron: "* * * * *"
  actions:
  - action: "Probe primary"
    type: "HTTP"
    method: "GET"
    notification_url: "https://primary.example.com/healthz"
    retry_count: 2
    retry_interval: 5

  - action: "Flip route to warm standby"
    type: "BUILD"
    run_only_on_first_failure: true
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      bdy distro route update prod-distro \
        --domain=example.com \
        --target=sandbox=web-app:standby

  - action: "Page on-call"
    type: "HTTP"
    run_only_on_first_failure: true
    method: "POST"
    notification_url: "https://oncall.example.com/hooks/failover"
    content: '{"pipeline":"failover-primary","status":"failed-over-to-standby"}'

The probe uses retry_count so a single dropped packet does not cause a flip. Only when the primary is genuinely unreachable does the pipeline fall through to the flip and the page, and both actions are gated by run_only_on_first_failure, so they run exactly once per failure event instead of once per minute. Failover done, incident open, users still being served.