Chaos engineering

How does chaos engineering work?

A chaos experiment is a controlled test of a hypothesis about a running system. The team picks a metric they can measure continuously (the "steady state") and predicts it will hold when a specific fault is injected. Then they inject the fault in the smallest realistic scope, watch the metric, and compare it against the pre-experiment baseline. If steady state breaks, they have found a real reliability weakness. If it holds, they have earned evidence that this failure mode is handled.

Most teams follow the shape codified in the Principles of Chaos Engineering:

1. Define steady state. A business-visible signal, not a technical detail. Checkout success rate, requests-per-second, p99 latency, the number of active sessions. If you cannot measure it, you cannot run an experiment against it.
2. Hypothesise. "With the payments cache node killed, checkout success stays within 1% of baseline." A specific, falsifiable prediction.
3. Inject the fault. Realistic and bounded: kill one pod, add 300 ms of latency to one dependency, drop 5% of packets on one link. Use the smallest scope that could disprove the hypothesis.
4. Verify and learn. Compare the metric against baseline. A broken steady state is not a failed test, it is new information: file a reliability ticket, fix the weakness, re-run the experiment.

The catalogue of faults teams inject is wide: process kills, container restarts, node terminations, CPU and memory pressure, disk-full conditions, clock skew, DNS failure, dependency latency, network partitions, region outages, expired credentials, slow-response injection into third parties. Each maps to a real production incident someone has already had.

What separates chaos engineering from failure testing or fault injection?

The confusion is fair. The mechanics look the same.

• Failure testing exercises a specific code path against a specific fault, usually in isolation. Unit or integration level. Deterministic. Green or red.
• Fault injection is the technique: any deliberate perturbation of a system's inputs, timing or resources. Chaos engineering uses fault injection; so do fuzzers, so do property tests.
• Chaos engineering is fault injection at the system boundary, against a live system, driven by a hypothesis about emergent behaviour. It looks for the failure modes nobody wrote a test for: the retry storm, the thundering herd, the timeout mismatch between two services that only surfaces when the network is slow.

That last property is why the practice matters. Distributed systems fail in ways individual components pass their own tests. You cannot reproduce a cascading timeout with a unit test on the service that started it.

Why does chaos engineering matter for deployment reliability?

Every CI/CD improvement optimises for change velocity. Chaos engineering optimises for the cost of a bad change: the blast radius and the MTTR when something eventually breaks in production. Those two levers move the same DORA numbers as deployment frequency does; they just move them from the other side.

Three concrete pay-offs teams see:

• Latent bugs surface before customers do. The missing timeout, the retry with no jitter, the health check that returns 200 while the backend is dead: these hide in normal traffic and only appear under stress. A weekly chaos experiment finds them on a Tuesday morning instead of during Black Friday.
• Runbooks get exercised. Chaos experiments generate real incidents at planned times, which is the only reliable way to keep on-call runbooks (and the people running them) sharp. A runbook nobody has executed in six months is fiction.
• Error budget spend becomes deliberate. Because the failure is scheduled and bounded, it burns budget you chose to burn, not budget an outage stole. That earns permission for higher change velocity elsewhere.

There is a real trade-off to name. Running chaos in production requires trust, an abort mechanism, and buy-in from anyone whose pager can go off. Teams that skip the political groundwork and just start killing pods lose the mandate to keep doing it after the first user-visible incident. Start in staging, publish steady-state metrics, tell stakeholders when and where the next experiment will run.

How do popular tools handle chaos engineering?

Chaos tooling splits along the stack, and no single tool covers every layer.

• Chaos Monkey (Netflix) is where the practice started. It kills random instances in a service group during business hours, on the theory that any resilient service should tolerate it. Narrow scope, huge cultural impact: the tool most credited with normalising planned failure.
• Gremlin is the commercial platform. Wide fault library (CPU, memory, disk, network, shutdown, DNS, latency, packet loss), a safety net that halts experiments when steady state degrades, SSO and audit trails. Expect to pay per host, but you get an operations story you would otherwise build yourself.
• Chaos Mesh and LitmusChaos are the CNCF-graduated Kubernetes-native options. They ship as CRDs, integrate with kube-scheduler and admission webhooks, and cover pod, network, IO, kernel, JVM and time faults. If your workloads live on Kubernetes, these are the better fit than anything else on this list: they know your cluster's topology, they respect its RBAC, and the experiments version-control alongside your manifests.
• AWS Fault Injection Simulator and Azure Chaos Studio are the cloud-native equivalents. If your outage risk lives at "an AZ goes away" or "an EBS volume degrades", the cloud providers have direct hooks the third-party tools have to approximate.
• Toxiproxy (Shopify) and Pumba are lightweight, single-purpose tools. Toxiproxy sits between two services and simulates specific TCP faults; Pumba does the same for Docker networks. Both are ideal in CI: cheap, deterministic, easy to script.
• Jenkins, GitHub Actions and GitLab CI run any of the above as pipeline steps. They handle the orchestration, the scheduling and the notification; the actual fault injection is delegated to whichever chaos tool you picked.

Buddy is one recommended option for teams whose stack is not Kubernetes-shaped: Docker Compose services, VMs, PaaS backends, mixed estates. A Buddy pipeline can pull a chaos tool image (Pumba, Toxiproxy, a bespoke script), inject the fault as a BUILD action, run an HTTP steady-state probe between actions, and stop the run on a broken hypothesis. Scheduled pipelines double as game-day drivers, and the pipeline file lives in the same repo as the app code, so the experiment history version-controls itself. It is not the tool to pick if you want the broadest fault library out of the box; Gremlin and Chaos Mesh have more variety. It is a straightforward way to run chaos experiments alongside the rest of a delivery workflow.

The honest read: pick the chaos tool for the layer you fear most (Kubernetes-native, cloud-native, network-only), then pick whatever pipeline runner you already have to schedule and gate it. The pipeline runner is the boring choice; the fault library is the load-bearing one.

Example

The pipeline below runs a chaos experiment against a staging sandbox: baseline a health metric, inject 300 ms of network latency into the web container for two minutes with Pumba, verify the app recovers, and record the run. The experiment is bounded (staging, one target service, two-minute duration), the abort mechanism is Buddy's own "stop run" button, and the whole configuration lives in the app repo.

# .buddy/buddy.yml - latency chaos experiment against staging
- pipeline: "chaos-latency-web"
  events:
  - type: "PUSH"
    refs:
    - "refs/heads/chaos/*"
  actions:
  - action: "Baseline steady state"
    type: "HTTP"
    method: "GET"
    notification_url: "https://staging.example.com/healthz"
    retry_count: 3
    retry_interval: 10
  - action: "Inject 300ms latency to web container for 2 min"
    type: "BUILD"
    docker_image_name: "gaiaadm/pumba"
    docker_image_tag: "latest"
    commands: |-
      pumba netem --duration 2m --tc-image gaiadocker/iproute2 delay --time 300 --jitter 50 re2:^staging-web
  - action: "Verify app recovers within 3 min"
    type: "HTTP"
    method: "GET"
    notification_url: "https://staging.example.com/healthz"
    retry_count: 10
    retry_interval: 18
  - action: "Record experiment result"
    type: "BUILD"
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      echo "chaos-latency-web passed at $(date -u +%FT%TZ)" >> /buddy/data/chaos-log.txt

If the "Verify app recovers" step fails, the pipeline stops in place: the run status becomes the finding, not the noise. File the ticket, fix the retry or timeout, push again to the chaos branch, re-run. That loop is the whole practice. Every green run is confidence, every red run is a bug you would otherwise have shipped.