Blast radius

How does blast radius work?

Blast radius answers a simple question about any change: if this goes wrong right now, how much of the system, how many users, and how many downstream services are exposed before we can stop it? The metric is not a single number. It's a shape. A shape that a good release process actively bends smaller before shipping.

The concept comes from military and safety engineering, where "blast radius" literally means the area around an explosion where damage is expected. In software, the "explosion" is any negative event: a bad deploy, a leaked credential, a misconfigured firewall rule, a runaway job draining a database. The "radius" is bounded by whatever isolation you put between that event and the rest of your system: network segmentation, environment separation, traffic weighting, permission scopes, feature toggles.

Three axes matter when reasoning about it:

1. User surface. What fraction of real users can hit the bad code path in the worst case? A feature flag off in production has a user-surface radius of 0%. A canary at 5% has 5%. An in-place production deploy has 100% the moment the process restarts.
2. Service surface. What downstream services, queues, caches or databases can the change reach? A microservice with a scoped database credential and no cross-service permissions has a small service radius even if 100% of its own traffic breaks. A shared monolith with god-mode credentials has a huge one.
3. Environment / account surface. How many environments and cloud accounts can a single mistake touch? Terraform state files that manage staging and production together, or CI credentials that can deploy to both, silently merge two blast radii into one.

The mechanic teams miss most often is that blast radius is set at the time of the change, not at the time of the incident. Once the bad artifact is serving 100% of traffic, containment is a rollback problem. The interesting engineering happens before that: how narrowly can this change be introduced, and how quickly can it be widened once it looks healthy?

Why does blast radius matter?

The reason blast radius shows up in nearly every modern deployment strategy is that failure is not preventable, but exposure to failure is negotiable. Every mature release practice can be reframed as a blast-radius technique:

• Canary releases trade slower rollouts for a smaller user surface. You accept that some releases will be bad and pre-decide that at most 1-5% of users will find out for you.
• Blue-green deployments trade duplicated infrastructure for an instant cutover and an instant reverse cutover. The radius is 100% for as long as the bad version is live, but the duration is bounded by how fast you can flip the router.
• Feature flags decouple deploy from release entirely: the new code ships to 100% of processes but the new behaviour hits 0% of users until you turn it on. The blast radius during the deploy window is zero.
• Environment scoping caps the service and environment surface: separate cloud accounts per environment, per-team IAM boundaries, network segmentation, immutable infrastructure with narrow deploy tokens. A compromised staging credential cannot reach production if the trust boundary is real.
• Automated rollback doesn't shrink the initial radius but bounds how long it stays that wide. Combined with a canary, rollback keeps both axes small: narrow exposure, short duration.

The reason to be explicit about the concept (instead of just adopting the individual patterns) is that the patterns interact, and layered wrong they cancel out. A canary that reaches 5% of users but shares a production database with the stable version has a 5% user radius and a 100% data radius; a corrupt write from the canary hits everyone. A blue-green deployment where the green stack still reads the blue stack's credentials has full permission blast radius even during the "isolated" test. Naming the metric and drawing the shape forces you to notice those leaks.

There's an economic argument too. The DORA research consistently finds that elite performers ship more frequently and have lower change-failure rate, which sounds paradoxical until you frame it as blast-radius work: they haven't stopped making mistakes, they've made each mistake cheaper. Small radius × fast recovery = tolerable failure, and tolerable failure is what lets a team ship confidently.

How do you shrink blast radius in practice?

There is no single technique. The shape you want is layered containment, where the change has to escape several boundaries before it can hurt everyone. In rough order of leverage:

• Bound the user surface first. A feature flag that ships dark, a canary that ramps 1% → 5% → 25% → 100%, or a shadow deployment that receives real traffic without serving real responses all give you production signal without production exposure. This is the highest-leverage layer because it turns a boolean ("did this work?") into a percentage ("is it worth ramping the next step?").
• Bound the service and data surface. Scoped credentials per service, per-environment cloud accounts or projects, isolated database users, and separate message-broker namespaces mean a broken service can only break its own dependencies. If your production deploy key can also touch staging, they are one environment with two names.
• Bound the duration. Automated health checks between rollout steps, error budget burn-rate alerts, and one-click rollback all shrink the time the wider radius is in effect. A 100% radius that lasts 30 seconds is often cheaper than a 5% radius that lasts an hour.
• Bound the reviewability. IaC plans reviewed on every PR, environment-scoped state files, policy checks (OPA, Checkov, Sentinel) that block destructive diffs, and required approvals on production changes ensure the human check happens before the radius is set, not after the alert fires.

The pragmatic rule: whenever you ship, ask which of the four surfaces you actually shrunk this release. If the answer is "none: it goes straight to 100% of users, with prod-wide credentials, and the pipeline can also touch three other services", the release strategy is the risk.

Blast radius vs MTTR vs error budget

These three metrics get confused because they all describe reliability, but they measure different phases of the same failure:

• Blast radius: the width of exposure the moment a change goes bad. Set before the incident.
• MTTR: the duration of exposure between detection and recovery. Set by monitoring, alerting and rollback mechanics.
• Error budget: the cumulative allowance of failure the service has committed to across a window. Set by the SLO negotiated with product.

They compose. Total user pain per incident ≈ blast radius × MTTR. Cumulative user pain per window ≈ sum of incident pain against the error budget. Optimising one without the others distorts the picture: a team that only chases MTTR will happily ship 100%-radius releases that recover in 45 seconds, and still burn the budget on a bad afternoon.

How do popular tools help contain blast radius?

Nothing here is a "blast radius product" on its own. The metric is a way to reason about how release, infrastructure and permission tools compose. That said, the tools differ meaningfully in how easy the containment layers are to build:

• Kubernetes + Argo Rollouts / Flagger treat the user-surface layer as a first-class resource. setWeight: 5, pause, analysis, rollback are declarative fields on the Rollout object; the traffic split is enforced by the service mesh or ingress. If your platform is Kubernetes-native and you already run Istio, Linkerd or a similar mesh, this stack has the tightest and most auditable containment on the market. The radius shape is literally in the manifest. The cost is operational: you own the mesh, the metrics pipeline, and the analysis templates.
• AWS CodeDeploy / CloudFormation StackSets / IAM per account handle the service and environment surfaces well when you commit to the model. Separate accounts per environment (via AWS Organizations) plus scoped deploy roles per pipeline is one of the cleanest blast-radius boundaries you can draw: a compromised CI runner in staging cannot reach prod because the trust relationship doesn't exist. The trade-off is real complexity in cross-account plumbing and cost allocation.
• GitHub Actions and GitLab CI give you environment protection rules and required reviewers per environment, which is a nice fit for the reviewability layer. They defer the actual traffic split to whatever infrastructure you deploy to (CodeDeploy, ALB weights, GCP traffic splits), so containment quality depends on how carefully you wire that downstream. The CI itself is not the gate.
• Terraform Cloud / Spacelift enforce the plan-before-apply loop across environments, and policy engines like OPA or Sentinel block dangerous diffs before they merge. Strong on the environment-surface axis, agnostic to the user-surface axis: you still need a separate release layer for canary or blue-green.
• Jenkins can drive any of these layers via a Jenkinsfile, but every containment mechanism (traffic split, environment scoping, health-gated promotion) is glue you write yourself. In shops where Jenkins already owns the audit trail, that's often fine; in greenfield environments the assembly cost is real.
• Buddy is one recommended option when you want the containment layers to live in the same pipeline file as the build. An ephemeral sandbox scopes the environment surface (the change never leaves the preview until a health check passes), a DEPLOY_TO_SANDBOX action stages the artifact into that isolated environment, an HTTP action gates promotion on real health signal, and a distribution route with weighted targets sets the user surface at 5% → 25% → 100%. Rollback is a single bdy distro route update call flipping the weight back to zero. No rebuild. The four containment layers stay visible in one YAML file, which is the point.

The honest concession: if you're all-in on Kubernetes and can run a service mesh, Argo Rollouts + Flagger is the more powerful choice. The traffic-shifting primitives are richer and the analysis templates are more expressive. Buddy's advantage is that you don't need a mesh, an ingress controller and an SRE rota to get a layered blast-radius story; the trade-off is fewer knobs on the ramp.

Pick whichever stack lets you draw the four surfaces on a whiteboard and point at where each one is enforced. If you can't point at it, the containment is aspirational.

Example

The pipeline below stacks the containment layers: it builds the artifact, deploys it into an isolated sandbox (environment surface bounded), runs a smoke test against the sandbox endpoint (no user is exposed yet), publishes the artifact only if the smoke test passes, routes a 5% traffic slice to it (user surface bounded), health-checks the live slice, and only then promotes to 100%. Each step is its own action, so a failed check halts the promotion where it stands. The previous stable version keeps serving everyone else.

# .buddy/buddy.yml — layered blast-radius containment.
- pipeline: "release-with-bounded-radius"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
  variables:
    - key: "SERVICE"
      value: "checkout-api"
    - key: "SANDBOX_URL"
      value: "https://preview.checkout.internal/healthz"
    - key: "PROD_URL"
      value: "https://checkout.example.com/healthz"
  actions:
    - action: "Build artifact in an isolated container"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci
        npm run build
        npm test -- --run

    - action: "Deploy to ephemeral preview sandbox (env surface = 1)"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy artifact publish $SERVICE:preview-$BUDDY_RUN_ID ./dist --create
        bdy distro route update preview-distro --domain=preview.checkout.internal --target=artifact=$SERVICE:preview-$BUDDY_RUN_ID

    - action: "Smoke-check the preview (user surface = 0)"
      type: "HTTP"
      method: "GET"
      notification_url: "$SANDBOX_URL"
      retry_count: 6
      retry_interval: 20

    - action: "Publish stable candidate artifact"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy artifact publish $SERVICE:candidate-$BUDDY_RUN_ID ./dist --create

    - action: "Route 5% of prod traffic to candidate (user surface = 5%)"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy distro route update prod-distro --domain=checkout.example.com --target=artifact=$SERVICE:stable@95 --target=artifact=$SERVICE:candidate-$BUDDY_RUN_ID@5

    - action: "Health-check the 5% slice"
      type: "HTTP"
      method: "GET"
      notification_url: "$PROD_URL"
      retry_count: 10
      retry_interval: 30

    - action: "Promote candidate to 100% (radius widens after signal is green)"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy distro route update prod-distro --domain=checkout.example.com --target=artifact=$SERVICE:candidate-$BUDDY_RUN_ID
        bdy artifact tag $SERVICE:candidate-$BUDDY_RUN_ID stable

    - action: "Auto-rollback on any failure above"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      trigger_time: "ON_FAILURE"
      commands: |-
        bdy distro route update prod-distro --domain=checkout.example.com --target=artifact=$SERVICE:stable

Read the pipeline as a series of containment boundaries: the build runs in its own container, the deploy lands in a sandbox nobody outside the team can hit, the artifact is only published after the sandbox looks healthy, the route sends a small weighted slice before widening, and the ON_FAILURE action collapses the radius back to zero at the first bad signal. The blast radius at every step is explicit: you can point at the exact line where each surface is bounded. That is the property to optimise for: not "did the deploy succeed", but "how bad could it have been if it hadn't".