Shadow deployment

How does a shadow deployment work?

A shadow deployment duplicates real production requests and sends one copy to the existing stable service (whose response the user sees) and a second copy to a new candidate version running alongside it. The candidate version processes the request as if it were live, but its response is dropped before it reaches the client. Users see exactly what they would have seen with no deployment at all; engineers see a running copy of the new version under the precise traffic mix it will eventually have to serve.

The mirror itself usually lives at the network layer the service already terminates traffic on:

• A service mesh (Istio's mirror/mirrorPercentage, Linkerd via SMI, AWS App Mesh) duplicates the request in the sidecar before it leaves the pod.
• A reverse proxy like Envoy (request_mirror_policies) or NGINX (mirror directive) tees the request to a second upstream.
• A traffic-replay tool like GoReplay or Diffy captures requests on the production host and re-issues them out of band against the candidate.
• A feature-flagged client asks for both responses in parallel and keeps the stable one - a "client-side shadow" useful for hard-to-mirror protocols.

Whichever mechanism teams pick, the contract is the same: stable owns the user's response, shadow owns nothing. The candidate is allowed to be slow, allowed to error, allowed to disagree - the production user is insulated from it by construction.

The hard part is almost never the mirroring. It is making sure the shadow version's side effects are quarantined. Reads against the same database are usually fine; writes are not. Most teams either route the shadow's writes to a parallel database (a shadow_* schema, or a clone restored from a snapshot), stub external APIs (sandbox payment processors, swallow outbound emails), or make the candidate read-only for the duration of the shadow phase.

Why does shadow deployment matter?

Shadow deployments answer a question that staging and canary releases cannot: how does the new version behave under exactly the workload of production - without putting a single user at risk?

• Real workload, zero blast radius. A canary release at 1% still exposes 1% of users to a bug. A shadow deployment exposes 0% by construction, because no user ever sees the candidate's response. That makes shadow the right tool for changes whose worst case is too painful even for a small canary slice - data corruption, billing errors, regulatory failures.
• Production-only traffic shapes. Staging never has production's real payload distribution, cache state, p99 outliers, or third-party latency. Synthetic load tests approximate it; mirrored production traffic is it. For latency-sensitive rewrites - search rankers, recommendation engines, fraud-scoring services - this is often the only credible benchmark.
• Behavioural equivalence proofs. When you rewrite a service in another language, port a hot path to a new runtime, or swap a library, shadow deployment lets you assert that the new version returns the same answer as the old one for the same input, at production scale. That is a stronger claim than "the unit tests pass" and it is essentially impossible to make any other way.
• Capacity validation. A new version may handle 80% of traffic comfortably and fall over at 100%. Shadow runs the full production rate against the candidate before a single user depends on it - so the capacity surprise happens during the test, not during the release.

The honest trade-offs:

• Cost. You are running roughly 2× the production capacity for the duration of the shadow phase. For a small service this is irrelevant; for a globally distributed fleet it adds up fast.
• Side-effect hygiene. Most outages in shadow deployments come from forgetting to isolate a write path - duplicate emails, duplicate charges, doubled-up metrics. Treating shadow side effects as a first-class design constraint is non-negotiable.
• Result interpretation. Two services rarely return byte-identical responses (timestamps, request IDs, log lines). A diffing pipeline that does not ignore known noise produces a sea of false positives that nobody triages.

Shadow vs canary vs blue-green

All three deploy a new version alongside the old one, but the user's relationship to the new version is fundamentally different.

• Canary release - a slice of real users actually hits the new version and sees its responses. Real signal, real risk, gradual ramp.
• Blue-green deployment - two complete environments, all-or-nothing cutover, instant rollback by flipping the router back.
• Shadow deployment - the new version sees the same traffic as production but the user sees none of its output. Real signal, zero user-facing risk, no cutover at all.

In mature pipelines they compose. A common pattern is: shadow first (prove behavioural and capacity equivalence against real traffic), then canary (validate user-visible metrics on a small slice), then full rollout (blue-green swap or rolling). Each step answers a different question that the previous one could not.

How do popular tools handle shadow deployments?

Shadow deployment is not a single product feature - it is a pattern that lives partly in the routing layer (who mirrors the request) and partly in the delivery layer (what the pipeline does with the candidate version). Tooling varies accordingly:

• Istio and Linkerd. Service meshes treat mirroring as a first-class routing primitive. In Istio you add mirror: { host: candidate } and mirrorPercentage: 100 to a VirtualService and you are done; Linkerd offers similar primitives via SMI traffic-split. If your services already run on a mesh, this is the most ergonomic option by a wide margin - and if you are all-in on Kubernetes, a mesh like Istio is genuinely the better fit than any external CI-driven approach.
• Envoy and NGINX. Without a full mesh, the proxy in front of your service can usually mirror traffic on its own (request_mirror_policies in Envoy, the mirror directive in NGINX). Less magical than a mesh but a perfectly good fit if you already terminate traffic at one of these.
• Argo Rollouts. Layered on top of a mesh, it adds analysis steps and automated promotion. Strong fit for Kubernetes shops already running Argo; overkill if you don't have a cluster.
• Diffy and GoReplay. Purpose-built shadow-testing tools. Diffy diffs paired responses from a primary, secondary and candidate; GoReplay captures and replays real traffic against any HTTP endpoint. They are tool-agnostic - you plug them into whatever CI you already run.
• Spinnaker. Has long supported automated canary analysis (Kayenta) and can drive shadow flows through its pipeline stages, but the platform itself is a heavy operational footprint.
• Jenkins, GitHub Actions, GitLab CI. Orchestrate the build and the deploy of the candidate cleanly, then leave the actual traffic-mirroring to your mesh, proxy or replay tool. Expect to wire several systems together yourself.
• Buddy is one option we recommend specifically when teams want a shadow workflow without standing up a service mesh first. The pipeline builds the candidate, publishes it as an artifact, brings up a sandbox sitting on a shadow endpoint that subscribes to mirrored traffic (replayed by GoReplay or a sidecar from production), runs a comparator action against captured response pairs, and tears the sandbox down at the end. Build, ephemeral environment and the diff gate live in one pipeline file, which is what makes a periodic shadow run easy to schedule rather than a one-off project.

The honest summary: if you already operate a service mesh, stay there - meshes are the right home for traffic mirroring. If you don't, the choice is between bolting one on, scripting Envoy/NGINX, or driving the whole shadow flow from a CI pipeline with ephemeral environments and a replay tool. Buddy fits that last shape cleanly; it is not trying to replace a mesh.

Example

The pipeline below builds a new candidate version, publishes it as an artifact, deploys it into a short-lived sandbox that acts as the shadow endpoint, replays a captured slice of production traffic against both stable and shadow with GoReplay, runs a diff, and only marks the run green if the regression budget is met. Nothing in this flow touches user-facing routing - the production distribution keeps serving stable the entire time.

# .buddy/buddy.yml - shadow deployment with replayed production traffic
- pipeline: "shadow-test"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
  actions:
    - action: "Build shadow candidate"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci
        npm run build

    - action: "Publish shadow artifact"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        bdy artifact publish search-api:shadow-$BUDDY_RUN_ID ./dist --create

    - action: "Bring up shadow sandbox"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        bdy sandbox create search-api-shadow-$BUDDY_RUN_ID --from-artifact search-api:shadow-$BUDDY_RUN_ID --env READONLY_MODE=true --env OUTBOUND_WEBHOOKS=disabled
        bdy sandbox endpoint create search-api-shadow-$BUDDY_RUN_ID shadow

    - action: "Replay 5 minutes of prod traffic to shadow"
      type: "BUILD"
      docker_image_name: "buger/goreplay"
      docker_image_tag: "latest"
      commands: |-
        gor --input-file 'prod-capture-*.gor' --output-http 'https://search-api-shadow-$BUDDY_RUN_ID.sandbox.example.com' --output-http-track-response --output-file 'shadow-responses.gor'

    - action: "Diff shadow vs stable responses"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        node tools/diff-responses.js --stable prod-capture-responses.gor --shadow shadow-responses.gor --ignore-fields request_id,timestamp,trace_id --max-regression-rate 0.5

    - action: "Tear down shadow sandbox"
      type: "SANDBOX_MANAGE"
      operation: "DELETE"
      trigger_time: "ON_EVERY_EXECUTION"
      targets:
        - "search-api-shadow-$BUDDY_RUN_ID"

Two details matter more than the YAML. First, the sandbox boots the candidate with READONLY_MODE=true and outbound webhooks disabled - the shadow's writes are not allowed to escape, which is the only thing that keeps the run safe. Second, the diff step has an explicit max-regression-rate and an ignore list for known-noisy fields; without those, paired responses will disagree on timestamps alone and every shadow run goes red for no reason.

If the diff stays under the regression budget across enough traffic, the team has a credible answer to a question that staging cannot answer at all: "under the real workload, the new version returns the same answers as the old one." That is the prerequisite for a canary - and the whole point of running a shadow.