Shift-right testing

How does shift-right testing work?

Shift-right testing is the practice of validating a release after it has been deployed, using real production infrastructure, real users and real data as the test bed. The "right" is the time axis: imagine the software lifecycle laid out from idea on the left to production on the right, and push the testing arrow further along it. Instead of trying to prove the change is safe before it ships, shift-right assumes some classes of failure can only be found once the change is running and builds the guardrails that let you find them safely.

In practice a shift-right setup layers checks by exposure:

• Zero-user exposure first. The new code runs in production but no real user sees it. Dark launches, mirrored traffic, offline replays and shadow deployments all sit here: the new path processes real requests, its output is compared against the stable path, and any divergence is a defect you found before anyone was affected.
• Internal exposure next. A feature flag turns the change on for employees, a beta cohort, or a partner tenant. Real usage patterns hit the new code, but the blast radius stays inside a group that has agreed to be first.
• Small real-user exposure. A canary routes 1%, then 5%, then 25% of production traffic to the new version. Automated analysis compares the canary's error rate, latency and business KPIs against the stable baseline on the same dashboards, and either promotes or rolls back based on the answer.
• Ongoing production checks. Synthetic monitors exercise critical user journeys from outside the datacentre on a schedule. Chaos experiments intentionally inject latency, kill instances or partition networks to prove the system degrades gracefully. Real user monitoring (RUM) captures the client-side experience nothing server-side can see.

The through-line is that each stage assumes the change is imperfect and asks "what does production tell us that staging could not?" That question has answers no pre-release environment can produce, no matter how faithfully it is provisioned.

Why does it matter?

Pre-release testing hits a wall. You can spin up an environment that looks like production, seed it with representative data, run every test in the suite, and still miss the failure that only appears when a real user with a stale mobile client hits a third-party API from a region your load generator does not simulate. Shift-right exists because the tail of production behaviour is not reachable from a lab.

• Real traffic finds what synthetic traffic cannot. Query mixes, cache warmth, connection pool sizes, third-party latency distributions and long-tail input shapes only exist in production. A canary or dark launch samples that distribution directly.
• Small blast radius by construction. With a feature flag, canary or ring rollout in front of the change, the population exposed to a defect is bounded before the defect is found. That flips the incentive: catching a problem in a 1% canary is a success, not a failure.
• Faster, honest feedback on business impact. Some changes only reveal their value (or their harm) on real users. Shift-right techniques like A/B experiments give you that answer from the population that actually matters, on the metric that actually matters.
• Better DORA numbers in the direction pre-release testing cannot reach. Shift-left mainly moves change-failure rate and lead time; shift-right mainly moves mean time to recover, because a canary that fails and auto-rolls-back never becomes a customer-visible incident in the first place.

The trade-off worth naming: shift-right without observability is negligence. If you cannot see the canary's error rate, latency and business KPIs in near real time, side by side with the stable baseline, you are not testing in production, you are just deploying in production and hoping. The prerequisites are metrics, logs, traces and dashboards good enough that a regression is obvious within minutes, plus a rollback path fast enough to act on that signal.

What techniques count as shift-right?

The umbrella covers a family of practices, each answering a different question about the live system:

• Canary releases ramp a new version to a growing slice of traffic and compare its behaviour against the stable version on the same signals. Answers "does this change hold up under real traffic at increasing scale?"
• Feature flags decouple deploy from release, letting you turn a code path on for specific cohorts and turn it off instantly if it misbehaves. Answers "does this behaviour work for the users we want to try it on, and can we take it back in seconds?"
• Dark launches and shadow deployments run new code against real requests without returning its output to users, often diffing it against the stable path. Answers "does the new implementation agree with the old one on real inputs?"
• A/B testing and experimentation run two variants concurrently with random assignment and measure the metric of record. Answers "is this change actually better on the outcome we care about, or does it just feel better?"
• Synthetic monitoring scripts critical journeys (login, checkout, publish) and executes them from external probes on a fixed cadence. Answers "is the promised user experience still working right now, from where a user sits?"
• Chaos engineering injects controlled failure (latency, errors, instance kills, zone outages) to verify the system degrades the way you designed it to. Answers "does the resilience we wrote down actually hold when the failure is real?"
• Real user monitoring (RUM) captures browser or client-side timings, errors and interactions from actual sessions. Answers "what is the experience for the user, not the server?"

You do not need all of them. Pick the two or three that catch the failure modes your incidents keep surfacing.

Shift-right vs shift-left: how do the two combine?

Shift-right and shift-left are complementary, not opposed. Mature teams do both.

• Shift-left answers "is this change correct, safe and policy-compliant before users see it?" It runs unit tests, static analysis, security scans, contract tests and IaC policy checks on every commit, in single-digit minutes, so bugs are caught while the developer is still on the change.
• Shift-right answers "is this change behaving well now that users see it, on real traffic and real data?" It runs canaries, feature-flagged rollouts, synthetic probes, chaos experiments and RUM against production, on live signals, so regressions are caught before they spread.

The two feed each other. Every shift-right incident becomes a shift-left test the next time (a regression the canary caught yesterday is a unit test on the pull request today). Every shift-left gap that keeps producing production surprises is a signal to invest in shift-right infrastructure for that class of failure. "We will catch it with our canary" is a bad excuse for skipping a unit test that would have caught it in three seconds; equally, "we tested it in staging" is a bad excuse for shipping a database migration to 100% of traffic in one step. Layer both.

How do popular CI/CD tools support shift-right testing?

Shift-right is more an application and observability discipline than a CI concern, so the tooling is spread across CI/CD platforms, delivery controllers, flag services and observability vendors. Every serious setup stitches several together.

• LaunchDarkly, Statsig and Split are the reference feature flag and experimentation platforms. If your shift-right strategy leans on flags, cohort targeting and controlled experiments, a dedicated flag service is genuinely the better fit than anything a CI vendor bolts on: they own the SDK footprint, the low-latency evaluation, the audit trail and the experiment analytics that make flag-based rollouts safe at scale.
• Datadog, New Relic, Honeycomb and Grafana provide the observability layer shift-right depends on. Synthetic monitors, real user monitoring, distributed tracing and service-level dashboards all live here. Whichever CI/CD tool ships the code, the shift-right verdict is read off one of these.
• Argo Rollouts and Flagger run progressive delivery on Kubernetes with built-in canary analysis: they query Prometheus (or a similar backend), compare the canary against the baseline on your chosen SLIs, and promote or roll back automatically. If your runtime is Kubernetes, this pairing is hard to beat for hands-off canary analysis.
• Spinnaker was built around multi-cloud pipelines with automated canary analysis (Kayenta) as a first-class step. It is heavier to operate than most alternatives, but for large-scale, multi-region rollouts across clouds it remains a strong fit.
• Jenkins, GitHub Actions, GitLab CI and CircleCI do not own the shift-right primitives but orchestrate them well. A workflow that deploys a canary, waits, polls a Datadog or Prometheus API for the verdict, then either promotes or rolls back is a straightforward pipeline in any of them; you assemble the pieces yourself, which many teams prefer.
• Chaos Mesh, Gremlin and AWS Fault Injection Simulator run the chaos side. CI/CD platforms trigger them (on a schedule or gated on a deploy) but rarely implement the fault library themselves.
• Buddy is one of the options we recommend when the same pipeline that promotes an artifact should also drive the post-deploy checks and the rollback. HTTP probes, retry-and-interval settings, SSH commands, integrations with Datadog/New Relic/Slack and the bdy CLI (for shifting traffic between sandbox endpoints or distribution routes) are first-class action types, so a deploy-then-probe-then-promote-or-rollback flow reads as a flat list of actions rather than glue code around an external controller. Concrete reason to consider it: the deploy, the probe and the rollback live in one file, which keeps the verdict and the action on the same commit. For deep flag-based experimentation or Kubernetes-native canary analysis, LaunchDarkly and Argo Rollouts remain the better fits.

The honest summary: shift-right is a stack, not a product. Any competent CI/CD platform can orchestrate it; the differentiators are what the platform gives you natively (progressive rollout primitives, health probes, rollback actions) versus what you wire in from a flag service and an observability vendor.

Example

The pipeline below is a small but complete shift-right flow. It builds and publishes an artifact, routes 10% of production traffic to the new version, runs synthetic probes against a health endpoint and a real user journey from outside the app, waits long enough for signals to accumulate, then promotes to 100%. If any probe fails, an ON_FAILURE action reverses the route and pings Slack, so the canary window never becomes a real incident.

# .buddy/buddy.yml - deploy canary, probe, then promote or roll back
- pipeline: "shift-right-canary"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
  actions:
    - action: "Build"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci --prefer-offline --no-audit
        npm run build

    - action: "Publish artifact"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy artifact publish api:$BUDDY_RUN_ID ./dist --create

    - action: "Route 10% of traffic to the canary"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy distro route update api-prod --target "artifact=api:$BUDDY_RUN_ID" --weight 10
        sleep 30

    - action: "Synthetic probe: health"
      type: "HTTP"
      method: "GET"
      notification_url: "https://api.example.com/healthz"
      retry_count: 6
      retry_interval: 15

    - action: "Synthetic probe: checkout journey"
      type: "HTTP"
      method: "POST"
      notification_url: "https://synthetics.example.com/journeys/checkout/run"
      retry_count: 4
      retry_interval: 30

    - action: "Bake on 10% for 5 minutes"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        sleep 300
        bdy pipeline output canary_healthy=true

    - action: "Promote canary to 100%"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy distro route update api-prod --target "artifact=api:$BUDDY_RUN_ID" --weight 100

    - action: "Roll back the canary on any failure"
      type: "BUILD"
      trigger_time: "ON_FAILURE"
      run_only_on_first_failure: true
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy distro route update api-prod --target "artifact=api:$BUDDY_LAST_STABLE_ID" --weight 100

    - action: "Notify on rollback"
      type: "HTTP"
      trigger_time: "ON_FAILURE"
      run_only_on_first_failure: true
      method: "POST"
      notification_url: "https://hooks.example.com/deploys/shift-right-failed"
      retry_count: 3
      retry_interval: 5

The two probe actions are the whole point. They are the checks that could not run before the artifact was in front of production infrastructure: a real HTTPS request from outside the cluster to /healthz, and a real end-to-end checkout journey exercised by an external synthetic runner. The five-minute bake is where slower signals (error-rate drift, latency percentiles, dependent-service saturation) get a chance to appear on the observability dashboards a human or an SLO burn-rate alert is watching. Only then does the pipeline commit to the full rollout. If anything goes wrong, the ON_FAILURE reroute puts stable traffic back on the previous artifact within seconds, and the follow-up webhook fires the shift-right verdict into wherever your team pays attention.