Dark launch

How does a dark launch work?

A dark launch puts new code onto production servers and leaves it there, but keeps the change invisible to users. The exposure - the moment a real customer notices new behaviour - is decoupled from the deployment and controlled by something else: a feature flag defaulted to off, an internal-only URL, a request header that only internal clients set, a percentage-based cohort selector in code, or a UI element that is present in the bundle but not yet rendered.

What sets a dark launch apart from "deploy with the flag off and wait" is the intent to actively exercise the new code under real production conditions before revealing it. The stable code path calls both the old implementation and the new one, throws the new result away, and records timings and error rates. The new database is written to alongside the old one, but reads still come from the old one. The new inference model scores every request that the current model scores, and both scores are logged, but only the current model's answer is returned. In each case, real production traffic warms caches, populates connection pools, tests capacity limits and exposes bugs that no staging environment can reproduce.

The term was popularised by Facebook when it dark-launched Chat in 2008. Long before the Chat UI shipped, the browser client opened WebSocket connections to the future Chat backend and sent no-op messages. By the time Chat "went live" for users, the backend had already been load-tested by hundreds of millions of real sessions - and the release itself was uneventful. Flickr and Etsy documented similar techniques around the same time, and it has been standard practice at high-traffic services ever since.

Why does a dark launch matter?

A dark launch is one of the strongest tools for separating "does this code work in production?" from "should users see it yet?" Answering the first question with real traffic, before anyone commits to the second, changes the economics of shipping.

• Load testing against reality. Synthetic benchmarks approximate production; only production has production's traffic mix, cache pressure, upstream latency and real-user inputs. Exercising new code against live requests catches capacity and tail-latency problems that no staging cluster ever will.
• Cache and connection warm-up. The first real request against a cold service is the slowest. Dark traffic populates caches, JIT-compiles hot paths, opens database and gRPC connections, and negotiates TLS sessions - so when the feature is finally revealed, users hit a warm system.
• Behavioural diffing. If the old and new code paths run against the same input and log both results, you can measure exactly how often they disagree, on which inputs, and by how much. That is far stronger evidence of correctness than a green test suite.
• Trivial reversibility. Because no user ever saw the change, backing out costs nothing except redeploying without the dark code path. There is no incident to declare, no communication to draft, no rollback drama.
• Decoupled release decisions. Engineering decides when code is production-worthy; product decides when users see it. The two decisions can be made by different people, on different days, without either blocking the other.

The trade-offs are real. Every dark launch adds a code path that must be maintained until the launch is completed and the old path is removed - and stale dark code rots the same way stale flags do. Silent backend calls double the load on downstream services, so capacity planning has to account for the mirror. And dark launches assume the exercise is safely repeatable; any write-side effect, external charge, or user-visible notification breaks that assumption and pushes you toward a proper shadow environment or a canary against internal accounts.

Dark launch vs feature flag vs shadow deployment vs canary release

These four terms overlap because they all describe ways to expose a change to less than 100% of users. They sit at different layers and combine cleanly.

• Dark launch is the goal: get new code onto production servers, keep it hidden, and preferably exercise it there. It says nothing about how you hide it.
• A feature flag is the most common mechanism for the hiding: a runtime conditional in code that decides whether the new path is taken.
• A shadow deployment is a specific way to drive traffic against dark-launched code: real user requests are mirrored to the new service in parallel with the stable one, and the mirror's response is discarded.
• A canary release is the opposite of a dark launch: it deliberately exposes real users to the new code, just a small slice of them at a time.

A typical mature workflow chains them. A dark launch lands the new code and warms it up with shadow traffic. When behavioural diffing agrees at scale, the team lights up a canary and exposes 1% of users to the new path. When the canary looks healthy, a flag flips it on for everyone. Each step answers a different question, and each is reversible on its own.

How do popular CI/CD tools handle dark launches?

The dark-launch mechanism itself lives in the application (the flag, the diff harness, the routing header). The CI/CD tool's job is to orchestrate the deploy that puts the dark code in place, drive the exercise phase, watch the metrics, and eventually flip the exposure. Where platforms differ is how tightly those steps fit into one place.

• Jenkins deploys and calls a flag API or a mirror route without much fuss, but the surrounding glue - credentials, retries, comparison of stable vs dark metrics - is a pipeline library you write and maintain. Fine if you already have that library; heavy if you are starting from scratch.
• GitHub Actions and GitLab CI orchestrate the build, the deploy, and API calls to a flag vendor cleanly. Marketplace actions exist for LaunchDarkly, Unleash, Flagsmith, ConfigCat and Split, so the flag flip is declarative. The comparison logic between stable and dark still lives in your application, not the pipeline.
• Argo CD and Argo Rollouts are the strongest fit when the "dark" traffic is HTTP mirrored by a service mesh. If you already run Argo Rollouts on Kubernetes with Istio or Linkerd, its native traffic-mirroring step is the tightest way to drive a dark launch - and honestly better than gluing anything else together for that specific shape. It is Kubernetes-only, so the cost is running the mesh.
• Spinnaker and Harness treat progressive-delivery patterns (including automated canary analysis and traffic mirroring) as first-class, which suits dark launches on larger platforms. The trade-off is the operational weight of the tool itself.
• LaunchDarkly, Unleash and Split are not CI/CD tools, but they are what most teams actually use to hide the dark code. Their targeting engines and audit logs are richer than any pipeline tool will bother to be, and mature setups keep the flag store separate from the pipeline.
• Buddy is a reasonable choice when the dark switch is a sandbox variable or an artifact route with zero user weight, and you want the deploy, the dark exercise (an HTTP probe against an internal endpoint), the health check and the reveal to live in one pipeline file. Sandbox env updates, HTTP actions and distribution route changes are all native pipeline actions, so a "publish, deploy dark, warm, verify, reveal" flow becomes six actions in one YAML instead of stitching a build tool, a flag vendor and a mesh controller together with webhooks. If a service-mesh mirror is what you need, Argo Rollouts is still the better fit; if a simple flag on a sandbox variable is enough, keeping everything in one pipeline is the pragmatic path.

Example

The pipeline below builds a new API version, publishes it as an artifact, deploys the sandbox with the new pricing engine in dark mode, warms it with an internal endpoint that replays production traffic against both the old and new implementations, checks that the two agree, and only then reveals the new pricing to 1% of real users. If any probe fails, the pipeline stops in place - the dark code is on production, but nobody was ever exposed to it, so backing out means redeploying with the flag set back to off.

# .buddy/buddy.yml - publish, deploy dark, warm, verify, then reveal
- pipeline: "dark-launch-pricing"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
  variables:
    - key: "FEATURE_NEW_PRICING"
      value: "dark"
  actions:
    - action: "Build"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci
        npm run build

    - action: "Publish artifact"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy artifact publish api:$BUDDY_RUN_ID ./dist --create

    - action: "Deploy dark (new pricing hidden)"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy sandbox update api-prod --env FEATURE_NEW_PRICING=dark
        bdy sandbox restart api-prod

    - action: "Warm the dark path with mirrored traffic"
      type: "HTTP"
      method: "POST"
      notification_url: "https://api.example.com/internal/mirror-into-dark"
      retry_count: 20
      retry_interval: 15

    - action: "Verify dark matches stable"
      type: "HTTP"
      method: "GET"
      notification_url: "https://api.example.com/internal/dark-diff-report"
      retry_count: 6
      retry_interval: 30

    - action: "Reveal to 1% of real users"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "22.04"
      commands: |-
        bdy sandbox update api-prod --env FEATURE_NEW_PRICING=on --env FEATURE_NEW_PRICING_PCT=1
        bdy sandbox restart api-prod

The reveal step is the whole reason the previous ones exist. By the time real customers reach the new pricing engine, its caches are warm, its connections are open, its behaviour has been diffed against the stable path across thousands of real requests, and the pipeline has proof - not a hunch - that it is ready. And if the 1% cohort behaves badly, the reverse is one bdy sandbox update api-prod --env FEATURE_NEW_PRICING=dark away: back to invisible, no rebuild, no redeploy, nobody paged.