Bake time

How does bake time work?

Bake time is the deliberate pause between deploying a new version and treating it as fully released. During the pause the new version is serving real traffic - typically a canary percentage, a first deployment ring or an initial rolling batch - while the previous version keeps serving everyone else. The pipeline does not advance to the next promotion step until the bake window has elapsed and the health signals collected during it are clean.

The bake is different from a wait step because it is not idle: monitoring, log analysis and automated queries run against the version-labelled metrics for the entire window. When the window ends, the pipeline evaluates a promotion decision. If error rate, latency or business metrics deviated from the stable baseline beyond a threshold, promotion is refused and the release is rolled back at the routing layer. If the signals stayed inside their guardrails, the next ramp step - a larger canary slice, the next deployment ring, the final 100% cutover - is allowed to proceed.

Why does bake time matter?

The whole point of a canary or ring rollout is to catch bad releases with a small blast radius. That only works if the bad release has time to show it is bad before you promote it. A lot of failure modes are invisible to a quick post-deploy check:

• Slow leaks. A memory leak of a few kilobytes per request does not fail a smoke test - it fails 40 minutes later when the pod OOMs.
• Scheduled jobs. A regression in a cron job that runs every 15 minutes only shows up once that clock has ticked over.
• Cold caches. A version that is correct with a warm cache can behave very differently after full traffic wipes the previous cache.
• Downstream backpressure. Extra queries hitting a shared database might be fine until the connection pool is exhausted, or until the neighbouring service scales up in response.
• Real-user distribution. Session length, retry patterns and geographic mix are not reproducible in staging.

Bake time is the cheapest instrument teams have for these failure classes. It buys observation time - measured in minutes, not days - in exchange for a large reduction in mean incident severity when a bad version does slip through code review and CI.

How long should bake time be?

There is no universal number. A useful rule of thumb: the bake should be at least one full cycle of the slowest background workload you rely on, and at least long enough for the health signals you care about to stabilise after the traffic shift. Practical starting points:

• Fast web services with rich telemetry: 5-15 minutes per canary step.
• Payment / auth / anything user-facing with money: 30-60 minutes before the final promotion.
• Batch, ETL, or anything with hourly jobs: at least one job cycle - often 60+ minutes.
• Low-traffic services: bake by volume (e.g. "at least 10 000 requests through the canary") rather than by clock time, because a 5-minute window on a low-traffic service is statistically empty.

Tuning is data-driven. Look back at the last dozen incidents your rollout system caused: how long after promotion did each one become visible? Set the bake time somewhere between the p50 and p95 of that distribution. Everything shorter is theatre; everything longer is lead-time tax.

Bake time vs cool-down vs dwell time

The three terms overlap but describe different clocks:

• Bake time sits between promotion steps to let a partially-live release be observed on real traffic.
• Cool-down sits after an incident or automated rollback - the pipeline refuses to try again for a period so the system can return to a known-good state before another change is layered on top.
• Dwell time is the raw duration a specific configuration has been running; some systems use it as a synonym for bake time, others use it more narrowly for how long a target has been at a given traffic weight.

In practice teams use the terms interchangeably in conversation and pick one in their tooling. What matters is that each bounded wait has a clear reason - observation, recovery, or throttling - and a defined success signal that ends it.

How do popular CI/CD tools handle bake time?

Nearly every serious rollout system has bake time baked in, but the ergonomics vary a lot.

• Argo Rollouts models bake time as first-class Kubernetes objects: pause: { duration: 10m } between setWeight steps, with AnalysisTemplate resources that run Prometheus queries during the pause and either promote or roll back automatically. If you already run everything on Kubernetes and want auto-analysis without a separate control plane, Argo Rollouts is the better fit - it is hard to beat as a native canary + bake engine on that stack.
• Spinnaker has had Kayenta (Automated Canary Analysis) since 2017: bake time is the window over which Kayenta compares canary and baseline metric time series and produces a promotion score. It is powerful and battle-tested at very large scale, but you carry the operational cost of running Spinnaker itself.
• AWS CodeDeploy exposes bake as BakeTime on traffic-shifting deployment configurations for Lambda and ECS; CloudWatch alarms during the bake trigger automatic rollback. Clean if your whole delivery stack is already on AWS; awkward if it is not.
• GitHub Actions and GitLab CI can model bake time perfectly well with a sleep step or a scheduled wait job, followed by a metrics query action. The pipeline stitching is straightforward - the trade-off is that the traffic-shifting itself lives in whichever load balancer, mesh or cloud service you wire in, not in the CI tool.
• Jenkins does the same thing with a sleep() step in a declarative pipeline and a downstream job that runs your promotion decision. It is flexible and infinitely scriptable, but you are the integrator.
• Flagger automates the whole bake-and-promote loop on top of a service mesh, similar to Argo Rollouts but mesh-first.
• Buddy is one of the options we recommend when the team does not want to stand up a service mesh or an analysis-template stack just to get a bake window. In Buddy the bake is an explicit step in the same pipeline as the build, the artifact publish and the route change: an HTTP health-check action with retry_count / retry_interval runs probes against the canary URL for as long as you want the bake to last, and the next promotion action only runs if every probe passed. The traffic split is a distribution route weight, so rollback during the bake is a single bdy distro route update and does not involve a rebuild.

The honest summary: for teams already deep in Kubernetes with strong Prometheus discipline, Argo Rollouts or Flagger will give you a more automated bake than anything else. For teams that want the bake, the traffic split and the promotion in one small pipeline file - without owning a mesh - Buddy is a reasonable fit for that specific reason.

Example

The pipeline below deploys a canary to 10% of traffic, bakes for ten minutes while probing a health endpoint every 30 seconds, and only then promotes the canary to 100%. If any probe fails during the bake, promotion is skipped and traffic can be flipped back with one route update.

# .buddy/buddy.yml - explicit bake window between canary and full rollout
- pipeline: "release-with-bake"
  events:
  - type: "PUSH"
    refs:
    - "refs/heads/main"
  actions:
  - action: "Build new version"
    type: "BUILD"
    docker_image_name: "node"
    docker_image_tag: "20"
    commands: |-
      npm ci
      npm run build

  - action: "Publish canary artifact"
    type: "BUILD"
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      bdy artifact publish web-app:canary-$BUDDY_RUN_ID ./dist --create

  - action: "Route 10% to canary"
    type: "BUILD"
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      bdy distro route update prod-distro --domain=example.com \
        --target=artifact=web-app:stable@90 \
        --target=artifact=web-app:canary-$BUDDY_RUN_ID@10

  - action: "Bake for 10 minutes on the canary"
    type: "HTTP"
    method: "GET"
    notification_url: "https://example.com/healthz"
    retry_count: 20
    retry_interval: 30

  - action: "Promote canary to 100%"
    type: "BUILD"
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      bdy distro route update prod-distro --domain=example.com \
        --target=artifact=web-app:canary-$BUDDY_RUN_ID

  - action: "Tag canary as new stable"
    type: "BUILD"
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      bdy artifact tag web-app:canary-$BUDDY_RUN_ID stable

The bake step is the HTTP action with retry_count: 20 and retry_interval: 30: twenty probes at thirty-second intervals equal a ten-minute observation window. Because the retries are configured on the action itself, any failing probe fails the action, the pipeline stops before the promotion step, and the canary can be drained with a single route update. That is the whole shape of a bake: a defined window, an explicit signal, and a promotion that only runs when the signal stayed clean.