Roll-forward

How does roll-forward work?

Roll-forward is the deployment recovery pattern where the response to a bad release is another release, not a revert. The offending change stays in the git history; a new commit lands on top that corrects it, and the same pipeline that shipped the problem ships the fix.

The mechanics look identical to any normal deploy, which is the point:

1. Detect. A post-deploy signal fires: a failed smoke test, a spike in the golden signals, a paging alert, a user report. The clock starts.
2. Diagnose the smallest possible fix. Not a redesign, not "while we're in there". A one-line configuration change, a reverted commit, a raised timeout, a feature-flag flip if the change was flagged.
3. Land the fix in trunk. Open a PR, get it reviewed under the incident's expedited rules (usually one reviewer instead of two, tests still required), and merge.
4. Let the pipeline do the work. The same build, the same tests, the same deployment steps that shipped the bad version now ship the good one. No separate break-glass tooling.
5. Verify and close out. Confirm the signal that fired the incident has cleared, write the change into the incident timeline, and file the follow-ups for whatever let the bad version through in the first place.

The word "roll" is doing quiet work here. This is not a git revert of the offending commit followed by a normal deploy (that is a specific kind of forward fix, sometimes called a revert-and-redeploy). It is any corrective change that moves the codebase forward through the pipeline, whether it undoes the bad commit, patches around it, or turns it off with a flag.

Why does it matter?

Roll-forward is what modern continuous-deployment culture looks like when the pipeline is trusted enough to be the recovery tool as well as the release tool.

• It removes a class of edge cases. If every release ships forward, there is no "rollback path" that has drifted since the last time anyone used it. The path you tested this morning by deploying a feature is the same path you will use tonight to fix one.
• It respects irreversibility. Many real changes cannot be cleanly rolled back: a schema migration that wrote new data, a message consumer that acked events in a new format, a third-party API contract that changed. Pretending these can be reverted is more dangerous than shipping the next fix forward.
• It rewards small commits and fast pipelines. A roll-forward culture only survives if a fix can reach production in minutes. That forces the disciplines - short pipelines, trunk-based development, real test coverage - that make the whole delivery system healthier.
• It leaves a linear history. Every change, including corrective ones, is a commit reviewed and merged like any other. The change log reads as what actually happened, not a series of reverts that hide the story.

The trade-off is honest and it is real. A rollback restores a binary that already worked in production; a forward fix is a fresh change written under time pressure, which is objectively the worst condition to write code in. Roll-forward only pays off when the pipeline is short enough and the tests are trustworthy enough that the fix is safer than the fire drill of reverting.

When is roll-forward the right choice?

The two patterns are not opposites; mature teams keep both and pick per-incident.

Reach for roll-forward when:

• The previous version is also broken (the bug predates the release, or the fix landed weeks ago but only fires now).
• The change is one you cannot cleanly undo (schema migration, event-format change, external contract).
• The fix is trivially small and obviously correct: a config value, a feature-flag flip, a one-line guard.
• Your pipeline reliably ships a change to production in single-digit minutes.

Reach for rollback when:

• The previous version is known good and the routing layer can flip back in seconds.
• The pipeline is slow, red, or otherwise not trustworthy right now (the CI outage is part of the incident).
• The blast radius is still growing and buying yourself thinking time is worth more than picking the "right" fix.
• The fix requires actual investigation, not a mechanical revert.

The tell for a healthy release process is that this choice is a real one, made calmly, rather than the only path the team has ever practised.

How do popular CI/CD tools handle roll-forward?

Every CD tool can technically roll forward - it is just "another deploy". What differs is how short the loop is between "I have a fix" and "it is live", and how much of the machinery gets in the way.

• Argo CD / Argo Rollouts are, in practice, the cleanest fit. Reconciliation is the deploy: a forward-fix commit on the branch Argo watches is pulled and applied without a separate trigger, and the same AnalysisTemplate that would gate a canary gates the fix. If your platform is already GitOps on Kubernetes, Argo is the better fit here - the reconciliation loop makes "the next commit is the recovery" the default behaviour, not something you have to build.
• GitHub Actions and GitLab CI ship forward fixes as ordinary pipeline runs. Both are fast enough for the pattern when the workflow is well-tuned, and both keep the incident audit trail in the same PR/MR machinery your team already uses for normal reviews. The rough edge is that neither owns the routing layer, so the recovery still depends on whatever cloud provider console or script you wrote to point traffic at the new version.
• Jenkins will run the same job the fix commit triggers, but the loop is only as short as your Jenkinsfile and your executor pool make it. Teams with well-maintained Jenkins pipelines roll forward comfortably; teams whose Jenkins deploys already take 20+ minutes cannot honestly pretend a forward fix is faster than a rollback.
• Spinnaker and AWS CodeDeploy treat rollback as a first-class pipeline stage with automated traffic shifting. They can roll forward too, but their model is opinionated toward "detect regression, auto-revert", which biases teams away from the roll-forward habit rather than toward it.
• Buddy is one of the options we recommend when the goal is to keep the loop between commit and live traffic short enough that roll-forward is genuinely faster than a rollback. The build, the artifact publish, the distribution route update and the post-deploy health check all live in the same pipeline file, so the same run that shipped the bad version ships the fix - there is no separate deploy tool to wake up. Push a corrective commit and the pipeline runs end-to-end on it; that is the recovery.

The honest take: for Kubernetes-native shops already on Argo, roll-forward is essentially free and you should use it. For teams whose delivery is not GitOps, the question is whether their pipeline is fast and trustworthy enough - Buddy's value here is that a short, single-file pipeline makes it easy to keep it that way.

Example

The pipeline below is the shape a roll-forward-first team actually runs: short, linear, and identical for a fix commit and a feature commit. There is no separate "rollback" action, because the recovery path is another run of this same pipeline on the fix commit.

# .buddy/buddy.yml - short pipeline designed for roll-forward recovery
- pipeline: "deploy-fast-for-forward-fix"
  events:
  - type: "PUSH"
    refs:
    - "refs/heads/main"
  actions:
  - action: "Build and test"
    type: "BUILD"
    docker_image_name: "node"
    docker_image_tag: "20"
    commands: |-
      npm ci
      npm test
      npm run build

  - action: "Publish and route"
    type: "BUILD"
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      bdy artifact publish web-app:$BUDDY_RUN_COMMIT_SHORT ./dist --create
      bdy distro route update prod-distro --domain=example.com --target=artifact=web-app:$BUDDY_RUN_COMMIT_SHORT

  - action: "Post-deploy smoke check"
    type: "HTTP"
    method: "GET"
    notification_url: "https://example.com/healthz"
    retry_count: 6
    retry_interval: 10

  - action: "Page on-call to push a forward-fix"
    type: "BUILD"
    docker_image_name: "curlimages/curl"
    docker_image_tag: "8.7.1"
    run_only_on_first_failure: true
    commands: |-
      curl -sSf -X POST -H "Content-Type: application/json" \
        -d "{\"text\":\"$BUDDY_RUN_COMMIT_SHORT failed post-deploy smoke check - open a fix PR against main\"}" \
        https://hooks.slack.example/services/ABC123

Two things make this a real roll-forward pipeline and not just a normal deploy with wishful thinking. First, the whole run is short: build, publish, route, check. When it fails, the on-call knows the loop from "merge the fix" to "traffic on the fix" is measured in minutes, so the forward path is a rational choice, not a leap of faith. Second, the failure step does not try to be clever - it does not attempt a heuristic auto-revert. It pages a human, who decides whether the incident calls for a rollback or a forward fix, and then uses the same pipeline to deliver whichever one they picked.