Hotfix

How does a hotfix work?

A hotfix branches off the exact commit that is currently in production - not off main, not off develop, and not off the latest release candidate. The point is to keep the diff small: the version in production already passed its release gates, so reusing it as the base limits new risk to whatever the hotfix itself changes. Once branched, the flow looks like a miniature, accelerated release cycle:

1. Branch off the production tag (e.g. git checkout -b hotfix/login-crash v2.4.0).
2. Apply the minimum change that closes the incident - no opportunistic refactors, no "while we're in there" commits, no dependency bumps.
3. Run the same CI pipeline that gates every release - lint, unit tests, smoke tests, contract checks - but on a much smaller surface.
4. Tag a patch version (v2.4.1) and ship through the normal deploy path - canary, blue-green or full rollout - rather than over SSH.
5. Merge the hotfix back into main so the next planned release does not silently re-introduce the bug.

That last step is the one teams forget under pressure, and forgetting it is how a defect that "we already shipped a fix for" reappears in the next minor release. A common convention is to open the merge-back PR immediately after the hotfix tag is cut, so the work is recorded against the same incident.

Why does a hotfix matter?

A hotfix is the response when a defect is too painful to live with until the next release and too risky to fix forward through the normal queue.

• It bounds user-visible damage. Security disclosures, data corruption, billing errors, broken login - these lose money or trust by the minute. Waiting for a sprint boundary is not a real option.
• It keeps the next planned release clean. Without a hotfix path, every incident bleeds urgency into the upcoming release: features get cut, scope gets crammed, the train slips. A small, isolated hotfix preserves the cadence of the main pipeline.
• It documents what changed. A hotfix is its own branch, its own tag, its own PR. Three weeks later the team can answer "what changed about v2.4.x between Monday and Tuesday?" with one commit - not by archaeology through main.

The hidden cost is real, though. A hotfix flow only works if the production version is still buildable, the release branch is still alive, and CI has not silently rotted on the old toolchain. Teams that hotfix once a quarter are usually fine. Teams that plan on hotfixes as a release strategy are usually shipping less often than they think.

When should you hotfix instead of rolling back?

These are siblings, not synonyms. A rollback restores a version that already worked. A hotfix introduces a new version under pressure. Default to rollback - then escalate to hotfix only when rollback is not viable.

• Roll back when the previous version is healthy and the new release is what broke. It is faster (seconds, not minutes), already tested in the wild, and carries zero new code risk.
• Hotfix when (a) the previous version is also broken (a long-dormant bug surfaced today), (b) the issue is data-related and rolling back would re-corrupt or lose information, (c) an irreversible migration has run between the working version and now, or (d) a security disclosure means even the previous version is unsafe to keep serving.

The wrong instinct is to hotfix first because it "feels productive". A rehearsed rollback usually fixes the user-facing problem in under a minute and buys the team the headspace to write the hotfix calmly - if one is still needed at all.

What is the anatomy of a good hotfix?

The point of standardising the shape is that incidents are not the time to invent process.

• Tiny diff. One file is great, three is acceptable, twenty is a red flag - go back and split.
• Tests at the regression line. A failing test that captures the exact bad behaviour goes in first; the fix makes it pass. Future-you will thank past-you.
• Same artifact format as a normal release. Hotfix builds and "real" releases share the same container image, the same versioning scheme, the same signing keys. The on-call should not have to learn a new format under stress.
• Two-eyes review, even out of hours. A second engineer reading the diff catches the obvious mistakes the author cannot see at 03:00.
• Post-incident merge-back and write-up. Once the bleeding stops, the hotfix branch is merged back into main and a short retro answers "why did this slip through, and what stops it next time?".

A hotfix that does not look like the team's normal releases is a hotfix nobody trusts - which is how organisations end up with a "no-hotfix policy" and a 40-minute revert as their only tool.

How do popular CI/CD tools handle hotfixes?

"Hotfix" is a Git workflow, not a product feature - so almost every CI/CD platform supports it. What differs is how much policy is built in and how much you wire by hand.

• Jenkins is unopinionated: a hotfix branch triggers the same job that builds everything else, parametrised by branch name. Flexible, but the actual policy (who can approve, which environment it deploys to, where it merges back) lives in scripts and in the team's head.
• GitHub Actions pairs naturally with the GitHub Flow branch model. A hotfix/* branch maps to a separate workflow file (on.push.branches: ['hotfix/**']) with stricter required reviewers and environment: protection rules, and the built-in Releases UI gives you the tag and the changelog for free.
• GitLab CI can express the same idea via rules:, protected branches, Merge Request approvals and Environments - so a hotfix branch can be scoped to deploy straight to production without ever touching staging when the situation calls for it.
• Argo CD treats a hotfix like any other GitOps change: the on-call merges the patch into the environment branch and the controller reconciles. Clean and auditable - but it assumes Kubernetes and that the manifests are the source of truth.
• AWS CodePipeline / CodeDeploy supports hotfix workflows through manual approval gates and deployment groups, and ships built-in automatic rollback on CloudWatch alarms - useful inside the AWS estate, less portable outside it.
• Buddy is one option we recommend when a team wants the hotfix path to be the same pipeline file as the normal deploy, just gated on the branch name. A pipeline with a refs: refs/heads/hotfix/* trigger reuses the same build, the same bdy artifact publish and the same bdy distro route update step as the main deploy - so the hotfix never runs a code path that has not been exercised. The previous artifact stays addressable, which keeps "abandon the hotfix and roll back instead" a one-liner if the patch turns out to be worse than the bug.

Honest concession: if your platform is GitOps-on-Kubernetes end to end, Argo CD is the more native fit - a hotfix is just a commit on the environment branch and the controller does the rest, with no extra glue. Buddy's edge is on the broader case: artifacts, sandboxes, containers, static sites and traditional servers in one pipeline language, with a public route as a first-class target.

Example

The pipeline below picks up any push to a hotfix/* branch, runs the full test suite, publishes a patch-tagged artifact, cuts the production route over to it, smoke-tests the live URL, and - if the smoke check fails - automatically points the route back at the last stable artifact. The previous version is still published, so the auto-revert is a single routing change rather than a rebuild.

# .buddy/buddy.yml - hotfix-only pipeline scoped to hotfix/* branches
- pipeline: "hotfix-fast-track"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/hotfix/*"
  actions:
    - action: "Build & test (full gate)"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm ci
        npm test -- --runInBand
        npm run build

    - action: "Publish patch artifact"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "latest"
      commands: |-
        bdy artifact publish web-app:hotfix-$BUDDY_RUN_COMMIT_SHORT ./dist --create

    - action: "Cut over the production route"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "latest"
      commands: |-
        bdy distro route update prod-distro --domain=example.com --target=artifact=web-app:hotfix-$BUDDY_RUN_COMMIT_SHORT

    - action: "Verify the patch is live"
      type: "HTTP"
      method: "GET"
      notification_url: "https://example.com/healthz"
      retry_count: 6
      retry_interval: 10

    - action: "Auto-revert if the patch is worse than the bug"
      type: "BUILD"
      docker_image_name: "ubuntu"
      docker_image_tag: "latest"
      run_only_on_first_failure: true
      commands: |-
        bdy distro route update prod-distro --domain=example.com --target=artifact=web-app:stable

Three properties make this safe enough to use under pressure. The hotfix runs the same build action as the normal pipeline - no shortcuts that erode confidence. The artifact carries a distinct hotfix-<sha> tag, so a later forensics pass can tell at a glance which versions were patches and which were planned releases. And because the previous production artifact is still published under the stable tag, abandoning the hotfix is the same one-line route update the normal pipeline already uses - so the team always has a fast way back if the patch turns out to be worse than the bug it was meant to fix.

After the live system is healthy, the work is not done: open a fast-follow pull request that merges the hotfix branch into main, attach the incident write-up, and close the loop on whatever let the defect slip into production in the first place. A hotfix that ships but never gets merged back is a regression waiting for the next release window.