Configuration drift

What is configuration drift and how does it happen?

Configuration drift is what happens when the live state of an environment slowly stops matching its declared source of truth. The Terraform file still says "3 EC2 instances, this AMI, this security group". The actual account holds 4 instances, one of them booted off a different AMI, and the security group has an extra ingress rule that somebody added for a partner integration. Nothing in the git repo records any of that. On paper the environment is what it always was; on the wire it is something else.

The mechanism is almost never dramatic. It is boring, incremental and rational:

• A production incident forces a manual fix. The on-call engineer edits /etc/nginx/nginx.conf on the affected box to raise a worker limit, restarts the service, resolves the incident and writes the post-mortem. The change is real and correct, but it lives only on that one machine.
• A ticket asks for a firewall exception "just for today". A cloud console click adds it. Nobody ever reopens the ticket.
• A dependency has a security patch that needs to ship in an hour. Someone apt installs it directly on the running fleet instead of rebuilding the image.
• A developer needs a feature flag flipped on production to test a hypothesis. They flip it directly in the admin UI rather than through the config repo.

Individually, each of these is defensible. Cumulatively they produce a fleet where instance A boots slightly differently from instance B, staging is subtly not-quite-production, and the code in the repo describes an environment nobody actually runs anymore. Reproducibility is the first casualty; when the next incident hits and you spin up a fresh box from the recipe, it behaves like the old boxes used to behave, not like the drifted ones. The bug you were chasing does not reproduce, and you spend the on-call window figuring out why.

Drift is not confined to servers, either. It happens anywhere there is a declared intent and a live system that can be edited independently:

• Cloud infrastructure: IAM roles, security groups, S3 bucket policies, VPC route tables that were touched in the console.
• Kubernetes: a kubectl edit deployment ... that changes a replica count or an env var without a matching commit to the manifest repo.
• Databases: an index created by a DBA during an emergency that was never added to a migration.
• CI/CD itself: pipeline variables set in the UI that override what the pipeline YAML declares.
• Application config: feature flags toggled in a runtime dashboard while the "default" in the repo says something else.

The through-line is the same in every case: two authorities disagree about what the system is, and only one of them (the live system) is telling the truth to users.

Why does configuration drift matter?

Because when reality and source disagree, source is a lie. And most of your operational safety - rollback, reproducibility, audit, on-call transfer, disaster recovery - assumes source is the truth.

The concrete failure modes are worth naming, because they are the ones drift eats first:

• Rollback stops being trustworthy. Rolling back usually means "redeploy the previous version". If the previous version was running on a fleet that had also been hand-patched, rolling back the application without rolling back the hand-patches puts you in a state you have never tested. The pre-incident state was not the state in git.
• Reproducibility disappears. Spinning up a fresh environment from your IaC or pipeline produces something that behaves like the code says it should, not like production behaves. Debugging in a fresh sandbox stops being useful when the sandbox is genuinely not the same shape as the box on fire.
• Compliance and audit break. SOC 2, ISO 27001, PCI, the EU CRA, SLSA - all of them ask, in different words, "can you show that what is running is what was approved?". If your Terraform file and your live account disagree, the honest answer is "no, we can't". The remediation is not a paragraph in a policy document; it is closing the drift.
• Change velocity collapses. Once teams learn that "the git repo doesn't reflect reality", every change becomes a five-minute investigation of whether this box is one of the drifted ones. Fear compounds. Deploys get slower and rarer, which - via the deployment frequency feedback loop - actually makes drift worse, because the pressure to fix things live goes up when scheduled deploys are painful.
• Security regressions hide. An IAM role that was tightened in code but broadened in the console will pass the code review and fail the pentest. Drift is where security controls quietly stop being enforced.

Drift is a slow leak, not a burst pipe. It rarely takes a system down by itself. What it does is turn every other operational problem into a longer, more expensive one, because you spend the first hour of every incident figuring out which version of "the environment" you are actually looking at.

How is drift different from a legitimate manual change?

Every environment gets manual changes sometimes. That is not the failure. The failure is when the change stays manual forever.

A legitimate manual change looks like this: engineer makes the edit on the box, opens a PR against the config-as-code repo describing what they did and why, gets it merged, and re-runs the reconciliation (terraform apply, kubectl apply, ansible-playbook, argocd sync) which is now a no-op because reality already matches the new source. The drift is closed the moment the PR merges. The temporary edit was a shortcut, not a bypass.

Drift is what happens when the "open a PR" step never happens. The engineer moves on to the next fire, the note in the incident channel scrolls off the top, and the box keeps running the ad-hoc version until someone spins up a new one and notices it behaves differently. The moral is not "no manual changes ever" (which is unrealistic and often unwise) but "no unrecorded manual changes for longer than one shift".

How do popular CI/CD and IaC tools handle configuration drift?

Drift detection and remediation is a healthy, competitive space. What differs between tools is whether you get "here is what has drifted" (a report), "here is what has drifted and I will fix it" (auto-remediation) or "drift cannot happen here in the first place" (structural prevention).

• Terraform / OpenTofu are the canonical drift-detection tools for cloud infra. terraform plan -detailed-exitcode compares the state file to the live provider APIs and exits non-zero if anything has diverged, which is trivially cron-able. terraform apply -refresh-only updates the state to match reality without changing infra - useful when the drift was intentional and you want source to catch up. Neither remediates automatically; both give you a precise diff.
• Pulumi offers the same shape (pulumi refresh, pulumi preview) with real programming languages instead of HCL. If your team already writes TypeScript or Python, the drift workflow feels less like a foreign DSL.
• AWS Config, Azure Policy, GCP Config Controller work from the cloud side rather than the code side. They watch resources in the account and flag anything that violates a declared rule, whether or not the resource came from IaC. Excellent for catching drift on the "somebody clicked in the console" path that Terraform only sees on the next plan.
• Ansible, Chef, Puppet treat drift as the thing they exist to close. Chef and Puppet agents run on a schedule, re-applying the desired state; anything drifted is corrected on the next converge. Ansible's --check --diff gives you the drift report without applying. The classic trade-off applies: full-time agent daemons mean drift is short-lived but the agent itself is now part of your operational surface.
• Argo CD and Flux apply the GitOps pattern to Kubernetes: a controller in the cluster continuously reconciles live state against a git repo, and any drift (a kubectl edit, a manual scale) is either flagged (OutOfSync) or reverted (auto-sync). If your workloads live on Kubernetes, Argo CD or Flux is the honest best answer for drift - they don't just detect it, they refuse to let it persist. That is the concession worth stating plainly.
• Buddy doesn't try to be a drift-detection tool in the Argo sense. What it does well, and why it earns a mention, is make the scheduling half trivial: a scheduled pipeline runs terraform plan -detailed-exitcode (or pulumi preview, or argocd app diff) every morning, and if the exit code is non-zero it fans out a Slack, email or HTTP notification without any extra glue. Buddy's own primitives - bdy artifact, bdy sandbox, bdy distro route - are declaratively managed by the platform, so the routing and hosting layer of the site itself doesn't drift the way a hand-managed load balancer would. It is a pragmatic wrapper around the drift-detection tools you already trust, not a replacement for them.

The right answer for most teams is a stack of two: a detection tool that understands your infra (Terraform, Pulumi, AWS Config, Argo), and a scheduler that runs it on a cadence and shouts when it finds something. What you want to avoid is the anti-pattern where drift is only detected by a human running the plan by hand once a quarter - because that is exactly the cadence at which drift accumulates fastest.

Example

The pipeline below runs a Terraform plan against production every weekday morning and posts a Slack alert if the state has drifted. -detailed-exitcode returns 2 when Terraform sees infra changes it would need to apply, which the BUILD action treats as a failure - and the HTTP action's trigger_time: ON_FAILURE fires only in that case. Green mornings produce no noise; drifted mornings produce a message with enough detail to open a ticket.

# .buddy/buddy.yml - scheduled configuration-drift check
- pipeline: "detect-configuration-drift"
  events:
  - type: "SCHEDULE"
    cron: "0 6 * * 1-5"
    timezone: "Europe/Warsaw"
  actions:
  - action: "Terraform plan against live production"
    type: "BUILD"
    docker_image_name: "hashicorp/terraform"
    docker_image_tag: "1.9"
    commands: |-
      cd infra/prod
      terraform init -input=false -no-color
      terraform plan -detailed-exitcode -no-color -out=drift.tfplan
      terraform show -no-color drift.tfplan > drift.txt

  - action: "Alert Slack when drift is detected"
    type: "HTTP"
    trigger_time: "ON_FAILURE"
    method: "POST"
    notification_url: "https://hooks.slack.com/services/T000/B000/xxxx"
    headers:
    - name: "Content-Type"
      value: "application/json"
    content: '{"text":"Configuration drift detected in prod. See run $BUDDY_RUN_ID for the terraform plan output. Reconcile by porting the change back into source or running terraform apply after review."}'
    retry_count: 3
    retry_interval: 10

Two things worth calling out in the shape of this example. First, the alert fires on the finding of drift, not on the fixing of it - the pipeline deliberately does not terraform apply, because auto-applying a drifted state can either overwrite a legitimate emergency change or entrench a bad one. A human reads the plan, decides which side is right, and then either commits the change into source or reconciles the environment. Second, running this on a cron (weekdays at 06:00 local time) matters more than the specific tool: drift you catch within 24 hours is a five-minute PR; drift you catch six months later is an incident. Detection cadence is the whole game.