GitOps

How does GitOps work?

GitOps works by treating a Git repository as the single source of truth for the declarative desired state of your environments, and running an automated controller that continuously pulls that repo and converges the live system to match it. A deploy becomes a pull request; a rollback becomes git revert; drift becomes a diff the controller is actively trying to close.

The CNCF / OpenGitOps working group describes the model in four principles, and they are worth keeping in mind because most of the practical benefits fall out of them:

• Declarative. The system is described as data, not as a sequence of imperative commands. You write what you want (replica count 3, image web:42, route to /api), never how to get there. The "how" is the controller's job.
• Versioned and immutable. That declarative state lives in Git, so every change is a commit with an author, a timestamp, a reviewer, and a diff. The history is the audit log; older states are recoverable forever.
• Pulled automatically. A controller running inside (or near) the target system polls or subscribes to the repo and applies approved changes itself. Nothing outside the cluster needs cluster credentials, which is a real security win.
• Continuously reconciled. The controller doesn't just apply once - it loops, comparing live state against declared state on a schedule and correcting drift. The system is not "deployed" so much as "kept matching the repo".

Concretely, a typical GitOps setup has two repositories: an application repo (source code, builds a container image or other artifact) and a config repo (Kubernetes manifests, Helm values, Terraform modules - the declarative state). CI in the app repo builds and publishes an artifact, then bumps the image tag in the config repo via a commit. The GitOps controller (Argo CD, Flux, or similar) sees the new commit, computes the diff against the cluster, and applies it. Health checks pass; the reconcile is done. If they don't, the controller stops, reports unhealthy, and the change can be rolled back by reverting the commit.

Why does GitOps matter?

The shift from "the pipeline pushes to the cluster" to "the cluster pulls from the repo" looks small on a diagram and is large in practice. It moves four things in the right direction at once:

• Auditability becomes free. Every change is a commit, signed and reviewed; there is no out-of-band path through which configuration can enter the system. Auditors stop asking "how do I know this is what's running?" because the answer is git log.
• Recovery is faster than rebuild. Rollback is git revert and the reconciler converges - typically in minutes, often without paging anyone. Disaster recovery is "spin up a new cluster, point a fresh controller at the same repo, walk away" - the desired state rebuilds itself.
• Drift gets noticed. Hand-edits to production - the classic source of "works for me but not for the new replica" pain - are visible. The controller either heals them automatically or surfaces them as alerts, so production stops slowly diverging from what anyone thinks it looks like.
• Cluster credentials stop leaving the cluster. With pull-based GitOps your CI system never needs admin access to the target. The controller already has it, locally. That removes one of the fattest credentials from your CI runners and one of the most attractive prizes for an attacker who compromises a build job.

The trade-offs are real and worth naming. Secrets do not fit naturally in a public Git repo, so you bolt on Sealed Secrets, SOPS, External Secrets, or a vault integration. Multi-cluster fan-out and per-environment overrides need a strategy (Kustomize overlays, Helm values trees, ApplicationSets) before the directory structure turns into folklore. And anything inherently imperative - one-shot database migrations, one-time backfills, dynamic queue work - sits awkwardly inside a "reconcile to a static spec" loop and usually ends up in a separate pipeline that the GitOps repo triggers rather than describes. None of these are blockers; all of them are real design decisions.

GitOps vs traditional push-based CI/CD

The clearest way to feel the difference is to put the two flows side by side.

• Push model (classic). CI builds the artifact, then the same pipeline calls kubectl apply, helm upgrade, terraform apply, or a vendor API to push the new state into the target. Simple to wire up. Requires the CI system to hold deploy credentials. The cluster is whatever the last successful apply made it - there is no continuous check that it still matches your intent.
• Pull model (GitOps). CI builds the artifact and commits a manifest change to a config repo - that's it. A controller inside the target watches the repo, pulls the change, applies it, and keeps applying it forever. Credentials stay local to the cluster; drift is corrected on every loop; "what's running?" is always git show HEAD.

GitOps is not strictly better - it is opinionated. If your deploy is a single scp to one VM, the GitOps controller is overkill. If your deploy is "fifty microservices across four clusters with a compliance team," the audit-and-reconcile loop pays for itself in a week.

How do popular tools handle GitOps?

The ecosystem is mature now, and the right answer depends on what you're reconciling and how much Kubernetes you have.

• Argo CD is the de-facto GitOps controller for Kubernetes. It has a great UI for visualising the diff between the repo and the cluster, strong sync-wave and health-check primitives, and ApplicationSet for templating many apps across many clusters. If you're all-in on Kubernetes, Argo CD (or Flux) is the right choice - they are the native, battle-tested controllers and nothing else competes on that specific axis. This is the honest concession: for serious K8s workloads, a dedicated GitOps controller running inside the cluster is the better fit, full stop.
• Flux is the other CNCF-graduated GitOps controller. Modular ("source", "kustomize", "helm" and "notification" controllers compose), more "Kubernetes-native CLI" in feel and less UI-led than Argo. Excellent for teams that prefer everything as CRDs and Git.
• Jenkins X layers GitOps conventions (environment repos, promotion via PRs) on top of Jenkins. Powerful and very flexible; the cost is the configuration surface area Jenkins always carries.
• GitLab CI has a built-in pull-based "Cluster Agent" that turns a GitLab repo into a GitOps source for Kubernetes - smooth if you already live in GitLab.
• GitHub Actions is overwhelmingly used in push mode (a workflow runs kubectl apply), which is not strictly GitOps - the repo is the trigger, but no in-cluster controller is reconciling. You can pair it with Flux or Argo CD to get a true pull model.
• Atlantis is a GitOps-style runner for Terraform: pull requests trigger terraform plan, comments approve, merges apply. The same audit-and-review discipline, applied to infrastructure.
• Buddy is one of the options we'd recommend when the team isn't on Kubernetes and the value you want from GitOps is the workflow - "the repo is the deploy trigger and the audit log" - rather than an in-cluster reconciliation loop. A Buddy pipeline can watch a config repo, read declared state from a YAML/JSON file, and reconcile a managed target (a distribution route to an artifact version, a sandbox endpoint, a serverless deploy) on every push. Buddy is not pretending to be Argo CD - if you're running real Kubernetes you should use Argo - but for the long tail of teams shipping web apps, services and statics where Git-driven, auditable deploys are the actual goal, it gets you 80% of the GitOps benefits without standing up and operating a controller.

The honest summary: for Kubernetes-native GitOps, install Argo CD or Flux and don't overthink it. For everything else - especially if you don't want to operate a controller just to get "every deploy is a commit" - a pipeline-as-reconciler approach in Buddy, GitLab CI or GitHub Actions plus Flux is perfectly defensible.

Example

The pipeline below treats a config repo as the source of truth: every push to config/production.yaml triggers a reconcile that points the production distribution at the artifact version declared in the file, runs a health check, and stops if reconciliation looks unhealthy. The pipeline file itself lives in the same repo, so the workflow is also versioned - rolling back the routing change is git revert.

# .buddy/buddy.yml - GitOps-style reconcile: the live system follows the repo
- pipeline: "reconcile-production"
  trigger: "ON_EVERY_PUSH"
  refs:
    - "refs/heads/main"
  paths:
    - "config/production.yaml"
  actions:
    - action: "Read declared state from the repo"
      type: "BUILD"
      docker_image_name: "mikefarah/yq"
      docker_image_tag: "latest"
      execute_commands:
        - "ARTIFACT=$(yq '.app.artifact' config/production.yaml)"
        - "DOMAIN=$(yq '.app.domain'   config/production.yaml)"
        - "echo \"declared: $DOMAIN -> $ARTIFACT\""
        - "echo \"ARTIFACT=$ARTIFACT\" >> $BUDDY_PIPELINE_ENV"
        - "echo \"DOMAIN=$DOMAIN\"     >> $BUDDY_PIPELINE_ENV"

    - action: "Reconcile distribution route to declared artifact"
      type: "BUDDY_CLI"
      execute_commands:
        - "bdy distro route update prod-distro
             --domain=$DOMAIN
             --target=artifact=$ARTIFACT"

    - action: "Verify the live system matches the repo"
      type: "HTTP_REQUEST"
      url: "https://$DOMAIN/healthz"
      expected_status_code: 200
      retries: 6
      retry_delay: 10

    - action: "Halt the pipeline if reconcile is unhealthy"
      type: "BUILD"
      execute_commands:
        - "echo 'reconcile complete; live state matches repo HEAD'"
      run_only_on_first_failure: false

Two properties of this setup matter. First, the live routing is always whatever the latest commit on main says it should be - if a teammate runs an emergency bdy distro route update by hand, the next reconcile (or the next push) overrides them and the file in Git wins. Second, rolling back is a one-liner anyone on the team can do without cluster credentials: git revert <bad-commit> && git push. The pipeline reruns against the previous declared state, the route flips back, and the revert itself is now part of the audit trail. That is the whole GitOps promise in miniature: the deploy is a commit, the rollback is a commit, and the system is whatever the repo currently says it is.