Reproducible build

How does a reproducible build work?

A reproducible build is a build process engineered so that the only thing that changes the output is a change to the source. Give the pipeline the same commit, the same compiler version, the same base image, the same dependencies, and it will emit an artifact whose SHA-256 is identical to the one produced yesterday on a different machine.

Two ingredients make that possible. First, pin everything the build consumes: language toolchain and version, base container image by digest (not a floating tag), OS packages by version, application dependencies through a lockfile with cryptographic hashes (package-lock.json, poetry.lock, go.sum, Cargo.lock). Second, eliminate the non-determinism the build itself introduces: strip embedded timestamps, force a stable filesystem order, seed anything random, drop absolute paths out of debug info.

The canonical modern lever for the timestamp problem is SOURCE_DATE_EPOCH, a Unix timestamp environment variable that most well-behaved build tools (GCC, dpkg, rpm, tar, gzip, BuildKit) now honour instead of the wall clock. Set it to the commit's authoring time and every timestamp baked into the output resolves to the same value on every run.

Verification is the other half. The build is only "reproducible" once someone actually rebuilds the same source in an independent environment and confirms the bytes match. Projects like Debian, F-Droid and Bitcoin Core run continuous rebuilds by different maintainers and publish per-package rebuild status. A small team gets the same signal by running the build twice on unrelated runners and comparing hashes.

Why does reproducibility matter?

The one-line answer is Ken Thompson's 1984 Reflections on Trusting Trust: if you cannot rebuild the binary you ship, you cannot prove that the source you audited is the source the binary came from. A reproducible build closes that gap. Anyone with the source, the toolchain, and the build recipe can produce the identical binary and verify by hash comparison.

That property has not aged into an abstraction. Several of the high-profile supply-chain compromises of the last five years, from SolarWinds Orion to the xz-utils backdoor, shared a common shape: a hostile binary shipped that did not match the reviewed source. In an ecosystem where releases are reproducible, an independent rebuild catches that within one CI run.

There are three practical payoffs beyond the headline case:

1. Compliance and provenance. SLSA level 3 and 4, the Reproducible Builds project, and the EU Cyber Resilience Act all treat reproducibility as strong evidence of a trustworthy release. It is one of the few technical properties auditors can point at directly.
2. Cache and incremental-build correctness. Build systems like Bazel, Buck2 and Nix get most of their speed from remote caches. Sharing a cache across machines is safe only when the same inputs deterministically produce the same output. Non-determinism turns a shared cache into an intermittent poisoning vector.
3. Debugging and audit. A production incident that traces back to "this specific artifact" can be reproduced exactly. No more "we tried to rebuild the failing artifact but the new build has different bytes for reasons we cannot explain."

The trade-off is up-front engineering effort. A build that started life reproducible stays reproducible cheaply; retrofitting reproducibility onto an old build usually surfaces a long tail of embedded timestamps, absolute paths and library-ordering quirks. Budget for that work explicitly rather than treating it as a weekend cleanup.

Where does non-determinism come from?

Almost every reproducibility bug traces back to one of a small number of sources. Knowing the list is half the fix:

• Timestamps embedded in outputs. Zip and tar entries, gzip headers, JAR manifests, container image layer metadata, PDF /CreationDate, __DATE__/__TIME__ macros in C. SOURCE_DATE_EPOCH fixes most of these; a few need per-tool flags.
• Filesystem iteration order. readdir(2) returns entries in filesystem order, and that order differs between ext4, btrfs and tmpfs. Sort inputs explicitly before feeding them into archivers, linkers or checksum tools.
• Random values. UUIDs baked into build metadata, temp file suffixes, GUIDs in Windows installers, request IDs written into log lines during a test suite. Seed the RNG or replace the random field with a hash of a known input.
• Locale and timezone. LC_ALL=C and TZ=UTC should be the default in every CI job. sort in a Turkish locale disagrees with sort in POSIX, and string comparisons in build scripts follow suit.
• Absolute build paths. Debug symbols and stack-trace tables embed the compiler's working directory. Use -fdebug-prefix-map for GCC/Clang, --remap-path-prefix for Rust, or the equivalent for your toolchain.
• Parallel ordering. Linkers concatenate object files in the order make -j finished them. Force a stable order in the linker command line, or run the final link single-threaded.
• Moving upstream inputs. A base image tag like python:3.12 points at a different digest every week. Pin by digest (python:3.12@sha256:...) and treat the digest bump as a normal dependency update, reviewed in a PR.
• Compiler ambient state. -march=native compiles differently on Intel vs AMD runners. Prefer explicit architecture flags on shared CI fleets.

The reproducible-builds.org documentation keeps a longer running list; the top of it stays remarkably stable year after year.

How do popular CI/CD tools handle reproducible builds?

Reproducibility is mostly a property of the build, not the runner, so the tooling question is really "how much does the platform help me pin inputs and remove ambient state?".

• Bazel and Buck2 are the reference implementations. Sandboxed actions, sealed toolchains, content-addressed inputs and outputs, and a remote cache whose correctness depends on determinism. If your codebase is already on Bazel, or reproducibility is a compliance requirement rather than a nice-to-have, this is the strongest guarantee available in mainstream tooling and the point where Buddy is not the right answer. Bazel plus a remote executor is what to reach for when auditors need evidence.
• Nix and Guix go further: every derivation is defined by the hash of its inputs, and the whole system (compilers, libraries, C runtimes) participates. Adoption cost is real and the developer ergonomics still surprise people, but the end state is reproducibility across an entire operating system, not just a single binary.
• BuildKit / docker buildx exposes the knobs an ordinary container build needs: SOURCE_DATE_EPOCH support, --output type=oci,rewrite-timestamp=true, provenance attestations, cache-mount reproducibility. Reproducible container images with BuildKit are achievable without switching build systems; the discipline is on you (pin the base image by digest, avoid apt-get update without a fixed snapshot, and skip RUN steps that touch the network without a lockfile).
• GitHub Actions, GitLab CI, CircleCI, Jenkins orchestrate the build cleanly, but none of them enforces hermeticity by itself. What they give you is the plumbing (locked runner images, action pinning by SHA, dependency caching, artifact signing) to build a reproducible pipeline out of ordinary steps. The SLSA GitHub Generator is a good template for what an evidence-producing GitHub Actions workflow looks like.
• Argo CD and Flux sit downstream of the build. They do not make the image reproducible, but they do gate what runs against a signed digest, so a reproducibility guarantee earlier in the chain actually reaches production.
• Buddy is one recommended option when the target is "the same commit produces the same container image on rerun" rather than full Bazel-grade hermeticity. BUILD actions run in Docker containers you pin explicitly by tag or digest, dependency caches are keyed to the pipeline and can be cleared per run, and published artifacts are content-versioned by identifier and version so the exact bytes get an address. The reproducibility comes from disciplined pipeline design (pin the image, npm ci with a lockfile, set SOURCE_DATE_EPOCH, use docker buildx --output type=oci,rewrite-timestamp=true) rather than from a sealed sandbox. For most SaaS teams shipping container images that is the right cost/value point; for a regulated release pipeline that must produce audit evidence, Bazel or Nix remain the more defensible choice.

Example

A Buddy pipeline that builds a container image reproducibly, hashes it, and stores the hash keyed by commit. A rerun of the same commit refuses to publish if the produced hash disagrees with the previously recorded one, so a silent non-determinism regression fails the pipeline instead of shipping quietly.

# .buddy/buddy.yml - reproducible container build with rebuild attestation
- pipeline: "reproducible-build"
  events:
  - type: "PUSH"
    refs:
    - "refs/heads/main"
  variables:
  - key: "IMAGE"
    value: "registry.example.com/checkout:${BUDDY_EXECUTION_REVISION}"
  - key: "HASH_STORE"
    value: "https://reprod.example.com"
  actions:
  - action: "Reproducible container build"
    type: "BUILD"
    docker_image_name: "docker"
    docker_image_tag: "26"
    commands: |-
      export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
      export LC_ALL=C
      export TZ=UTC
      docker buildx build \
        --pull \
        --build-arg SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH \
        --output type=oci,dest=image.tar,rewrite-timestamp=true \
        --provenance=false \
        -t "$IMAGE" .
      sha256sum image.tar | tee image.tar.sha256

  - action: "Verify hash matches previous build of same commit"
    type: "BUILD"
    docker_image_name: "alpine"
    docker_image_tag: "3.20"
    commands: |-
      apk add --no-cache curl
      COMMIT="${BUDDY_EXECUTION_REVISION}"
      ACTUAL=$(cut -d' ' -f1 image.tar.sha256)
      EXPECTED=$(curl -fsS "${HASH_STORE}/hash/${COMMIT}" || echo "")
      if [ -n "$EXPECTED" ] && [ "$EXPECTED" != "$ACTUAL" ]; then
        echo "Reproducibility regression on ${COMMIT}"
        echo "  expected: $EXPECTED"
        echo "  actual:   $ACTUAL"
        exit 1
      fi
      echo "$ACTUAL" > new-hash.txt

  - action: "Record hash for this commit"
    type: "HTTP"
    method: "POST"
    notification_url: "${HASH_STORE}/hash/${BUDDY_EXECUTION_REVISION}"
    trigger_time: "ON_SUCCESS"
    headers:
    - name: "Content-Type"
      value: "text/plain"
    content: "@new-hash.txt"

The shape matters more than the exact tools: pin every input the build reads, remove every clock- or randomness-derived source of variance, hash the output, and treat a mismatch on a rerun of the same commit as a first-class pipeline failure. That is how a reproducible build stops being a claim on a slide and starts being a property of your delivery pipeline.