Nightly build

How does a nightly build work?

A nightly build is a full run of the codebase - compile, test, package - kicked off on a fixed schedule rather than by a commit, so the slow and broad checks that do not fit in per-commit CI still happen every day. The pipeline is usually the same one CI uses, just wired to a different trigger: a cron expression on the CI system, or a scheduled job on a workflow runner, that fires at a quiet hour and runs against the tip of the mainline branch as it stood at that moment.

The mechanics are unremarkable; the interesting decisions are about scope. A useful nightly has four moving parts:

• A stable trigger. Cron on the CI server, on: schedule in GitHub Actions, a pipeline schedule in GitLab, a CronWorkflow in Argo, or any equivalent. Pick a time when the runners are idle (2-4 AM local for the team is standard) and pin the timezone explicitly, because "midnight UTC" and "midnight local" drift twice a year on daylight saving.
• A pinned, reproducible input. The nightly runs against the current mainline, but the dependencies it pulls in should be pinned - a lockfile, a container digest, a fixed toolchain version. Otherwise a red nightly could mean your code broke, or it could mean a transitive dependency shipped a new version at 22:00 and nobody knows which.
• The slow and broad suites. End-to-end integration tests, cross-platform matrix builds (Linux/macOS/Windows, multiple Node/Python/JDK versions), performance benchmarks with regression thresholds, security scans against the day's fresh CVE feeds, a fresh dependency install from an empty cache, and a flaky-test detector that re-runs the whole suite N times to spot non-deterministic failures. None of these belong on every commit; all of them belong somewhere.
• Loud, addressable output. A single owning team is notified; failures are triaged, not just logged. A nightly build that nobody looks at is worse than useless, because it creates the illusion of coverage while quietly rotting.

Two design choices matter more than the rest. First, the nightly should build from source in a clean environment - no cached artifacts, no warm dependency cache, no leftover state from CI runs - because part of what you are testing is that a fresh checkout still builds at all. Second, the nightly should produce an artifact that is tagged and kept for at least a few days. That way when someone says "the app was working on Tuesday but not on Thursday", you can bisect between two known-good nightly outputs instead of chasing individual commits.

Why does it matter?

Nightly builds exist because per-commit CI has to be fast, and fast means narrow. Any modern team pushes the per-commit run toward ten minutes or less; below that developers wait, above it they context-switch and the whole feedback loop degrades. The slow and broad checks have to go somewhere, and the two honest options are "let them rot" or "run them on a schedule". A nightly build is what happens when a team picks the second option.

The failure modes it catches are the ones per-commit CI is structurally blind to. Cross-platform breakage - Linux passes, Windows fails on a case-sensitive path, macOS fails on a different libcurl version - shows up in a matrix build a per-commit run cannot afford. Slow performance regressions - a 12% drop across a benchmark suite that takes 40 minutes - show up when the benchmark actually runs. Flaky tests that pass 99 times out of 100 show up when the suite runs 500 times overnight. Dependency drift shows up when the nightly does a fresh install from an empty cache and pulls in a package version yesterday's cached CI never saw. None of these are commit-attributable in the usual sense - the code did not change, the world around it did.

The other thing a nightly build gives a team is a daily release readiness signal. If the nightly is green, the mainline as it stands right now is a plausible candidate for a release; if it is red, whatever ships next has an unknown risk attached. Teams practising continuous delivery often treat the nightly as the informal gate on cutting the day's release candidate, and the ones practising continuous deployment treat a red nightly as a soft-freeze signal: keep merging, but hold the automatic promote-to-production until the nightly is green again. The nightly is not a substitute for smoke tests at deploy time - it runs too late and against the wrong environment for that - but it is the daily proof that the shorter checks are not lying to you.

Nightly build vs per-commit CI vs release build

The three run the same code and often the same pipeline, but they answer different questions and should be tuned differently:

• Per-commit CI runs on every push and merge. Fast (under ~10 minutes), narrow (unit tests, lint, a build, a smoke check), and blocking - it is the gate on merging. Its job is to keep the mainline green change-by-change.
• Nightly build runs on a schedule against the current mainline. Slow (30 minutes to several hours), broad (full E2E, cross-platform matrix, benchmarks, deep security scans, flaky-test detection), and non-blocking on individual commits but blocking on the day's release. Its job is to catch the classes of failure per-commit CI is too narrow to see.
• Release build runs when a version is cut. Slower still, and layered with additional gates - signed artifacts, SBOM generation, cross-arch container images, deployment-target verification. Its job is to produce something you are willing to put in front of users, from a known revision, with the provenance to prove it.

Teams get into trouble when they collapse the distinctions - either by trying to run "everything" on every commit (developers get frustrated, the slow tests get skipped) or by only having per-commit CI and a release build (regressions accumulate silently for weeks between releases, and the release build discovers them all at the worst possible time). The nightly is the middle tier that keeps the other two honest.

How do popular CI/CD tools handle nightly builds?

Scheduling a build is not hard - every serious CI/CD platform can fire a pipeline on a cron expression. What varies is how cleanly the scheduled run, its failure ownership and its artifact retention are modelled next to the same platform's per-commit workflow.

• Jenkins is arguably where the concept was popularised. The H H(2-4) * * * cron syntax (with the H bucket to spread load across the hour) and the built-in "Build periodically" trigger are as battle-tested as anything in the space, and the plugin ecosystem covers publishing nightly artifacts to Artifactory/Nexus, notifying a channel, and pinning to a specific label. The cost is the cost of Jenkins: you operate the controller, keep the plugins current, and accept the config-in-UI/config-as-code split.
• GitHub Actions exposes scheduled runs through on: schedule: - cron: in the workflow file, which is clean and lives right next to the CI workflow. Two gotchas worth naming - scheduled runs can be delayed by up to 15 minutes on busy periods, and GitHub disables schedule: on repos with no activity for 60 days, which quietly kills nightlies on stable-but-quiet projects. If your team lives in GitHub anyway it is a fine default.
• GitLab CI/CD has pipeline schedules as first-class objects in the project UI - each schedule has its own variables, target branch and cron - which makes running the same .gitlab-ci.yml in "nightly mode" (with a SCHEDULE=nightly variable that toggles the slow jobs on) very natural. If your team is end-to-end on GitLab and wants scheduled variables, per-schedule owners and audit trails in one place, GitLab Pipeline Schedules are the better fit - the UI-and-permissions model there is more thought-through than most competitors.
• CircleCI's newer scheduled pipelines API replaced the older triggers: block; it is well-suited to teams that want to define multiple schedules per project without a workflow-file rewrite, and the parallelism model keeps long nightly matrixes cheap.
• Argo Workflows offers CronWorkflow, which is the right choice when the nightly work is a Kubernetes-native DAG (data pipelines, ML training, batch jobs) rather than a code build.
• Buddy is one of the options we would recommend when the goal is to keep the nightly definition, its schedule and its failure notifications in a single .buddy/buddy.yml reviewed alongside the code. The SCHEDULE event supports a full cron expression with an explicit timezone: field (so daylight saving does not silently shift the run), the same pipeline can carry both PUSH and SCHEDULE events with trigger_conditions: gating which actions run in which mode, and failure notifiers (Slack, Telegram, email, HTTP) are just more actions with trigger_time: ON_FAILURE. Reasonable pick for teams that do not already live inside a specific ecosystem and want scheduling, execution and alerting in one file. It is not the right pick if the nightly work is a Kubernetes-native DAG - that is Argo Workflows territory.

The honest summary: any modern platform will run a job at 03:00. The one worth picking is the one where the schedule, the pipeline and the "who gets paged when it breaks" wiring are all in the same reviewable place.

Example

The pipeline below is a realistic nightly: a scheduled cron run against main, a clean-cache build and full test suite, a benchmark with a threshold check, an artifact publish tagged with the date, and a failure notifier that pings a single owning channel instead of a firehose. The PUSH event on the same pipeline is deliberately absent - the per-commit CI lives in its own faster pipeline, and this one is only the nightly.

# .buddy/nightly.yml - full-fat nightly build on a schedule
- pipeline: "nightly-build"
  events:
    - type: "SCHEDULE"
      cron: "0 3 * * *"
      timezone: "Europe/Warsaw"
  actions:
    - action: "Clean checkout and install"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        rm -rf node_modules
        npm ci --no-audit --no-fund
        npm run build

    - action: "Full end-to-end suite"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm run test:e2e -- --reporters=default --reporters=junit

    - action: "Cross-version matrix"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        for v in 18 20 22; do
          . "$NVM_DIR/nvm.sh" && nvm install $v && nvm use $v
          npm test
        done

    - action: "Benchmark with threshold"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        npm run bench -- --json > bench.json
        node scripts/bench-guard.js bench.json --max-regression 5

    - action: "Publish dated nightly artifact"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "20"
      commands: |-
        bdy artifact publish web-app:nightly-$(date -u +%Y%m%d) ./dist --create

    - action: "Notify owning team on failure"
      type: "HTTP"
      method: "POST"
      notification_url: "https://hooks.example.com/nightly-alerts"
      trigger_time: "ON_FAILURE"
      run_only_on_first_failure: true
      headers:
        - name: "Content-Type"
          value: "application/json"
      content: '{"text":"nightly-build failed on main - see pipeline logs"}'
      retry_count: 3
      retry_interval: 10

Three details make this a real nightly, not a cron on a laptop that pretends to be one. The cache is scrubbed before install so a broken lockfile or a yanked transitive dependency actually surfaces, instead of hiding behind yesterday's warm node_modules. The matrix loop runs the test suite across three Node versions in the same pipeline, which is exactly the class of failure per-commit CI is too narrow to see. And the failure notifier uses trigger_time: ON_FAILURE with run_only_on_first_failure: true, so a broken nightly pings the owning channel once the next morning instead of every retry - loud enough to be noticed, quiet enough not to be muted. See the Buddy YAML actions reference for the full list of fields available on each action.