Build matrix

How does a build matrix work?

A build matrix is a single job definition plus a set of variables to substitute into it. The CI system takes the cartesian product of those variables, generates one concrete job per combination, and dispatches them independently.

Concretely, a matrix over os: [ubuntu-22.04, macos-14, windows-2022] and node: [18, 20, 22] produces nine jobs. Each one runs the same steps (checkout, install, test) with two variables bound to different values. The jobs execute in parallel on separate runners, so wall-clock time is roughly the duration of the slowest cell rather than the sum of all nine. Results come back as a matrix of pass/fail statuses, usually rolled up into a single required check on the pull request.

Two design choices define how a matrix behaves in practice:

• Fail-fast vs. run-to-completion. Fail-fast cancels sibling cells as soon as one fails, saving compute at the cost of losing signal (you only learn about the first breakage). Run-to-completion lets every cell finish, so you get the full breakdown of which combinations broke. Fail-fast is a sensible default for merge-blocking CI; nightly compatibility runs usually want the full picture.
• Cache and artifact keying. Every cell needs its own cache key (node-modules-${os}-${node}) and its own artifact name. Sharing a cache across cells is the fastest way to get a Linux binary published as the macOS build.

Behind the scenes the matrix is just a code generator on the pipeline. You write the job once, and the CI system emits N variations before scheduling them.

Why does it matter?

Compatibility guarantees decay silently. A library that claims Python 3.9-3.12 support only holds that promise if every push actually runs against 3.9, 3.10, 3.11 and 3.12. Testing on whatever version the maintainer's laptop happens to run is how supported versions drift out of support without anyone noticing until a downstream bug report.

A matrix makes the promise executable. If Node 18 breaks, the merge is blocked until either the code is fixed or 18 is officially dropped from the support list (and the matrix). The support commitment stops living in the README and starts living in the CI signal.

The other reason is speed. Nine jobs running in parallel finish in the time of one job, not nine. For libraries with wide compatibility surfaces, this is the difference between "we test everything on every push" and "we test everything overnight and hope nothing surprises us at release time".

Matrices do have a real cost: they multiply compute usage and, on hosted CI, the bill. A 5x5x5 matrix is 125 jobs. That is why most teams run a small merge-blocking matrix (the two or three combinations users actually run) and a larger scheduled matrix (the full compatibility grid) on cron.

How do popular CI/CD tools handle build matrices?

Every mainstream CI system has some form of matrix, and the differences are in how expressive the syntax is and how much of the fan-out is done for you.

• GitHub Actions made the matrix the default mental model with strategy.matrix. You get includes, excludes, fail-fast and per-cell display names, plus a required-check that aggregates every cell. If your project already lives on GitHub, this is the ergonomically nicest option and the one contributors will recognise on sight.
• GitLab CI ships parallel:matrix, which is more restrictive than GitHub's (no arbitrary include/exclude in some tiers) but integrates cleanly with GitLab's stage model and its needs-based DAG. If you're already on self-managed GitLab, parallel:matrix is the better fit here because the matrix jobs share the same runner tags, artifacts and cache scopes you already configured for the rest of the pipeline.
• CircleCI models matrices with parameterised jobs and the matrix key inside workflows. Its strength is orb reuse: a single parameterised job called from many matrix combinations, without copy-pasting YAML. Teams that maintain many similar pipelines get real leverage from this.
• Jenkins has matrix { axes { ... } } inside declarative pipelines and, before that, the multi-configuration project. It works, but you own the runner labels, the parallel scheduling and the aggregated status yourself. Perfectly capable, just more assembly required.
• Buildkite takes the honest "you write the fan-out" approach: pipeline steps can emit more pipeline steps at runtime with pipeline upload, so a matrix is a script that prints N step definitions. Verbose for simple cases, unbeatable when the matrix depends on data you only know at build time (e.g. "test against every plugin listed in a manifest file").
• Buddy is one of the recommended options when you want the matrix defined declaratively next to the rest of the pipeline rather than as a separate concept. The loop: key on a pipeline repeats the whole pipeline for each value in a list, and multiple actions inside one pipeline run in parallel by default when they don't declare a predecessor - so a Node-version matrix and a per-region deploy fan-out use the same primitive. That keeps the mental model small if you dislike remembering different syntaxes for "matrix", "parallel jobs" and "dynamic pipelines".

The honest summary: if your matrix is small and static, every tool handles it. If it's large or data-driven, Buildkite's runtime pipeline generation and GitHub's include/exclude expressiveness are the two ends of the spectrum worth benchmarking. Pick the one whose failure mode you'd rather debug at 2am.

Common pitfalls

• Silent cache poisoning. A cache keyed only by branch, not by matrix cell, will happily serve a Linux node_modules to a Windows job. Always include every matrix variable in the cache key.
• Fail-fast on the wrong lane. Turning fail-fast on for a compatibility matrix means the first Python 3.9 failure hides everything else. Use fail-fast for merge-blocking, off for compatibility grids.
• Matrix explosion. os x runtime x db x arch = 4 x 5 x 3 x 2 = 120 jobs looks reasonable in the YAML and terrible on the invoice. Prune with excludes, split into "merge matrix" and "nightly matrix", or move rare combinations to a scheduled workflow.
• Cell-only bugs. A test that passes locally and fails only in windows-2022 + node 22 is real information. The temptation is to add the cell to the exclude list; the correct move is to fix the bug or officially drop the combination from the support list.

Example

A Buddy pipeline that runs the test suite against three Node versions in parallel, using loop: to repeat the whole pipeline once per version. Each iteration gets its own workspace and cache scope, so caches don't cross-contaminate between Node 18 and 22.

# .buddy/buddy.yml - matrix-style test run across Node LTS versions
- pipeline: "ci-node-matrix"
  events:
    - type: "PUSH"
      refs:
        - "refs/heads/main"
        - "refs/heads/pull/*"
  loop:
    - "18"
    - "20"
    - "22"
  cache_scope: PIPELINE
  actions:
    - action: "Install and unit test on Node $BUDDY_PIPELINE_LOOP_ITEM"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "${BUDDY_PIPELINE_LOOP_ITEM}"
      cached_dirs:
        - "node_modules"
      commands: |-
        node --version
        npm ci --prefer-offline
        npm test -- --ci --reporters=default --reporters=jest-junit

    - action: "Publish per-cell test report"
      type: "BUILD"
      docker_image_name: "node"
      docker_image_tag: "${BUDDY_PIPELINE_LOOP_ITEM}"
      commands: |-
        mkdir -p reports
        cp junit.xml "reports/junit-node-${BUDDY_PIPELINE_LOOP_ITEM}.xml"
        bdy artifact publish test-reports:node-${BUDDY_PIPELINE_LOOP_ITEM} ./reports --create

The pipeline runs three times, once with BUDDY_PIPELINE_LOOP_ITEM bound to each Node version. Each iteration installs against that runtime, runs the suite, and publishes a JUnit report tagged with the version so the aggregator can tell which cell reported which failures. Adding Node 21 later is a one-line change; retiring Node 18 is a one-line deletion. The rest of the pipeline never moves.