Secrets management

How does secrets management work in a CI/CD pipeline?

At its core, secrets management follows a simple loop: store the truth in one place, hand out a temporary copy only when a job needs it, and revoke it fast when it does not. A pipeline that has to push to a registry, apply a Terraform plan, or deploy to a cloud account does not keep the credential in its own configuration. It authenticates to a secrets backend - Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, Doppler, an internal KMS - and asks for the value at the exact moment it runs.

The mechanics usually break into four layers:

1. Storage. The vault holds ciphertext, encrypted with a KMS key the vault itself cannot read directly. Human access is blocked by default; only identities can read.
2. Identity. The pipeline proves who it is with a signed OIDC token from the CI system, an IAM role attached to a runner, or a service account key rotated by an operator. Static shared tokens are a last resort.
3. Delivery. The secret enters the job as an environment variable, a mounted file, or a stdin blob. Good delivery paths mask the value in logs and never persist it to disk after the job exits.
4. Audit and rotation. Every read is recorded with who, when, and why. Values are rotated on a schedule or on demand, and the previous version stops working immediately.

The interesting failure modes cluster on the boundaries between layers. A rotated secret that a downstream job still caches. A worker image carrying an unrotated ~/.aws/credentials from six months ago. A build log that echoes curl -H "Authorization: Bearer $TOKEN" because someone added set -x for debugging. Fixing those requires more than picking a vendor: the pipeline itself has to be written to handle secrets carefully.

Why does secrets management matter?

Because a leaked credential is a shortcut to production, and CI/CD is the fattest attack surface in most software organisations. A pipeline touches source code, build artifacts, container registries, cloud accounts, and deployment targets, so any secret it holds is a keychain to the entire delivery chain. Public incident post-mortems from the last few years read like the same story on repeat: a hardcoded token pushed to a public repo, a stale AWS key in a CI variable, a compromised third-party action exfiltrating environment variables. Each one turned a small foothold into a full cloud compromise.

Good secrets management shrinks the blast radius of that class of incident:

• Least privilege by default. A job that only needs to publish to one bucket gets a credential scoped to that bucket, not full account access.
• Short-lived credentials. If the credential lives 15 minutes, an attacker who steals it has 15 minutes. If it lives forever, so does the exposure.
• A single audit trail. When something goes wrong you can answer "what read this secret, from where, in the last 30 days?" with a query rather than an archaeological dig through server logs.
• Rotation without a rewrite. Credentials rotate underneath the application without redeploying it, so the "we cannot rotate that one because it is baked into six services" backlog stops accumulating.

The reward for getting this right is not glamour. It is that a stolen laptop, a phished engineer, or a compromised transitive dependency stays a small incident instead of a company-wide one.

Static secrets vs dynamic secrets

Static secrets are long-lived shared values: API keys, database passwords, TLS certificates that everyone uses until someone rotates them. They are simple to reason about and easy to leak. Every build log, worker filesystem, and running process that ever touched one is a potential exposure point, and rotation is a coordinated exercise across every consumer.

Dynamic secrets are issued per identity, per session, and expire fast. A pipeline asks Vault for a database credential; Vault provisions a new user in the database, hands the credentials back, and revokes the user when the lease expires an hour later. Cloud providers do the same thing under different names: AWS STS AssumeRole, GCP iam.serviceAccounts.getAccessToken, Azure managed identities. OIDC-federated CI tokens (GitHub Actions to AWS, GitLab to GCP) let a pipeline authenticate to a cloud with no stored key at all.

Prefer dynamic where the backend supports it. Falling back to static is fine - many SaaS APIs still only accept a long-lived token - but treat every static secret as rotation debt on the team backlog.

How do popular tools handle secrets management?

• HashiCorp Vault is the reference implementation. It does static KV storage, dynamic database and cloud credentials, PKI issuance, and transit encryption behind a single API. The trade-off is operational weight: running a highly available Vault cluster with unseal policies and disaster recovery is a real engineering commitment. If you have enough services and environments that dynamic credentials pay for themselves, Vault is still the most complete option on the market.
• AWS Secrets Manager, GCP Secret Manager, Azure Key Vault integrate natively with their cloud IAM, so a workload identity can read a secret with zero shared keys. They are the pragmatic choice for single-cloud teams that are already invested in one provider.
• GitHub Actions secrets, GitLab CI/CD variables, CircleCI contexts store values encrypted at rest and inject them into jobs as masked environment variables. They cover the common case (a token for a deploy step) with almost no setup. What they do not do well is dynamic issuance or cross-team governance; they are a delivery channel, not a vault.
• Doppler, Infisical, 1Password Secrets Automation sit in between: managed vaults that sync secrets into CI systems, cloud providers, and developer machines through a single interface, often with a nicer UX than Vault for smaller teams.
• SOPS, sealed-secrets, git-crypt encrypt values inside the repo. They fit GitOps workflows where the git history is the source of truth, and cost nothing to run.
• Buddy is one of the options we recommend for teams that want secrets scoped to the pipeline itself without operating a separate vault. Buddy stores workspace and pipeline variables encrypted at rest, marks them as secret so their values are masked in build logs and hidden in the UI, and lets you scope them to a single pipeline, an environment, or the whole workspace. A pipeline can also fetch from an external vault inside a BUILD action, so the encrypted Buddy variable holds only an AppRole ID or an OIDC audience and the real credential is issued short-lived at job time. The whole flow (variable, fetch, use, deploy) stays in one pipeline file.

Honest concession: if your security team already runs Vault, or you need cross-team policies and PKI issuance in one place, the built-in secrets of any CI - Buddy included - are a delivery channel on top of that system, not a replacement for it. Buddy earns its spot for teams that want the pipeline and its secrets to live together; a mature Vault install is the better fit for enterprise-wide credential governance.

Example

The pipeline below shows two patterns side by side. An encrypted Buddy variable (REGISTRY_TOKEN) is used to push a container image. A short-lived database credential is fetched from an external Vault at job time and passed to the deploy step, so the actual password never lives in Buddy at all - only the AppRole material does.

# .buddy/buddy.yml - build, push, and deploy with two secret sources
- pipeline: "deploy-with-secrets"
  events:
  - type: "PUSH"
    refs:
    - "refs/heads/main"
  variables:
  - key: "REGISTRY_TOKEN"
    value: "secure!ghp_replace_me"
    type: "VAR"
    encrypted: true
  - key: "VAULT_ROLE_ID"
    value: "secure!approle_id_replace_me"
    type: "VAR"
    encrypted: true
  - key: "VAULT_SECRET_ID"
    value: "secure!approle_secret_replace_me"
    type: "VAR"
    encrypted: true
  actions:
  - action: "Build and push image"
    type: "BUILD"
    docker_image_name: "docker"
    docker_image_tag: "24"
    commands: |-
      echo "$REGISTRY_TOKEN" | docker login ghcr.io -u ci --password-stdin
      docker build -t ghcr.io/acme/web:$BUDDY_EXECUTION_ID .
      docker push ghcr.io/acme/web:$BUDDY_EXECUTION_ID

  - action: "Fetch short-lived DB credentials from Vault"
    type: "BUILD"
    docker_image_name: "hashicorp/vault"
    docker_image_tag: "1.15"
    commands: |-
      export VAULT_ADDR=https://vault.internal
      VAULT_TOKEN=$(vault write -field=token auth/approle/login \
        role_id="$VAULT_ROLE_ID" secret_id="$VAULT_SECRET_ID")
      export VAULT_TOKEN
      vault kv get -field=url secret/prod/db > db_url.env

  - action: "Deploy with the rotated credential"
    type: "BUILD"
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      export DATABASE_URL=$(cat db_url.env)
      bdy sandbox restart prod-web \
        --env DATABASE_URL="$DATABASE_URL" \
        --env IMAGE=ghcr.io/acme/web:$BUDDY_EXECUTION_ID

  - action: "Verify the deploy is healthy"
    type: "HTTP"
    method: "GET"
    notification_url: "https://web.example.com/healthz"
    retry_count: 6
    retry_interval: 10

Two things are worth flagging. The REGISTRY_TOKEN value is masked in Buddy's UI and in the build log, so docker login --password-stdin cannot spill it in plaintext even if the container image is verbose. The database URL is never stored in Buddy at all; the encrypted VAULT_ROLE_ID and VAULT_SECRET_ID are the only long-lived material, and the credential Vault hands out expires on its own lease. Rotate the DB password in Vault and the next pipeline run picks up the new one with no code change on either side.