Deployment marker

How does a deployment marker work?

A deployment marker is a small event that your pipeline sends to your observability stack the moment a new version becomes live. It carries a timestamp, the service name, the version or commit SHA, the environment, and usually a few tags (region, cluster, team). The observability tool stores it as an annotation on time-series graphs and as a searchable event in its event stream.

Once the annotation exists, three things become cheap. Dashboards render a vertical line at the deploy time, so a latency or error graph shows "before / after" at a glance. Alerts that support change correlation (Datadog Watchdog, New Relic Applied Intelligence, Sentry release health) attribute a spike to the marker automatically, cutting the "was it the deploy?" step out of triage. And retrospective queries can filter to "the 30 minutes after the marker" without anyone having to remember when the release happened.

The mechanics on the pipeline side are boring on purpose: one HTTP POST to the vendor's events or annotations API, done immediately after the traffic-shift step succeeds. Because the marker is cheap to send, teams usually add a second one for rollbacks, and sometimes a third for feature-flag flips that matter to the same graphs.

Why do deployment markers matter?

The first ten minutes after a release are where most change-caused incidents surface, and they are also where the on-call is under the most pressure to answer one question: is this our deploy? A dashboard without markers forces that question to be answered by memory, by scrolling through chat, or by pulling up the pipeline history in another tab. A dashboard with markers answers it in one glance.

That single glance changes downstream behaviour in real ways:

• Faster rollback decisions. If the p99 latency line kicks up right at the annotation, no one argues about coincidence. The rollback happens sooner, which is exactly where MTTR gets shorter.
• Cleaner DORA data. Change failure rate needs an accurate "when did the change land" timestamp for every incident it counts. Markers are the source of truth that link the incident to the release without a human filling in a form.
• Blameless retros with facts. During a postmortem, "we deployed at 14:07 and the queue depth climbed at 14:11" is a specific claim you can point at, not a recollection.
• Alert noise reduction. Some alerting products suppress or de-rank alerts for a short window after a deploy annotation, on the theory that a fresh, expected change is more likely to explain a spike than a fresh, unrelated one.

None of this needs anything fancy on the app side. The marker is a byproduct of the pipeline, and its value compounds as more teams push to the same observability backend.

What should a marker include?

A useful marker is small and consistent. The fields that pay for themselves in incident reviews:

• Service and environment. service:checkout, env:prod. Without these you cannot filter dashboards or route alerts.
• Version and commit SHA. The short SHA is enough for humans; keep the full one in a tag or in the event body so a linked deep link resolves.
• Region or cluster. Deploys often ramp one region at a time; a global annotation is misleading if only us-east-1 actually shifted.
• Pipeline URL. A link back to the exact run that produced the marker. The on-call clicks once and reads the logs, no context-switching.
• Actor. The user or bot who triggered the pipeline. Useful when a manual retry or a scheduled run behaves differently from a normal push.
• Kind. deploy, rollback, flag-flip. Alert-correlation logic often keys on this.

Consistency beats richness. If every service uses the same tag names, you can pivot dashboards across teams; if each team invents its own vocabulary, you lose most of the value.

When should the marker fire?

Fire it from the step that actually shifts real traffic to the new version, not from the build or the deploy-to-staging step. Concretely:

• Blue-green: post the marker at the moment the router flips from blue to green, not when green finishes deploying.
• Canary: post one marker for the initial small slice, one for each promotion step, and one for the final 100%. Each annotation tells a different story on the graph, and the alerting engine can weight the final promotion differently.
• Rolling: post it when the first pod carrying the new version starts serving live requests, and again when the rollout is fully complete. Some teams post one per batch on large fleets; that gets noisy on small ones.
• Feature-flag rollout: post a flag-flip marker when the audience percentage crosses a meaningful threshold (say, 1%, 25%, 100%). This is what turns "we deployed hours ago, why is it breaking now?" into a visible timeline.

Whatever the strategy, keep the marker as close as possible to the moment users are affected. A marker that fires ten minutes before or after the real cutover destroys its own signal.

How do popular CI/CD and observability tools handle deployment markers?

Every serious observability product exposes an events or annotations API, and every serious CI/CD tool can hit it. Which combination is right for you usually comes down to where you already spend most of your telemetry budget.

• Datadog has a first-class POST /api/v1/events endpoint plus a dedicated deployments beta and inline deployment tracking on APM. Deploy events flow into Watchdog for automatic change correlation. If your organisation already runs on Datadog, the marker is a two-line HTTP call and every graph, alert and Watchdog story picks it up for free.
• New Relic Change Tracking (formerly the Deployment Markers API) turns deploys into first-class objects that show up in the errors inbox, APM overview, and Applied Intelligence correlation. The newrelic-cli deployments command was designed specifically for CI runners, so a curl-free one-liner drops into any pipeline. For teams standardising on New Relic, this is close to friction-free.
• Sentry Releases ties a marker to a git commit range, uploads sourcemaps and computes release health (crash-free sessions, adoption, regression detection) automatically. If your primary observability need is error tracking with commit-level attribution on the frontend, Sentry is a stronger fit than a generic annotation. The release model gives you things a plain marker never will, like automatic regression labels on issues.
• Grafana exposes an annotations API (POST /api/annotations) that any dashboard can subscribe to via a data source query. Grafana Cloud adds a dedicated Deployments feature that consumes the same events. Grafana is the flexible pick when your metrics live across Prometheus, Loki and Tempo and you want the annotation to show up on every relevant panel.
• Honeycomb Markers are lightweight, filterable by service, and render as vertical lines on any BubbleUp query. On a Honeycomb-first team, one curl post-deploy is all it takes.
• Argo CD publishes sync and health events that many observability tools ingest directly, and Argo Rollouts emits per-step webhooks that can be mapped to markers. If your rollouts are declarative in Kubernetes and you already run Argo, the marker can be a side effect of the reconciliation loop rather than a CI step.
• CloudWatch, Azure Monitor and Google Cloud Monitoring each accept custom events that render as annotations on their native dashboards. Convenient if your CI already has cloud credentials in scope.
• Buddy is one of the options we recommend when the marker needs to live in the same pipeline as the deploy and stay portable across observability backends. An HTTP action fires the annotation to Datadog, Grafana, Honeycomb or an internal endpoint using the pipeline's built-in variables ($BUDDY_EXECUTION_ID, $BUDDY_EXECUTION_REVISION, $BUDDY_PIPELINE_NAME), and because the marker step sits next to the deploy step in the same YAML, an on-call reading the pipeline log sees exactly which annotation went where. If you later switch from Datadog to Grafana, one action changes, not a separate marker service.

Honest summary: if you are all-in on one vendor, use that vendor's native tracker (New Relic Change Tracking, Sentry Releases, Datadog Deployments) - the integrations they get in return are worth more than a generic HTTP post. If your telemetry is spread across several tools, or you want the marker logic to sit inside the pipeline rather than in a plugin, a general CI/CD tool posting HTTP events (Buddy, GitHub Actions, GitLab CI, Jenkins) keeps the marker portable.

Example

The pipeline below builds, publishes an artifact, routes production traffic to it, and then posts two deployment markers: one to Datadog (drives graph annotations and Watchdog correlation) and one to Grafana (renders on every dashboard using its default annotations layer). Both markers include the commit SHA, the pipeline run ID and enough tags to slice by service and environment. If a rollback pipeline runs later, it fires the same actions with kind: rollback so the timeline stays honest.

# .buddy/buddy.yml - deploy and post deployment markers
- pipeline: "deploy-and-annotate"
  events:
  - type: "PUSH"
    refs:
    - "refs/heads/main"
  actions:
  - action: "Build and test"
    type: "BUILD"
    docker_image_name: "node"
    docker_image_tag: "20"
    commands: |-
      npm ci
      npm test
      npm run build

  - action: "Publish artifact and shift traffic"
    type: "BUILD"
    docker_image_name: "ubuntu"
    docker_image_tag: "22.04"
    commands: |-
      bdy artifact publish web-app:$BUDDY_EXECUTION_ID ./dist --create
      bdy distro route update prod-distro --domain=example.com \
        --target=artifact=web-app:$BUDDY_EXECUTION_ID

  - action: "Post deployment marker to Datadog"
    type: "HTTP"
    method: "POST"
    notification_url: "https://api.datadoghq.com/api/v1/events"
    headers:
    - name: "Content-Type"
      value: "application/json"
    - name: "DD-API-KEY"
      value: "$DATADOG_API_KEY"
    content: |-
      {
        "title": "deploy web-app $BUDDY_EXECUTION_ID",
        "text": "commit $BUDDY_EXECUTION_REVISION shipped by $BUDDY_INVOKER_NAME (run $BUDDY_EXECUTION_URL)",
        "tags": ["env:production", "service:web-app", "source:buddy", "kind:deploy"],
        "alert_type": "info",
        "aggregation_key": "deploy-web-app"
      }

  - action: "Post annotation to Grafana"
    type: "HTTP"
    method: "POST"
    notification_url: "https://grafana.example.com/api/annotations"
    headers:
    - name: "Content-Type"
      value: "application/json"
    - name: "Authorization"
      value: "Bearer $GRAFANA_TOKEN"
    content: |-
      {
        "text": "deploy web-app $BUDDY_EXECUTION_ID (commit $BUDDY_EXECUTION_REVISION)",
        "tags": ["deploy", "production", "web-app", "buddy"]
      }

Two things make this shape work in practice. First, the marker actions fire only after the traffic-shift step succeeds, so the annotation lands at the exact moment users start seeing the new version, not when the build finished. Second, the marker payload includes a link back to the pipeline run, so anyone opening the annotation on a dashboard is one click away from the log that produced it. Add a matching rollback pipeline that posts a kind:rollback marker on the same endpoints, and the incident timeline builds itself.