home / incident-response / ci-token-tampered-artifact
CI Token Exposure and a Tampered Build Artifact
A leaked pipeline token is used to publish a build artifact that does not match its source.
- Type
- Security incident
- Severity
- High
- Category
- Software supply chain / credential exposure
- Duration
- About 6 hours to recovery
- Times
- Relative to detection (T+0)
Summary
A service token with permission to publish artifacts is exposed. It is then used to publish a build that was not produced from the reviewed source. The incident tested whether the team can notice the mismatch, contain it quickly, find everything that was touched, and restore trust in what it ships.
My role
Incident lead. I owned the decisions, assigned the investigation and communication tasks, and kept the timeline. Other roles: pipeline owner, registry owner, and a communications contact for downstream consumers.
Timeline
relative to detection- T+0
Detection
An alert fires: a build job published an artifact outside the normal release window, using a service token. The on-call engineer opens an incident.
- T+15m
Triage
The artifact is compared with the output of a clean rebuild from the same source. The hashes differ, so the artifact is treated as untrusted. The incident is raised to high severity.
- T+30m
Containment
The service token is revoked, the pipeline is paused, and the published artifact is pulled from the registry. Downstream consumers are asked to stop deploying it.
- T+1h
Scoping
Pipeline and registry logs are reviewed to find every artifact built or published with the token, and every system that pulled them.
- T+3h
Eradication
Affected runners are rebuilt from clean images. All secrets that were visible to the pipeline are rotated, not only the one that was used.
- T+6h
Recovery
Artifacts are rebuilt from verified source on clean runners, compared by hash, and republished. Deployments resume in stages.
- T+2d
Review
Blameless review held. Findings and actions are recorded.
Key decisions
Treat the artifact as untrusted before knowing how it happened
The hash mismatch against a clean rebuild was enough. Waiting for a root cause would have let consumers keep deploying it. Reproducible builds made this a quick, objective check instead of a judgement call.
Rotate everything the pipeline could see
Revoking the one token that was used is not enough, because the same job context exposed other secrets. Rotation cost a few hours of disruption; leaving them cost an unknown risk.
Rebuild runners instead of cleaning them
A runner that held the token cannot be shown clean by inspection. Rebuilding from a known image is faster to trust than a forensic clean-up.
Actions taken
- Contain: revoke the token, pause the pipeline, pull the artifact, notify consumers.
- Scope: list every build and publish made with the token, and every consumer that pulled them.
- Eradicate: rebuild runners, rotate all pipeline-visible secrets, review pipeline definition changes.
- Recover: rebuild from verified source, compare hashes, republish, resume in stages.
- Communicate: short factual updates to consumers at each stage, including what is and is not known.
Post-incident improvements
| Area | Change |
|---|---|
| People | Named owner for the pipeline and for the registry; the on-call runbook includes this case. |
| Process | Publishing outside the release window requires a second approval. Secrets are inventoried and rotated on a schedule. |
| Tooling | Short-lived, scoped tokens issued per job instead of long-lived service tokens. Independent rebuild and hash comparison before release. Alert on publishes outside the window. |
| Detection | Alert on token use from an unexpected runner or address, and on artifacts whose hash differs from a clean rebuild. |
Lessons
- A reproducible build turns “is this artifact the one we reviewed?” into a check that takes minutes.
- Long-lived tokens turn one exposure into a durable foothold. Scope and lifetime matter more than where the token is stored.
- Decide what to rotate before the incident, not during it.
Tags
#incident-response#supply-chain#ci-cd#credentials