Your Build System Knows Everything

On the fourth of January CircleCI published a security alert containing one instruction, which was to immediately rotate any and all secrets stored in CircleCI. Not the secrets belonging to a particular project. Not the ones a customer could show had been used inside some window. All of them, from every project and every shared context, on the working assumption that anything the platform held should now be treated as known to somebody else.

I have read a lot of vendor notices and that one is unusual, because it does not ask you to assess anything. An advisory naming a feature and a date is asking you to scope. This one takes scoping off the table, which is either extreme caution or a statement about what the company could not yet rule out, and from outside there is no way to tell which. The detailed report does not exist yet, and I am writing without it deliberately, because the part worth arguing about does not depend on it.

What the instruction is, as a work order, is an inventory exercise nobody has done. Somebody has to produce a list of every credential the build system holds and then, for each one, find the system it authenticates to and rotate it there. CircleCI was explicit about that second half: rotate keys at the source, meaning the system to which they provide access. Deleting a value out of a CI configuration accomplishes nothing at all if the key it held is still valid at the far end.

And once you write that list out, the striking thing is that it is not a list of build secrets. It is a list of everything your organization can be, plus the platform's own API tokens, which are a way to invoke every credential on it without reading one. That is the subject here. A continuous integration system is a credential vault that also executes arbitrary code submitted by strangers, and it is almost never bought, staffed, or reviewed as either of those things.

Cloud keys are the ones people think of. Access keys for whichever provider you deploy to, in project environment variables or in a shared context so several projects use one copy. They are long-lived by default, because a job that runs at three in the morning cannot ask a human for a fresh one. Their permissions are usually whatever it took to make the deploy stop failing, which is a boundary set by frustration rather than by design.

The tokens that can write to your source come next. An OAuth grant to GitHub, Bitbucket or GitLab, plus deploy keys per project. This is the entry that unsettles me most. Anything that can push a commit or move a tag can put code into the repository that builds and ships your software, which is a supply chain position rather than a build credential, and it is usually granted write access because somebody wanted a version bump automated.

Publishing credentials reach everything downstream. Registry passwords, package publishing tokens, and signing material if you were unlucky enough to keep it there. These write to artifacts other people install, which makes them worth more than the cloud keys beside them. A cloud account compromise is your problem. A publishing compromise is your customers' problem, and you get to explain it.

And then there is whatever your self-hosted runners hold. Registration tokens for the build machines you run on your own network, because some job needed to reach something private. Those machines sit inside the perimeter, receive the same secrets, and are usually outside whatever inventory covers the rest of the estate, because they were stood up by the team that needed them rather than by the team that patches things.

A build system is not a safe. A safe holds valuables and does nothing else. This holds valuables and then, on every push, checks out a working copy of a source tree and runs the instructions it finds inside it. The instructions are in the tree. So what your build system executes is decided by whoever can get code into a branch it builds, and on an open project that is anybody with a browser.

GitHub's own security team wrote the sharp version of this up in August 2021, in a post about pwn requests. The mechanism is a workflow trigger, pull_request_target, which exists because maintainers wanted automation on fork pull requests that could still see repository secrets. Combine it with an explicit checkout of the contributor's branch and untrusted code runs holding the trusted token. The post says outright that this may lead to repository compromise, and that a workflow need not reference a secret to leak one, because the token is in the process anyway.

The other direction is the one Codecov demonstrated. Its Bash Uploader, a script a great many pipelines fetched over the network and piped straight into a shell, was modified on the thirty-first of January 2021 to add one curl call sending the output of env to a host the attackers controlled. In a build container env is the secret store. That ran for two months, until the first of April, when a customer compared the checksum of what they had downloaded against the published one. Codecov disclosed it on the fifteenth.

The two mechanisms look nothing alike and they are the same shape. Somewhere in your build there is an instruction to fetch something and execute it, and the people who can influence what that turns out to be include contributors, dependency maintainers, action authors, base image publishers and whoever operates the networks in between. Each of them is running code in a process holding your production credentials. That is not a misconfiguration anybody introduced. It is what a build is.

The inventory does not exist until you need it. Nobody keeps a current map of which credential in the build system authenticates to which service, because nothing forces one to be kept. Secrets accumulate under names like DEPLOY_KEY and ACME_API_TOKEN, set by people who have since changed teams, for services that may or may not still be in use. The first day of the response goes on working out what you have, and part of the list never gets identified at all.

Rotation has to happen at the far end. Changing a value in CI changes nothing about the credential. The old key stays valid wherever it was issued, so each rotation is two operations in two systems, often owned by two teams, and the gap between them is a window in which either the deploy is broken or the old key is still live. Doing that across a few hundred entries is not a morning's work.

Some of it cannot be rotated on your own schedule. A signing key with a published fingerprint. A credential shared with a partner under a contract. An API key issued by a vendor whose support queue is measured in days. Each of those has a rotation procedure that requires another organization to agree to a date, which is why they are the ones still outstanding a month later, and they tend to be the ones with the longest reach.

And you get to do all of it again next time. This is the part that decides the argument for me. Rotation treats the compromise as an event that happened and is now finished. The property that made it expensive, which is that a static string in a configuration store is a bearer credential valid until somebody deliberately changes it, is exactly as true afterward. You have paid a large one-time cost and reduced the cost of the next one by nothing.

The idea is federation, and it is worth stating plainly because the vendor documentation buries it under configuration steps. Instead of storing a cloud access key in the build system, you configure the cloud account to trust the build platform as an identity provider. When a job runs, the platform mints a short-lived signed token describing that specific job. The job presents the token to the cloud provider, which validates the signature and checks the claims against a trust policy you wrote, and hands back a temporary credential that expires in minutes.

GitHub shipped this for Actions on the twenty-seventh of October 2021, and the part that matters is the token's subject claim, which reads like repo:acme/service:environment:prod. The provider is not asked to trust a repository in general. It is asked whether this run, of this repository, in this deployment environment, may assume this role, so the policy can grant the production role to one repository's production environment. No equivalent policy exists for a static key, because a key is a string and a string carries no context about who presents it.

Then count what is left in the store. Nothing, for that provider. No credential to steal, because none was written down; no rotation to perform, because the token expired shortly after the job that used it; and a stolen copy of your CI configuration yields a role name and an account number. The blast radius of a compromised build platform stops being your whole cloud estate and becomes whatever jobs an attacker can cause to run, which is a smaller problem and a far more defensible one.

I want to be precise about what actually changed, because it is not that the credential got shorter. It is that the credential stopped being a secret you hold and became an assertion somebody makes about one execution. Those are different objects. A secret is valuable to whoever ends up with it. An assertion is worthless to anyone who cannot reproduce the conditions it describes.

Everything that is not a cloud identity system stays. Package registry publishing tokens, container registry passwords, third-party service API keys, the credential for the vendor whose product predates the whole idea. Those are still long-lived bearer strings, because the far end has no notion of a federated identity to trust and no plans to acquire one. At most organizations that is the majority of the list, and pretending otherwise is how a good design turns into a slogan.

The trust policy becomes the thing worth attacking. A policy saying that any workflow in the organization may assume the production role has federated nothing. It has swapped a stolen key for a wildcard. The claims are only as useful as the conditions you match on, and the ones that carry weight are the repository, the environment and the branch reference. Getting this wrong is quieter than leaking a key, because a permissive trust policy never appears in anybody's incident notice.

Somebody still holds the first secret. Federation moves trust; it does not abolish it. Your cloud account now trusts the platform's signing key, which puts that key inside your authorization boundary whether you like it or not. Better than a copied string in a configuration store, and not the absence of a boundary.

The self-hosted runner keeps its own problems. A runner inside your network has network position, and network position is a credential nobody thinks to rotate. If the machine can open a socket to a database directly, no amount of token expiry changes what a job can do while it is running. The remedy there is segmentation rather than identity, and the two get confused constantly because both are filed under security.

None of it survives a job you did not intend to run. Short-lived credentials shrink the window; they do not close the door. An attacker who can cause a build to run on a branch your policy trusts receives a fresh, valid, legitimately issued credential on demand. Which puts the weight back on who can influence what the build executes, and that is a review question about your workflow definitions rather than a configuration question about your identity provider.

I should narrow this. Nothing here would have prevented what happened at CircleCI, whatever it turns out to have been, and a customer who had federated every cloud credential they owned still spent last week rotating registry tokens and deploy keys alongside everybody else. Federation is not a defense against your build vendor being compromised. It is a reduction in what that compromise is worth, which is a smaller claim, and it is the only one the mechanism actually supports.

What stays with me is the shape of the notice. A message that says rotate all of it, with no scoping and no exceptions, is a vendor saying they cannot enumerate what an attacker reached, and the reason they cannot is that everything in there was equally reachable and equally valid. The build system holds a key to every door and opens its own front gate to anyone with a patch to suggest. Most of them have been that for years. Last week is when everyone had to read the list.