Mirror the Registry You Depend On

On 14 March, Docker told the members of Free Team organizations that the plan was going away and that they had until 14 April to move. Two days later it published a post titled, more or less, we did a terrible job announcing this, which clarified among other things that public images would only leave Docker Hub if their maintainer chose to delete them.

On 24 March the whole thing was withdrawn. Scott Johnston wrote that both the communications and the policy were wrong and that Docker was reversing course, refunding anyone who had upgraded in the interval. That is a faster and more complete retraction than most companies manage, and I want to say so before I spend the rest of this arguing that it changes nothing.

Because the interesting part is what those ten days did to a reader. Docker put the affected population at under two percent of users, which sounds reassuring until you notice that the share of accounts tells you nothing about the share of pipelines. Nobody could enumerate which of their builds referenced a name that was about to become unowned, and that inability was the actual finding.

So here is the claim, and the withdrawal does not touch it. Most production systems carry a runtime dependency on a registry they do not run, they have never priced it, and they discovered it from an email rather than from an architecture review. The plan is gone. The dependency is exactly where it was.

A pull is a network call to another company. Every scale-up event, every replaced node, every fresh CI runner, every rollback to an older tag. Those are not build-time operations. They happen at three in the morning under load, which is the least convenient moment to discover that a third party is on the critical path of your capacity.

The name is a pointer, not the artifact. A tag is a mutable label inside a namespace somebody else administers. It can move, and the thing it points at can change without the string changing. A digest is the only reference that is a fact about content rather than a claim about ownership, and almost nothing in the wild is pinned that way.

The namespace is the asset nobody guards. If an organization is deleted, the name it held becomes available. Licenses do not protect a name, and neither does anything about the layers underneath it. Every deployment referring to that name is now trusting whoever registers it next, which is a very different risk from the outage everyone was worried about.

The limits were always there to be read. Since November 2020 anonymous pulls have been capped at 100 per six hours, counted per IPv4 address or IPv6 /64 subnet, and free authenticated accounts at 200. A CI fleet behind one NAT address shares a single bucket, which is why the failure usually arrives as an unexplained build failure rather than as a policy notice.

My answer is that image pulls have never failed loudly enough to become a line item. When the registry is slow you get a long pull; when you are rate limited you get an error that reads like a network blip; and both go into a retry loop that somebody added years ago for exactly this reason. The dependency is real, and it is invisible in precisely the way that keeps it off a diagram.

Things acquire owners when they produce invoices or incidents with names. A pull-through cache produces neither. It prevents an incident that has not happened, on a dependency nobody has written down, and the person who would build it cannot say what it saves without inventing a number. That is not a failure of judgment. It is what happens to any cost that never gets totaled.

The ten days in March were useful because they briefly made the total visible. For a week and a half, engineers who had never thought about their registry were reading a deprecation notice with a date on it and working out what would break. The plan was withdrawn before anyone had to find out, which is a mercy and also the reason almost nobody acted afterward.

It is a proxy with a disk. The upstream registry image itself runs in pull-through mode with one configuration key naming what it fronts, and every artifact manager on the market does the same job with a nicer interface. Clients point at it by setting a mirror in the container runtime's configuration, which is one file on each node.

It changes the shape of a bad day. With a warm cache, an upstream outage stops being an availability event and becomes a news item. You can still scale, still replace a failed node, still roll back to yesterday's tag, for as long as the cache holds what you actually run. That is the entire purchase.

It pays on ordinary days too. Pull latency on a scale-up drops to local network speed, cross-region egress falls, and the rate limits stop being reachable because a hundred nodes now make one upstream request between them. None of that is dramatic, which is why it is worth naming: the case does not depend on an outage arriving.

Be clear about what it is not. A cache holds what you have already asked for. A cold entry still misses, and a cache is not an archive with a retention policy. If you need a guarantee that a specific artifact exists in five years, you need a copy you deliberately pushed, not a copy you happen to have kept.

The full form is to publish into a registry you control everything you depend on: base images, sidecars, the build tooling, the handful of third-party services you run as containers. You copy them in deliberately, you tag them yourself, and your manifests never name an external host at all. Then the external registry is a supplier you restock from rather than a component you run on.

That is real work, and I would not pretend otherwise. Somebody has to decide what gets copied, keep it current, and answer for the copy being older than upstream when a vulnerability is announced. For most teams the pull-through cache gets ninety percent of the benefit for a tenth of the effort, and I would start there and stop there unless something specific pushed further.

The distinction I would insist on is between availability and trust, because mirroring is regularly sold as fixing both and it fixes one. A cache faithfully reproduces whatever it was handed, including a tag that moved to something it should not have. If you want the second property you want digests in your manifests, which costs one extra field and an occasional deliberate bump.

And digests are where the namespace problem goes to die. A digest cannot be reassigned by a company reorganizing its plans, a maintainer deleting an account, or a stranger claiming a freed name, because it is a hash of the thing rather than a label attached to it. It also makes upgrades explicit instead of ambient, which some teams will experience as a loss.

List the pulls you cannot survive losing. Walk the manifests and the build files and write down every external image reference. It is a grep and an hour, and the list is always longer than the guess, because half of it arrived inside a base image somebody chose in a hurry.

Put a cache in front of the lot. One instance, one mirror setting in each node's runtime configuration, and it is warm within a day of normal traffic. Do it before deciding anything else, because it is the only step here that reduces exposure without requiring a single change to a single manifest.

Pin base images by digest. Tags for convenience, digests for anything that runs in production. The cost is that an upgrade becomes a commit somebody makes rather than a surprise that arrives, and the benefit is that nobody who acquires the name later can change what your servers execute tonight.

Copy the few you truly could not rebuild. Not everything, and not on a schedule that needs a team. The small number of artifacts whose disappearance would stop you cold are worth a deliberate push into storage you own, with the same care you would give a database backup.

I should be careful about the size of this claim. Docker Hub is not going away, the plan was withdrawn inside ten days, and a team that ignores all of the above will very probably be fine for years. The argument is not that an outage is coming. It is that the cost of finding out is asymmetric: the cache is an afternoon, and the alternative is discovering the dependency during the hour you most need to scale.

What the ten days really exposed was the difference between a supplier and a component. A supplier can change its terms and you have time to respond. A component is inside the machine, and you find out where it sits when it stops turning. Nothing about a container image tells you which of the two you are holding, and the only way to make it a supplier is to keep a copy on your own shelf.