The HTTP/2 Bug Nobody Wrote

Cloudflare, Google and Amazon all published on the tenth of October, describing attacks they had absorbed through late August and September. Google put its peak at 398 million requests per second and noted that two minutes of it carried more requests than Wikipedia served article views in the whole of September. Cloudflare measured just above 201 million. AWS reported over 155 million. The three numbers are not comparable to each other and should not be added together, because each is one company measuring its own network.

What they have in common is more interesting than their size. Cloudflare said the traffic came from a botnet of roughly twenty thousand machines, which is small. Attacks of this shape have historically needed vastly more, and the ratio is the finding: something about the request being sent made each machine enormously more efficient than it should have been.

Three independent networks, running three unrelated stacks written by different people in different languages, hitting the same wall in the same weeks and disclosing on the same morning under one identifier. That pattern only happens for one reason. The problem was not in anybody's code. It was in the document all of that code was written to implement, and correctness was no defense.

HTTP/2 puts many requests on one connection. The version that shipped in 2015 replaced the old model of one request per connection with streams: many logical requests multiplexed over a single TCP connection, interleaved, in flight together. That is the feature. It is the reason a page with a hundred assets stopped needing six connections and a queue, and it did solve a real problem that browsers had been working around for a decade.

The server sets a limit on how many at once. Because unbounded concurrency would be absurd, the protocol has a setting, SETTINGS_MAX_CONCURRENT_STREAMS, and the server tells the client its value at the start of the connection. A hundred is a common choice. This is the control a capacity plan is built on: whatever else happens, one connection can only have that many requests outstanding.

A client can cancel a stream unilaterally. The protocol also lets either side abandon a single stream without disturbing the others, by sending a RST_STREAM frame. This too is a good feature, and it is why a browser can stop fetching an image you have scrolled past without tearing down the connection carrying the rest of the page.

Only live streams count against the limit. A canceled stream is finished, so it is no longer outstanding, so it no longer counts. As Cloudflare put it, when a client cancels a stream it instantly gets the ability to open another one, and stream concurrency on its own cannot mitigate the attack no matter what value the server chose. Open a stream, reset it in the same breath, repeat. The limit is real and it never binds.

A HEADERS frame arriving is a request, and a server does not wait to see whether the client will change its mind. It parses, allocates, decompresses the header block against connection state, matches a route, and in most architectures hands the request to a backend or a worker before any cancellation could arrive. The RST_STREAM lands after all of that. The work is done and paid for; only the accounting is reversed.

So the client's cost per request falls to a few dozen bytes in a frame it can pipeline without waiting for anything, while the server's cost per request stays exactly what it was. That asymmetry, and not the peak figures, is the whole vulnerability. Twenty thousand machines were enough because each one was buying requests at a discount that the protocol handed it in writing.

Here is the sentence I would want to keep. The feature decoupled the creation of a request from the accounting for a request, and every capacity model in the industry is written in terms of the accounting. Connections per second, concurrent streams, requests in flight: these are the numbers on dashboards and in autoscaling rules and in rate limiter configuration, and none of them describes a stream that was created and destroyed between two packets.

I want to be careful here, because this is not an argument that HTTP/2 was a mistake. Head-of-line blocking on the old protocol was a genuine cost paid by every user on every page load, multiplexing is the right fix for it, and I would not go back. What I would say is narrower: a protocol version is a change to the cost model of your service, and it arrived through a checkbox described in terms of page load time.

It came with the load balancer. Very few teams sat down and chose HTTP/2. It arrived enabled by default on a managed load balancer, a CDN, an ingress controller, or in a web server upgrade, and it was accepted because the release note said faster page loads and there was no reason to argue. The adoption decision, if there was one, was made by whoever accepted the default.

The benefit was real and almost nobody measured it. Multiplexing helps most for pages with many small assets over high-latency links. That describes some workloads and badly misdescribes others, particularly an API serving a handful of large JSON responses to a client in the same region. I have never seen a team measure the before and after on their own traffic, which means the benefit is usually a belief and the cost model change came along with it unexamined.

The edge speaks one protocol and the backend another. In the common arrangement the terminator speaks HTTP/2 to the world and plain HTTP/1.1 to the application behind it. That is a sensible design, and it means the application never sees a stream, and the operator reasoning about protection sits behind the very layer where the attack lives. Whatever counters requests for you is on the wrong side of the translation.

Nobody owns the question. Ask which protocol versions your public endpoints negotiate today and you will usually get an answer from three people who each know their own layer. The version is a property of the edge, set by a platform team, consumed by application teams who do not know it changed, and it is nobody's line item until an advisory names it.

On the eleventh, curl published a heap buffer overflow in its SOCKS5 handshake, present since 7.69.0 and fixed in 8.4.0 the same day. It is a memory-safety bug of the classic kind: one project, one wrong length check, one patch, and the remedy is to upgrade and be done. Painful and bounded. You can tell your management exactly what you did about it.

Rapid Reset has no version of that sentence. There was no single upstream to patch, because every server that implemented the specification faithfully behaved the same way, so the response was dozens of separate vendors independently inventing mitigations for behavior their software was correct to exhibit. That is why the fixes are all heuristics: cap resets per connection, close connections that abuse the pattern, count canceled streams against a budget. None of them is described in the protocol document.

The general form is a question I would now ask before adopting any protocol version, and it takes about an hour to answer. Not whether it is faster, which the announcement will tell you, but what it changes about the cost of sending one request to me and the cost of serving one. If the answer is that requests got cheaper for the sender and no cheaper for me, that is not an upgrade. That is a new capacity plan, and somebody should write it before the traffic arrives.

I should concede the practical scope of this, because for most operators the honest answer that week was to wait for a patch and change nothing else. If you serve through a large network, its people fixed this before you read about it, and the mitigations that matter are in software you did not write. Treating that as a failure of your own engineering would be self-flattery. The exposure that is genuinely yours is the endpoint you terminate yourself, and there are usually a few of those that nobody remembers standing up.

What stays with me is that the specification described this exactly. Streams can be canceled, canceled streams do not count, and both sentences were published in 2015 and read by thousands of implementers who all did the correct thing. There was no moment where somebody was careless. A protocol is a contract about what the other side may ask of you, and if you have adopted one without reading it in those terms, you have accepted a bill whose amount is set by whoever connects to you next.