Hosted Does Not Mean Patched

On the eighth of November 2022 Microsoft shipped a fix for CVE-2022-41080, described in the advisory as an Exchange Server elevation of privilege vulnerability. Microsoft scored it 8.8 and the National Vulnerability Database later scored it 9.8, the gap being whether you assume the attacker already holds a credential. Either way it read as a privilege bug on a mail server, in a month with a great many other entries, and it was triaged accordingly.

Six weeks later, on the twentieth of December, CrowdStrike published an analysis of a technique it called OWASSRF, found while investigating a series of Play ransomware intrusions. The technique chains CVE-2022-41080 with CVE-2022-41082 to reach the PowerShell remoting service through the Outlook Web Access front end. That is remote code execution, assembled from a privilege bug the notes had not connected to anything.

Then, in the first week of January, Rackspace named the root cause of the attack that took down its Hosted Exchange service in early December as a zero-day exploit associated with CVE-2022-41080. A vulnerability patched in November, described as a zero-day in January. Both descriptions are accurate. The bug was known and fixed; the way to reach code execution through it was not.

So here is the sentence the rest of this turns on, and it is not really about Exchange. A patch note describes what the vendor found, not what the bug is worth in combination with something else, and every triage process I have seen reads it as though it described the second thing.

A pair of bugs came with a fix you could apply on a Tuesday. In late September 2022 two Exchange vulnerabilities went public together, a server-side request forgery and a remote code execution, and CISA added both to its known exploited catalog on the thirtieth. Microsoft published mitigation guidance alongside the advisory, including a URL rewrite rule for the front end. No reboot, no cumulative update, no downtime.

The rule matched two words in a request. The published rewrite blocked requests whose decoded URI contained both autodiscover and powershell. That is not a repair to Exchange. It is a written description of the route the attack had been observed taking, expressed as a pattern, and it does exactly what the pattern says and nothing else.

OWASSRF walked in through a different door. CrowdStrike's finding was a previously undocumented way to reach PowerShell remoting through the Outlook Web Access endpoint instead. No autodiscover anywhere in the request, so the pattern does not match, so, in their words, the request will not be dropped. The mitigation was working correctly the entire time.

The November patch held and the mitigation did not. CrowdStrike's recommendation is worth quoting exactly, because it is the whole argument in one line: organizations should apply the November 8, 2022 patches for Exchange, since the URL rewrite mitigations for ProxyNotShell are not effective against this exploit method. CISA added CVE-2022-41080 to the catalog on the tenth of January, with a federal remediation deadline of the thirty-first.

A mitigation is a description of an attack. A patch is a change to the program. The first is written in the vocabulary of what somebody has already seen happen, and it holds for exactly as long as the attacker keeps doing that thing. The second alters what the code is able to do, and the code has no opinion about which endpoint the request arrived on.

I understand entirely why the mitigation gets chosen. It applies in minutes, needs no maintenance window, breaks no integrations, and can be reversed by deleting a rule. The patch is a cumulative update to Exchange, and Exchange cumulative updates have a reputation for being disruptive, which I am reporting as reputation rather than as anything I have measured. Given a choice between a change with a known cost and a change with an unknown one, on a service every employee touches, the mitigation wins that conversation in most rooms I have been in.

The asymmetry is in how the two fail. A patch that does not apply fails loudly, in a maintenance window, with somebody watching. A mitigation that no longer covers the attack fails silently and continues to report success, because the rule is still installed and still matching the thing it was written to match. Nothing anywhere tells you the ground has moved.

The operation transfers and the consequence does not. Managed hosting moves the running of a thing to somebody with more practice at running it, which is usually a real improvement. It does not move the mailboxes, the contracts inside them, the regulator who asks about them, or the customers who cannot work today. Those stay exactly where they were.

Nobody ever agreed who decides a patch is urgent. Read a hosting agreement and you will find availability targets, support response times and a liability cap. You will not find a clause naming who triages a vulnerability, against what criteria, or on what clock, or one saying that a vendor-supplied mitigation may or may not stand in for a patch. That decision is being made on your behalf every month by people you cannot name.

And you cannot check the answer. On a multi-tenant platform there is no build number you can query, no patch level you can audit, and no way to establish today whether the fix you read about last week is on the servers holding your mail. The reporting you get is the reporting the provider chose to offer, which is a description of the service rather than of the software.

One decision covers every tenant at once. The efficiency of shared hosting is that a small team operates one platform for everybody, and the exposure is the same sentence read backward. A single triage call, made once, applies to every customer simultaneously, so the thing you diversified away by not running it yourself has been quietly recentralized on the other side of the invoice.

Rackspace did not restore Hosted Exchange. It worked on recovering mailbox data for customers through its portal and moved them onto Microsoft 365 with licenses at no cost, and the service itself is not coming back. A product that existed at the end of November stopped existing, not because it was sold or sunset on a published schedule, but because an incident made continuing with it a worse business than not continuing with it.

I do not think that was the wrong call on its own terms, and it is worth saying so plainly. Rebuilding a multi-tenant Exchange platform to a standard you would defend afterward is an enormous undertaking, the market had been moving to Microsoft 365 for years regardless, and a grudging rebuild would probably have served those customers worse over three years than the migration did. It was a defensible decision. It was also not a decision any of the customers were party to.

Which is the risk I want named, because it appears in no service level agreement I have ever read. An availability credit prices minutes of downtime for a service that continues to exist. Nothing in the document covers the case where the service stops existing, and the continued existence of the thing holding your data turns out to be an output of somebody else's capital allocation rather than a term you negotiated.

Ask how patch urgency gets decided, and get it in writing. Who triages, against what criteria, on what clock, and what the escalation looks like when a customer disagrees. Any provider worth buying from has an answer. The useful signal is whether they have ever been asked, because a process that has to be invented during the call is not a process.

Ask what a mitigation is allowed to stand in for, and for how long. The honest answer is that mitigations are sometimes correct and always temporary. What you want is a stated maximum window, and a commitment that a vendor mitigation is a bridge to the patch rather than a substitute for it. If nobody has a number, the number is indefinite.

Ask how you get your data out on the worst day. Not the export documented for orderly migrations, which assumes a healthy control plane and a cooperative timeline. The one that works while the platform is offline and the support queue is measured in days. If the only copy of your mail lives in the provider's storage, then your recovery time is theirs and you have no way to shorten it.

Ask what happens if they stop offering it. Notice period, migration assistance, data format. Then ask whether those terms change when the reason is an incident rather than a strategy, because that is the case where you need them and the case where the provider is least able to honor them.

I should be careful not to let this become a claim that self-hosting would have gone better, because I do not believe it. Anyone running their own Exchange servers in November 2022 read the same advisory, saw the same words elevation of privilege, weighed the same disruptive update, and a great many reached for the same rule. The error was industry-wide because the information needed to make the call correctly was not in the document the call was made from. Managed hosting did not change those odds. It changed who made the call.

What stays with me is the rewrite rule itself. Two words, matched against a URL, standing in for a repair on a service the whole company depends on. It is a note taped to one door describing the person who came through it last time, and it was accurate, and it held right up until somebody used the other door. A patch does not care which door you use, and that is the entire difference between describing an attack and removing the thing it needs.