The Last Mile to a Phone

The build was finished on a Tuesday. It compiled clean, it ran on a handset wired to the machine that compiled it, and everybody who had been asked to look at it was content. What remained was the item nobody had put on the plan, because from a distance it looks like copying a file to a server. All that was left was other people's phones.

It is worth saying why that item is not small, because the reason is not that anyone was slow. Nothing between a finished binary and a user's device is a build step. Each of those steps is a statement about who you are, made in a form a phone can check while offline, and the tooling is peculiar to operate because it expresses a contract in the vocabulary of a compiler.

The decision underneath it was made once, by two companies, and the rest is downstream. Only signed, attributed, reviewed software runs on these platforms. Given that, a phone needs to verify a signature without asking anyone; a signature needs an issuer; an issuer needs to know who is asking; and knowing who is asking means an account, an agreement, and a chain of documents that must agree before a byte is installed.

So the certificates are not paperwork wrapped around an engineering process. The certificates are the engineering process. Read it that way and every irritating step becomes deterministic rather than arbitrary, which is satisfying, and on the afternoon it is failing, no help at all.

A membership is a contract before it is a login. You start by paying Apple 99 USD per membership year and accepting a license agreement, which is the root of everything that follows. It is not an account in the developer's sense of the word; it is a named party to a contract. Which is why Apple's documentation notes that distribution certificates can be requested only by Account Holders and Admins, and reserves the right to revoke any of them at Apple's discretion.

A certificate says who built it. Apple's certificate authority issues a signing certificate, and the private key for it is generated on one machine, in one keychain, at the moment somebody creates the signing request. That is the point of the design: a private key existing in exactly one place is a claim about a person. It is also the most awkward property in the system.

An App ID says what was built. The bundle identifier is registered with Apple, and the capabilities the app may use are attached to it there rather than to your code: push notifications, iCloud containers, app groups. Entitlements are permissions granted to an identifier, not features you switch on in a project file, which is why enabling one in Xcode and not in the portal produces a build that is correct and unshippable.

A provisioning profile binds the three together. The profile is a signed document stating that this certificate may sign this identifier, with these entitlements, for these devices. It exists because all of that is checked on the device, before the app runs, with no network available. It is the most sensible artifact in the chain and the one that causes the most trouble, because it derives from four things that change on their own schedules.

And a build has to match every part of it. Then the binary has to agree with the profile. The entitlements the code requests must be the entitlements the profile grants, the identifier compiled in must be the identifier registered, and the certificate that signed it must be one the profile names. Five facts, kept in a web portal, a keychain and a project file, and the upload succeeds only when every one of them says the same thing.

The failures share a shape: something is absent, and the tool reports where it looked rather than what is missing. Apple's support page documents the canonical one, the message "Xcode could not find a valid private-key/certificate pair for this profile in your keychain," and then explains that it often happens when you build from a different system than the one you used to request the certificate. The message names a keychain. The problem is a laptop in another city.

Expiry produces the same effect on a delay. Certificates and profiles both expire, and Apple's guidance when one does is to regenerate the profile, remove the stale copy from every device, and re-sign. None of it is difficult. What makes it expensive is that nothing warns you where you are looking, which is a build that worked every day for a year and now does not.

Then there is the pathology that follows directly from the design. A signing identity is a private key living in one keychain. A team is more than one person. So the key gets exported to a file, the file gets passed around, and the organization's identity is now an attachment in a chat thread with a password somebody remembers. Eventually a person tidying the portal revokes what looks like a duplicate, and every machine holding that identity stops being able to ship at the same instant.

Apple's documentation is precise about how bad that is. Revoke an App Store distribution certificate and apps already on the store are unaffected; you simply cannot upload anything new. Revoke an in-house enterprise certificate and users can no longer run the apps signed with it. Same button, two different blast radii, nothing in the interface ranking them. A system designed on the assumption that a signing identity belongs to a person is operated, everywhere, on the assumption that it belongs to a team.

The catastrophe was always the lost key. Android updates are accepted only when the new package is signed with the same key as the installed one, which is the mechanism preventing a stranger from shipping an update to your users. It also means losing your keystore historically ended the app. Not delayed it. Ended it: you publish under a new package name and start from zero installs, with your existing users on a version that can never be updated.

Play App Signing moved the key that matters. Under Play App Signing, Google holds the app signing key and uses it to sign the APKs generated from your bundle and delivered to devices. Since August 2021 new apps must publish as an Android App Bundle, which puts them on that path by construction; apps created before then may still upload an APK and hold their own keys, and Play recommends they stop.

Upload key and app signing key are the part people get wrong. The key you sign with is not the key your users' devices verify. The upload key only proves to Google that a bundle came from you; Google checks it, discards that signature, and re-signs with the app signing key. The consequence is the whole point. A lost upload key is a support request: you register a new one and carry on. A lost app signing key used to be terminal, and now cannot be lost by you, because you never held it.

Here is what that costs, stated honestly. You have handed the key that ultimately signs what ships to the operator of the store. Google's documentation notes that more than 90% of new apps use a Google-generated signing key, which is not downloadable, and that a key upgrade can be requested once a year for installs on Android N and above. So you can no longer produce, on your own hardware, the exact artifact your users run. For most teams that is the right trade, and it should still be a trade somebody made deliberately.

Five weeks from now, on 31 August 2023, Google Play requires new apps and app updates to target Android 13, which is API level 33. The rule for apps already published is sharper: from the same date an existing app must target API level 31 or above to stay available to new users whose devices run a newer Android than it targets. Below that, it is simply not offered to them.

Bumping a target API level is not a line in a config file. Each level you cross opts your code into behavior changes Google has already decided are improvements: restrictions on starting foreground services from the background, an explicit declaration of which components other apps may launch, Bluetooth permissions split into scan and connect grants, notifications becoming something a user must agree to. Every one is a real change with a real test pass behind it, and the release containing them all contains no features.

Apple's version of the moving floor is the toolchain rather than the runtime. Since 25 April 2023, per Apple's list of requirements, an iOS or iPadOS app submitted to the App Store must be built with Xcode 14.1 and the iOS 16.1 SDK or later. That reads like a footnote until you notice what it makes of your build machine: the Xcode version is a compliance artifact, so the macOS version underneath it is too, and the hardware underneath that.

Here is the sentence for whoever funds this. A mobile application has a maintenance floor unrelated to your users or your roadmap, set annually by two companies who did not consult you, and a team shipping no new features still owes a couple of releases a year to stay in the stores it is already in. Budgeting a mobile app as a project rather than a subscription is not optimism. It is a category error, discovered about eighteen months in.

Review is a human process with a queue in front of it, and both companies are candid about that. Apple's App Review page says that on average 90% of submissions are reviewed in less than 24 hours. Google says certain apps are subject to extended reviews, which may take up to 7 days or longer in exceptional cases. Both are honest and neither is a commitment, because an average has a tail and the tail is where your launch date lives.

The guidelines are a document that gets interpreted, which is structural rather than a complaint about reviewers. Apple notes on the same page that over 40% of unresolved issues relate to one guideline covering completeness: crashes, placeholder content, missing information. So the same binary can pass once and fail later, because the reviewer changed, or the guideline was clarified, or what you have always done is what is being looked at this quarter. Review is a judgment, and judgments are not idempotent.

Even the beta has a queue. Apple's TestFlight requires the first build you send to external testers be approved by TestFlight App Review before testing can begin, and builds stop working 90 days after upload, so a slow beta expires under you. This is usually where a team meets the queue for the first time, about ten days before a launch announced two months ago.

And that reshapes how the software gets written. A web team deploys ten times a day and treats a feature flag as good hygiene. A mobile team ships when two other companies say so, which makes flags and server-driven configuration something else: the only writable surface between releases, the only mechanism by which anything changes on a schedule you own. That is not sophistication. That is the floor.

Signing is no longer something you have to understand to use. Xcode manages certificates and profiles automatically now, creating what is missing and renewing what expired, and cloud-managed certificates mean the private key need not be born on a particular laptop. The day it fails you still need all five layers. Until then, a developer ships a first build without learning any of them.

A community built its way out. fastlane's match creates the certificates and profiles a team needs, stores them in a shared repository or bucket, and, in its own documentation's words, automatically repairs broken and expired credentials. A tool whose selling point is repairing your signing setup tells you what that setup does unattended.

Machines and schedules can be rented back. Hosted build services, Apple's own included, turn the Xcode version requirement into a setting rather than a machine you maintain. Play Console's managed publishing separates approval from publication, so review finishes on Google's schedule and the release happens on yours. Neither shortens the queue; both stop it picking your launch hour.

The alternative was tried and it was worse. A platform where unsigned, unattributed code installs freely is a platform with a malware problem, and the people who pay for it are not developers. The identity chain that is so tedious to operate is the same mechanism that means every application on a phone traces back to an accountable party, that an update can only come from whoever shipped the original, and that deleting something removes it.

Read the annoyances again with that in mind and they are security properties viewed from the wrong side. The signature-matching rule that makes a lost Android keystore catastrophic is the rule that stops a stranger updating your app. The certificate that expires is a claim about identity with an expiry on it, which is what such a claim should have. The profile regenerates because it is a statement of fact, and the facts changed.

Which leaves a complaint narrower than the one people usually make, and I think fairer. It is not that the trust model exists. It is that its developer experience was never anybody's objective, and two companies with more resources than most governments have left the error messages this unhelpful this long. Nobody is promoted for fixing a message, so every team learns the chain by breaking it.

I should concede that I have made a good day sound like a bad one. Once this is set up it mostly stays set up: the certificate renews, the profile regenerates, the pipeline runs, and months pass in which nobody thinks about it. The cost is that the knowledge is folkloric rather than documented, and the failures correlate unhelpfully with launches, because a launch is when you change five things at once.

What stays with me is how honest that error message is about its own limits. It names the keychain because a keychain is what a build tool can see, and it cannot say the thing that is true, which is that the private key sits on a laptop in another city and the person holding it is on leave. Somewhere under all of this there is a contract, and a compiler has never had a word for a contract.