Pip Against the Operating System

There are two package managers on a Debian or Ubuntu machine that both believe they own Python libraries, and for most of their lives neither had been told about the other. apt installs into /usr/lib/python3/dist-packages and writes down every file it put there, so dpkg can name the owner of any path on the disk. pip, run with sudo and no virtual environment, installs into /usr/local/lib/python3.11/dist-packages and writes down nothing apt can read.

Those two directories are not equals. The one under /usr/local comes earlier on sys.path, which is the entire point of /usr/local and has been since long before Python existed. So a library installed by pip does not collide with the apt version. It shadows it, silently, while apt goes on reporting, correctly by its own records, that its own version is the one installed.

The worse case was sudo pip install --upgrade, because upgrading means uninstalling, and pip was happy to remove files it had not put there. You could end up with a dpkg database describing files that no longer existed, and a package the system considered healthy whose contents had been replaced by a stranger. Nothing errored. The next thing to fail did so for reasons that pointed somewhere else.

Here is the sentence the rest of this is about. Two package managers claimed the same directory tree for twenty years, and the arrangement held only because almost nobody exercised both sides hard enough to see it. PEP 668 is the settlement, and like most settlements it looks like a loss to whoever had been quietly winning.

It is a marker file, and nothing else. PEP 668 defines one file, EXTERNALLY-MANAGED, sitting in the standard library directory. Its presence tells an installer that this site directory belongs to something other than the installer. No daemon, no lock, no permission bit. The whole enforcement surface is whether a file exists on the path sysconfig reports, so any distributor opts in by shipping one and any operator opts out by deleting it.

The distribution writes the error message. The file is parsed as an INI document, and if it carries an [externally-managed] section the installer reads an Error key out of it and prints that text inside its own failure. This is the genuinely unusual part. Your operating system vendor is now the author of a paragraph appearing above a pip stack trace, which is why the wording differs between distributions and why it reads like release notes.

pip started reading it in January. pip 23.0, released on 30 January 2023, added the logic to read the marker, described in its own changelog as letting a downstream distributor prevent users from modifying the externally managed environment. The reader arrived first and did nothing, because nothing shipped the file yet. Two weeks later the python3.11 package declared itself externally managed. The break needed both halves, landing six weeks apart in different projects.

The escape hatch is named after the damage. pip install --break-system-packages restores the old behavior exactly, and since every pip option has a matching environment variable, PIP_BREAK_SYSTEM_PACKAGES=1 turns it off for a whole container build. What I like is the name. It does not say --force or --allow-unsafe; it says what happens, so every Dockerfile setting it carries a one-line confession in version control.

The Python under /usr/bin on a server is not there for you. It is there because the machine is partly written in it. The thing that configures the interface you are connected over, the thing that applies security updates while you sleep, the thing that runs on first boot to set the hostname and install your keys: on a Debian-family system a striking number of those are Python programs importing from the directory sudo pip install writes into.

So the failure modes are not comparable. If apt breaks pip, you get an ImportError in an application, at a moment when you are looking at the application. If pip breaks apt, you get a host that can no longer patch itself or configure its network, and you find out at the worst available moment. When two systems contend for one resource, the one whose failure is unrecoverable ends up owning it, and that is not a political outcome.

It also explains why the resolution is a refusal rather than a merge. apt resolves against a distribution's full package graph under the constraint that the shipped system works as a unit; pip resolves against the whole of PyPI, against whatever the application asked for. Those are different problems with different definitions of correct, and the honest answer to running both over one directory is to stop.

Start with the Dockerfile that installs into the operating system. FROM ubuntu, then apt install python3-pip, then pip install -r requirements.txt is among the most common four lines in the industry, and what it does is resolve your application's dependencies into the distribution's namespace. Application and base image have been sharing one resolution the whole time. Nobody chose that; it is what you get when the default install location is a directory somebody else occupies.

Then there is the build that was never reproducible. If your application imports a library the base image also ships, which copy wins was decided by path order and by whether the last person to touch the image ran pip or apt. Rebuild against a newer base and the answer can change with no edit to your code or your requirements file. PEP 668 did not create that problem. It made a large one loud.

Next comes the tool installed globally, and the tool installed after it. sudo pip install for something you want on your PATH was the documented first step in a great many quickstarts for a decade, and it means the tool's pins are now the machine's pins. Install a second such tool with incompatible requirements and it silently changes the first, which is precisely the failure virtual environments were built to prevent.

And nobody has looked at the provisioning step in years. A configuration run that installs a Python library into the system interpreter so a later step can import it is widespread, and it is the case I have most sympathy for. It predates the team running it, it worked, and it breaks on a base image bump with an error naming none of that history. The people hit hardest did nothing wrong and got no notice, and it is still the right change.

A virtual environment is the correct answer and nearly free in a container: python3 -m venv /opt/acme, its bin at the front of PATH, and the application now has a dependency set the operating system neither sees nor competes with. For command line tools pipx does the same thing per tool, which is the direct replacement for the sudo pip install habit. For something the whole machine genuinely needs, install the distribution's build with apt, which is what apt was for.

The fourth is the flag, and I want to be fair to it. Setting --break-system-packages in a container rebuilt from scratch every time and running one process is not reckless, because the thing it can damage does not survive the build. Setting it on a long-lived server that patches itself is a different act with the same three words. The flag cannot tell those apart, which is why the useful rule is not banning it but requiring a comment saying which one this is.

A late error is documentation of a coupling you already had. Nothing about anyone's system got worse in April. A dependency between an application and its base image that had existed for years acquired a message. The instinct when that happens is to route around the message; the more useful instinct is to read it as a report of what your build has been doing, in a place you never looked, for as long as it has existed.

Two owners and no arbitration is not a design. It is an argument that has not started yet, and it stays quiet exactly as long as nobody exercises both sides in the same week. Shared mutable directories, config files edited by two tools, tables written by two services: in each case the calm is a function of low usage rather than of correctness, and the discovery gets made by whoever is on call.

The one who cannot recover ends up owning the resource. When two systems contend and somebody has to win, the tiebreaker worth using is which failure you can walk back. An application that will not start is a bad hour. A host that cannot apply its own security updates is a fleet-wide problem found months later. That asymmetry settles more architectural questions than preference does, and it is knowable before anything is built.

I should narrow this, because it has been unfair to the people it describes. The bare pip install everybody learned was not their mistake. It was in the documentation, it worked, it kept working for a decade, and the conflict underneath it was invisible from where they stood. A change correct in mechanism can still land as a tax on people who had no way to know, and packaging has been running on that debt rather than paying it.

What stays is the shape rather than the specifics. Somewhere in the system you run there is a directory two things write to and neither one knows it, and the reason nothing has gone wrong is not that the arrangement is sound. It is that usage has stayed low. The marker file is what it looks like when somebody finally writes the argument down, and the argument was always there, sitting in a path both sides had been resolving against since before either of them cared.