When the Debugger Stops Helping

There is a specific twenty minutes I have spent more times than I would like. A process is producing a wrong value, I attach a debugger, I set a breakpoint on the line where the wrong value has to be born, and I run it. The value is correct. I move the breakpoint down a line. Still correct. I put it past the point where the program has demonstrably already gone wrong, and it is correct there too, and now the program does not fail at all.

The instinct at that moment is to doubt yourself, and the instinct is wrong. Nothing about the program changed. What changed is that a debugger works by stopping the process, and stopping the process changes when things happen relative to each other. If the defect lives in the ordering of two events rather than in the arithmetic of either one, the act of looking has already removed it. The tool did not fail. It answered accurately, about a program that no longer had the bug.

That shape is worth taking seriously, because it is not special to debuggers. Every method I use to find a bug rests on an assumption about the program, and the assumption is normally so obviously true that I have never once said it out loud. A debugger assumes I am permitted to pause. Logs assume the interesting value was written down. Local reproduction assumes the cause is in the code. Bisecting assumes a run tells me the truth.

The bugs that cost a week are, with dull reliability, the ones that have broken an assumption rather than the ones that are logically difficult. So here is the sentence the rest of this is about: a debugging method does not fail loudly, it fails by continuing to produce confident output about a question you have stopped asking. Knowing which assumption each method stands on is most of the skill, because that is what tells you when to put it down.

The debugger needs a process you are allowed to stop. On a laptop, stopping costs nothing. Anywhere that matters, a stopped process is a process that has stopped answering its health check, and whatever supervises it will conclude it is dead and replace it, taking your session and the state you were reading with it. So the method is available in precisely the place the interesting failures are not. A machine you can freeze is a machine under no load, and load is the missing ingredient in most of what I am chasing.

It also needs timing you are free to perturb. A breakpoint inserts an unbounded delay into one thread and none into the others. For sequential arithmetic that is free. For anything involving a lock, a queue, a timeout or a retry it is a rewrite of the thing under test, and the rewrite happens to be the one that makes the program correct. The set of bugs a debugger can hold still and the set that exist only under a particular interleaving barely overlap.

The reason is neither nostalgia nor laziness. A print statement produces a recording rather than an interrogation. It runs at full speed, on the real machine, under the real load, with nobody sitting there, and what it leaves behind is a sequence you can read afterwards and set beside the sequence from a run that worked. A breakpoint gives you one frozen instant and requires you to have guessed correctly which instant to freeze.

Running at full speed is not the same as free, and the mechanism that makes printing honest is also capable of lying. Writing to standard output takes a lock and frequently a system call, so a print in a hot path is itself a synchronization point, and I have watched a race vanish because I instrumented it. The difference from a breakpoint is one of degree, but the degree is enormous: a print narrows the window in which the bug can occur, and a breakpoint removes the window entirely.

The real cost is that it is a change to the program, so the loop runs as fast as your build and deploy pipeline and no faster. That is the honest argument against the technique, and notice that it is an argument about your pipeline rather than about the technique. Where I can get an instrumented binary in front of the failure in two minutes, printing beats everything else on this list. Where that takes an hour, I will try almost anything first, including things that work worse.

The value you need is the one nobody wrote down. This is the entire failure mode and there is nothing subtle about it. The logs are there, they are structured, they have retention, and they contain the request identifier and the status and the duration and not the single field that would settle the question. Adding it is a code change, so the earliest you can have the answer is the next deploy, and there is no guarantee the failure will still be happening by then.

What helps is verbosity you can raise without a deploy. A process that can be told at runtime to turn debug output on for one component gives back most of what printing offers and skips the build entirely. It is not free. It is a control surface, so it has to be authenticated, and something has to turn it off again, because debug logging left on under load is how an incident acquires a second cause. I have paid that price on every system that had it and never regretted it.

Reproducing it locally is the ideal, because a bug you can summon on demand is a bug you have effectively already fixed. The assumption underneath is that the cause is in the code, so running the same code the same way must produce the same thing. When it stubbornly does not, the reflex is to go hunting for a difference in the environment: a library version, a compiler flag, a clock, an operating system.

Often enough the environment is not it. The cause is a row. There is one account whose name contains a character the normalization step was never told about, one record written by a schema that stopped existing two years ago, one identifier that is numeric everywhere except in the handful of cases where somebody pasted it with a leading zero. Your local database contains none of these, because you generated it, and everything you generate is well formed by construction.

What works is to stop chasing the environment and start acquiring the input. Get the identifier of the record that failed, pull that record alone through whatever redaction you are obliged to apply, and then reduce it until you hold the smallest thing that still fails. That artifact outlasts any local replica, because it is a test case, and the replica was only ever a hope that the difference was somewhere else.

Bisecting assumes a run tells the truth. Against a deterministic bug, git bisect is close to magic: a thousand commits, ten builds, one answer, no thinking required. Every step of it depends on a passing run meaning the defect is absent. Against something that appears one run in twenty, a pass means very little, and a single wrong verdict does not merely slow the search down. It redirects the whole remaining bisection into a stretch of history that never contained the bug.

The arithmetic is worse than it feels. Take a failure that shows up one run in twenty. Ten consecutive passes are then still about sixty percent likely on a commit that is guilty, which is to say ten green runs are barely evidence at all. Getting a verdict to a confidence you would actually bet on takes dozens of runs, at each of ten steps, which turns an afternoon into a week and is why the honest move is usually to admit the method does not apply here.

Record and replay answers this directly, and charges for it. Mozilla's rr records one execution and replays it deterministically as many times as you like, including backwards, which converts an intermittent bug into a fixed specimen the moment you catch it once. Its chaos mode exists to help you catch it once. The catch is the recording: it is a Linux tool, it is particular about processor architecture, it costs real throughput, and you must have been recording before the failure.

The techniques that survive contact with production are the ones already running before anything goes wrong. That is not a coincidence and it is not a product category; it falls straight out of the constraint. If a method needs to stop the process, or rebuild it, or attach to it while a human watches, it is unavailable at the exact moment you want it. What is left is the set of things that were already recording, switched on by somebody who did not yet know what they would be needed for.

The mechanism that has made most of this practical is the ability to attach small verified programs to points inside a running kernel, which lets you ask a live machine what it is doing without patching it, restarting it, or relying on the program to be honest about itself. In August the Linux Foundation took that technology under a foundation of its own, with the companies whose fleets already depend on it as founding members. Governance arrives after the fact, so that is a fair marker of when it stopped being an experiment.

The same premise keeps showing up as always-on collection rather than on-demand investigation. Polar Signals published Parca in October, an open source profiler built on that kernel machinery and intended to run continuously against everything rather than be pointed at a suspect. I have no verdict on the tool. What interests me is the premise, which is that the recording has to predate the question, because by the time you have the question the moment has already been freed.

And none of it can be installed during an incident, which is the only practical instruction in this piece. A tracing tool you have never run, on a kernel you have not checked for the support it needs, driven by a syntax you are reading out of Brendan Gregg's book while the pager is going off, is not a debugging technique. It is a second outage layered on the first. The afternoon to find out whether these things work in your environment is a Tuesday when nothing is broken and nobody is watching.

I should narrow this before it curdles into an argument against tools. Most bugs are nothing like the ones described here. The overwhelming majority are a wrong comparison, an off-by-one, or a null where something was assumed, and for those a debugger is the fastest instrument in existence and I reach for it without a thought. The methods in this piece fail on a minority. The minority matters only because it is where the time goes, and nobody has ever lost a week to a null check.

What I have actually taken from it is smaller than a technique. It is to notice the moment a tool stops disagreeing with me, because that is the tell. A debugger showing the right value at every breakpoint, a log containing everything except the field, a local run that passes cleanly, a bisect that lands on a commit touching nothing relevant: those are one event in four costumes, and the event is that I am now studying a program other than the one that broke. The bug went away when I looked at it. That was the clue, not the setback.