Most AI tooling rollouts fail the same way: a licence gets bought, an announcement gets made, and six weeks later nobody can say whether anything improved. Here's how I'd do it instead, from someone who has run these agents daily on his own hardware and managed the engineers who'd have to live with the decision.
22 March 2026·7 min read
The failure mode is consistent enough to be predictable, so it's worth naming before the advice.
Somebody senior reads about a productivity number, licences get bought, and the tool lands in an org that was never asked what problem it was solving. Adoption becomes a compliance exercise, engineers use it enough to be seen using it, and nobody learns anything about whether it works.
Lines of code, acceptance rate, number of suggestions taken. All of these are measurable and none of them are the thing you care about. Acceptance rate in particular is actively misleading: a tool that produces plausible code you accept and then spend an afternoon debugging scores well on it.
The most consequential question and the one that usually goes unasked. If an agent wrote it, who understands it? Who is on the hook at 2am? Review standards written for human-authored code don't survive contact with a machine that can produce four hundred plausible lines in a minute.
Which repositories, which model, hosted where, and what leaves the building. This gets sorted out after the tool is in use, which means it gets sorted out under pressure with people already depending on it.
So the order matters. These get decided before anyone installs anything.
The first question, and it's binary. Some codebases can go to a hosted model and some cannot. Client code under NDA, regulated data, anything where the contract has an opinion. Answer this per repository, write it down, and accept that the answer might force a local or self-hosted model for part of the estate. That constraint is much cheaper to design around than to retrofit.
"Productivity" is not an answer. Cycle time on small changes, time to first PR for new joiners, the backlog of tests nobody writes, migration work that is tedious rather than hard. Pick one or two, because the tool is far better at some of these than others and a blanket rollout hides that.
Decide this before you need it. My position: the author is whoever submits the PR, regardless of what produced the diff, and "the agent wrote it" is never an explanation in review. That has to be stated in advance, because the natural drift is toward diffuse responsibility.
Name someone whose job is to call it. Rollouts acquire momentum and sunk cost quickly, and without an explicit owner of the negative case, the honest signal never reaches the person who bought the licences.
Then a pilot that could actually fail, because one that cannot fail teaches you nothing.
Six to eight people. Volunteers, because early adopters generate signal and conscripts generate compliance. Deliberately mixed seniority, because the effect on a principal engineer and on someone two years in is not the same effect, and the difference is the most important thing you will learn.
Pilots on toy problems produce results that don't survive contact with a real codebase. Give it the actual repository, with its actual history, conventions and thirteen-year-old module nobody wants to touch. That module is where you find out what the tool is worth.
The first two weeks are enthusiasm and the next two are disillusionment. Neither is data. Run it long enough that people have settled into a working pattern, which in my experience is somewhere north of a month.
Before starting, have the group write what they expect to improve and by how much. It costs an hour and it is the only defence against reading whatever happened as confirmation of whatever you hoped.
Measuring it honestly is where most of the intellectual work is, and where most orgs give up and quote a vendor statistic instead.
The metrics I'd actually watch are lagging and boring: cycle time from first commit to merged, defect escape rate, and review load measured in reviewer hours rather than PR count. If a tool makes people faster at producing changes and slower at reviewing them, that is not a win, and only the third metric catches it.
I'd also watch rework directly. Time spent modifying code that was merged in the last thirty days is the closest thing to a lie detector for generated code, because the failure mode isn't code that doesn't work, it's code that works and is wrong in a way that surfaces later.
And I'd take qualitative signal seriously rather than treating it as soft. "Do you understand the code you merged this week" asked in a one-to-one gets you an answer that no dashboard will. If the honest answer starts trending toward no, that is the single most important number in the programme and it isn't a number.
The part that changes most, and the part I'd spend the most management attention on.
Code review was designed around a bottleneck that no longer exists. It assumed writing was slow and reading was fast, so the reviewer's job was to check work that took real effort to produce. Generation collapses the writing side and leaves reading exactly as slow as it was. The bottleneck moves to review, and if nothing else changes, review becomes the place the team quietly stops doing its job properly.
Practically, I'd expect three adjustments. Smaller PRs enforced harder, because a four-hundred-line generated diff gets rubber-stamped in a way a four-hundred-line handwritten one does not. Explicit author accountability, so the person submitting has to be able to defend every line under questioning. And more attention on the tests, since generated code that passes generated tests is a closed loop that proves nothing.
The failure I'd watch for is subtle: review that becomes plausibility checking rather than correctness checking. The code looks like what code for this task looks like, so it gets approved. That's a pattern-matching exercise, and it is precisely the thing the tool is better at than the reviewer.
Then the question I think is hardest, and where I distrust anyone who sounds certain.
Junior engineers get more benefit from these tools than anyone, and they may pay the highest long-term price. The benefit is obvious: unblocked faster, less time lost to syntax and scaffolding, a patient explainer available at 11pm. The cost is that a lot of engineering judgment is built by being stuck, and a tool that removes being stuck may also remove the mechanism.
I don't think the answer is withholding the tools, which is both patronising and unenforceable. I think it's being deliberate about which struggles are load-bearing. Debugging something hard, reading unfamiliar code until it makes sense, and sitting with an ambiguous requirement until it resolves are all things where the struggle is the learning, and they're worth protecting even when a tool could shortcut them.
As a manager this means my one-to-ones change. Less "did you ship it" and more "walk me through why it works", which is a question I should have been asking more anyway and which the tooling makes non-optional.
And the things I would not do, most of which I have watched go badly.
Nothing produces theatre faster than a dashboard of who used the tool. You will get usage and you will learn nothing, and you will have taught your engineers that the point is to be seen complying.
The vendor's figure is not your figure. Repeating it publicly sets an expectation you will then have to defend against your own data, and it poisons the pilot before it starts.
The most damaging possible move, and it converts a tooling decision into a threat. People do not experiment honestly with a tool they believe is being evaluated as their replacement, and you lose the signal permanently.
If part of the org is getting real leverage and the rest is not, that is a knowledge distribution problem, not a licensing one. Whatever the effective group learned about prompting, context and where the tool fails is the actual asset.
The through-line is that none of the hard questions here are about the tools. They're about ownership, review, how people learn, and how honest you're willing to be about measurement. Those were all questions before, and the tooling just makes ducking them more expensive.
I'd also say the thing that made me confident writing any of this: I run these agents daily, on my own hardware, against my own code, including the parts where they fall over. That's a different kind of knowledge from reading the case studies, and it is mostly what convinced me that the interesting decisions are organisational rather than technical.