Almost every argument about running models locally is an argument about cost. For regulated data it is not about cost at all. It is about whether you have converted a promise into a constraint, and that is a different question with a different answer.
3 August 2026·14 min read
Every organisation holding sensitive data is having the same conversation right now. The technology is obviously useful. The obvious way to use it is a hosted API. And somewhere in the second meeting, someone from legal or compliance asks where the data goes, and the room discovers it does not have an answer it can defend.
What usually happens next is one of two bad outcomes. The organisation freezes, decides AI is a problem for next year, and watches its people quietly paste things into consumer chat tools anyway. Or it moves ahead on the strength of a vendor's assurances that nobody in the room is qualified to evaluate.
I have been on both sides of this. I built and ran a healthcare platform where protected health information was the whole product, and I now run a rack of GPUs that serves models on my own network. This is the argument I would make to a regulated organisation, including the parts that make it a worse pitch.
Before any of the argument, a definition, because the word is doing too much work in most of these conversations.
When I say local, I mean inference running on infrastructure the organisation controls and administers. A server in your datacentre. A rack in a colocation cage. A private cloud tenancy where you hold the keys and the network boundary is yours. The defining property is not physical proximity. It is that your organisation is the accountable operator, and can say so to an auditor without pointing at a third party.
What I emphatically do not mean is an engineer running a model on their laptop. That is not local inference, it is shadow IT with a GPU in it, and it is worse than the hosted API it was meant to replace. The data is now on an endpoint nobody is monitoring, outside backup, outside access control, outside logging, and it leaves the building every evening in a bag.
That distinction matters because the two get conflated constantly, usually by people advocating for one and objecting to the other. An organisation that responds to this argument by letting individuals run their own models has not moved the boundary inward. It has dissolved it.
Somebody's name is on it. Patching, capacity, access, monitoring and backup are owned functions with a budget, the same as any other production system. If nobody owns it, it is not infrastructure, it is a science project holding regulated data.
Moving inference inside the boundary does not remove the need for controls, it relocates them. Who can query which model against which dataset is still an authorisation question, and it is now yours to answer rather than a vendor's.
Almost nobody needs a fully isolated network, and pretending you do is how projects die. The realistic target is that regulated data and the inference touching it stay inside a boundary you administer, while the rest of the environment works normally.
Worth stating plainly since it is elsewhere on this site. A rack in my house is where I learned what these models can and cannot do. It is not a template for how an organisation should run this, and I would fail my own audit on physical access alone.
Start with the thing that gets waved through: the idea that the contract solves it.
A business associate agreement is a real instrument. So is a data processing agreement, a no-training-on-your-data clause, a SOC 2 report, a documented subprocessor list. I am not dismissing any of it. If you are using a hosted service with regulated data, every one of those is necessary and you should be reading them closely rather than filing them.
But be precise about what they are. They are promises about behaviour, enforced after the fact, with a commercial remedy. They are not technical constraints on where bytes can travel. The distinction sounds academic until you are the one answering questions, at which point it is the entire conversation.
Your provider has subprocessors. Those subprocessors have their own arrangements. The contract chains, and the diligence you did on the vendor is not diligence you did on their infrastructure partner. Each link is probably fine. You are trusting a chain whose length you do not control.
Contractual protection is compensation, not prevention. If regulated data is exposed, the notification duty is yours, the reputational cost is yours, and the affected people are yours. A clause does not un-disclose anything.
Terms get revised. Companies get acquired. Free tiers get withdrawn and pricing models get rewritten, sometimes at short notice. I have written a whole other post about learning that lesson from a hypervisor vendor, and the stakes there were my weekends rather than somebody's medical records.
The honest problem. You are relying on an attestation about controls you cannot inspect, in an architecture you cannot see, operated by people you will never meet. That is a reasonable trade for most workloads. It is a specific and different trade for the ones where being wrong is a reportable event.
What "the data cannot leave" means once you are the person enforcing it.
In theory data classification is a tidy exercise: public, internal, confidential, regulated. In practice the hard part is not the taxonomy. It is that the boundary has to be enforced by people making small decisions quickly, most of whom are not thinking about compliance at the moment they make them.
A developer wants to debug a production issue and needs a realistic record to reproduce it. Someone wants to run an analysis and the fastest tool is the one that uploads a file. Support needs to understand what a customer is seeing. None of these people are careless. They are trying to do their jobs, and the boundary is in the way of the shortest path.
The conversation I had most often, as the person accountable for it, was some version of "can I just use the real data for this". Always from somebody with a legitimate reason and a deadline, usually correct that it would be faster. Saying no once is easy. Saying no consistently, for years, while also being the person who set the roadmap those people were trying to hit, is where a boundary actually lives or quietly stops existing.
The thing I learned running that platform is that a boundary only holds if the compliant path is also the convenient one. Every control that makes the right thing harder than the wrong thing is a control that will be routed around, politely, by good people, within about a month. That is not a moral failing in your organisation. It is a design failure in the control.
That is the actual argument for local inference, and it has nothing to do with hourly rates. If the model runs inside the boundary, using it is not a decision anybody has to make carefully. The convenient path and the compliant path become the same path, and you stop relying on continuous good judgment from busy people.
The reason this is worth discussing now rather than two years ago is capability, not hardware. So here is an honest account of what local models actually do, from running them daily rather than from benchmarks.
I keep a running log of every model I have deployed since the rig came online, with expected and measured throughput on the same four-GPU configuration. The short version is that the gap to frontier closed faster than I expected, and it closed unevenly.
Pulling fields out of documents, normalising messy text into a schema, classifying and routing. This is the highest-value regulated-data use case in most organisations and it is comfortably within reach of open models on modest hardware. If your problem is "we have twenty years of documents and no structure", you do not need frontier capability.
Local models that plan and ship code across a dozen tool calls, on hardware you can buy. Mid-2025 is when that stopped being a demo for me. For an organisation that cannot send its source code to a hosted model, this is the difference between having the tooling and not.
Good enough to save real time, not good enough to be unsupervised, and that is the same standard you would apply to a hosted model anyway. The compliance win is that the sensitive draft never leaves.
The hardest analytical work still favours the largest hosted models and probably will for a while. Anyone telling you otherwise is selling something. The useful question is not whether local matches frontier, it is what fraction of your actual workload needs frontier, and in most regulated organisations the honest answer is a small one.
And the hardware, described accurately rather than aspirationally, because the credibility of everything above depends on not overselling this part.
My build came to roughly $6,900 for four consumer GPUs and a server platform. That is not nothing, and it is comfortably inside the discretionary spend of a mid-sized team. The barrier to running serious inference on your own hardware is no longer money.
Four 24 GB cards is not one 96 GB card. It is four islands. A model has to fit its shard or pay an interconnect tax, and on consumer hardware that interconnect is slow. This determines which models you can run far more than raw compute does.
Without fast card-to-card links, multi-GPU training scales poorly. Inference barely notices, because pipeline-parallel only passes small activations between stages. Buy this class of hardware to run models. If you need to train something substantial, rent it, and rent it from somewhere your data classification permits.
Power, heat, and the electrical work to support both. I have written about discovering this the expensive way. In a datacentre this is somebody's job. In an office closet it is a fire code question, and it deserves an answer before the hardware arrives.
Which brings up the obligation nobody includes in the pitch, and the one I would put in front of any organisation before they buy hardware.
In regulated software the rule I would not negotiate is that nothing ships without a suite that exercises the happy path and, more importantly, the edges. That is uncontroversial for deterministic code. Given this input, assert that output. It is the cheapest insurance in the industry and I have watched it save a clinical calculation from a misplaced parenthesis, which is a sentence I get to write only because the suite existed.
Model pipelines break that assumption, and most teams discover it after they have shipped one. The same input does not reliably produce the same output. There is no assertion to write. The failure mode is not an exception or a null, it is a plausible answer that is wrong, delivered with exactly the same confidence as a correct one. Traditional testing is built to catch absence, and this is a system whose characteristic failure is confident presence.
So the thing you build is not a test suite in the familiar sense. It is a validation harness, and it costs real engineering time that belongs in the plan from the start rather than in a later sprint that never arrives.
Real inputs, drawn from your actual data distribution rather than invented examples, with outputs a qualified human has verified. This is tedious to build and it is the entire foundation. Without it you have opinions about whether the system works. In a regulated setting, opinions are not the standard.
Malformed input is the easy half. The ambiguous record, the one with a field that means two things, the document that is legible to a person and not to a parser, the case where the correct answer is "I cannot determine this". That last category matters most, because a model will confidently answer a question it should have declined, and a suite that only tests answerable cases will never catch it.
Assertions become ranges, distributions and acceptance thresholds rather than exact matches. Deciding what "close enough" means is a domain question, not an engineering one, and it needs the person accountable for the clinical or financial meaning of the output in the room when it is set.
The one that surprises people. Swapping in a newer open model is not an upgrade, it is a change to a system that produces regulated output. My own log shows models replaced as better weights land, which in a lab is the whole point and in a regulated environment is a governance event. Better benchmarks are not evidence about your corpus. Re-run the harness or do not ship it.
Which model, which version, which prompt, which parameters, against which input, producing which output, on what date. If a decision is questioned two years from now, the answer has to be reconstructable. Running the model yourself is what makes this recordable at all, and it is also what makes it your obligation to record.
I want to be honest that this is the part of the argument that costs money. Everything above about boundaries and control is a reason to bring inference in-house. This is the bill that arrives afterwards, and an organisation that takes the control benefit without funding the validation work has not improved its position. It has moved the risk somewhere it can no longer point at a vendor.
There is one benefit that never appears on a pricing page and that I think is underrated by everyone who has not sat through an audit.
"Where did this data go" is a question you will eventually be asked, in writing, by someone entitled to a complete answer. When inference happens on hardware you own, inside a network you control, the answer is a sentence. When it happens through a hosted API, the honest answer is a diagram, a set of contracts, a subprocessor list, and an attestation about controls you have not personally verified.
Both answers can be acceptable. Only one of them is short. In an environment where you are periodically required to demonstrate rather than assert, the short answer has a value that compounds every time you are asked for it.
The same logic applies to the breach arithmetic. IBM's 2026 study puts the global average cost of a breach at $4.99 million and the US average at $11.5 million, with healthcare the costliest sector for the thirteenth consecutive year at $6.64 million. Multiply the per-record figure by your dataset before deciding that where inference happens is a procurement question.
So the practical version, for someone who has to actually decide something on Monday.
Answer "which data, which tier" per system, in writing, before anyone evaluates a tool. The answer will be different for different parts of your estate, and a single organisation-wide policy will be wrong for most of it. This costs a week and saves a rebuild.
Not a platform. One workflow, on one dataset, with a measurable outcome. Document extraction is usually the right first choice: high volume, clear success criteria, and squarely within what local models do well.
Public and synthetic data has no reason to be local. Regulated data may have no option to be hosted. Most organisations need both, and the useful posture is a boundary you can articulate rather than a side you have picked.
The mistake I would most expect an organisation to make here. Bringing inference in-house is not permission to let everyone query everything. You still need authorisation on who can run what against which data, you still need the queries logged, and you now need those things because you are the operator rather than because a vendor promised them. Done properly this is better than what a hosted arrangement gives you, since the logs are yours. Done carelessly it is a single system with access to everything and no record of who asked it what.
The counterweight, and I would say this first in any room where people are getting enthusiastic. Running your own inference means owning patching, physical security, capacity, availability and the on-call that comes with it. And the failure mode to watch for is not the cloud, it is the well-meaning engineer who solves this for themselves with a workstation under a desk. That is a compliance finding wearing the costume of an initiative. If you cannot resource it to the standard you would demand of a vendor, the vendor is the better answer and the contract is the control you have.
The version of this argument I distrust is the one that says local is safer. It is not inherently safer. It is differently exposed, and the exposure moves from a party you cannot inspect to one you can, which is only an improvement if you actually inspect it.
What I would defend is narrower. For a specific and growing class of workloads, open models are now good enough that the constraint on using AI with sensitive data is no longer capability. It is architecture and will. That was not true two years ago, and I think a lot of organisations are still making the decision they made when it was.
I did not arrive at this from a strategy deck. I built the platform where the boundary mattered, and then years later I built the rack that could sit inside one. The argument is the same in both directions: put the capability where the data already is, and you stop asking people to be careful.