Two months ago a hypervisor vendor pushed me off a platform I had run for years. Three weeks ago the company behind my file server quietly confirmed that its FreeBSD line has no future. Same quarter, same lesson, and both roads led to Debian. This is the storage half of that story.
5 April 2024·15 min read
The thing that prompted writing this down.
In December, iXsystems' senior VP of engineering said on Reddit that TrueNAS CORE would get a 13.1 release and that it would be "a maintenance-only type update". In March, The Register put the question directly to the company, and their marketing VP confirmed there are "no plans for a FreeBSD 14-based TrueNAS at this time", with the 13.1 release positioned as a longer-lived maintenance train "for those who want to continue running on the BSD product before migrating to SCALE later".
Read that last clause again. Not "if you choose to migrate". Later. CORE is in what the company calls a sustaining engineering phase. That is the industry's politest available phrase for a product that will be kept alive but not moved forward.
There was no announcement, no blog post, no migration timeline. The news arrived through a Reddit comment and a journalist asking a direct question, which tells you something about how confident the company was that its users would enjoy hearing it. There are mutterings about a fork, as there always are.
I do not think this is a scandal. Maintaining two operating system bases for one product is expensive, Linux is where the container and virtualization ecosystem lives, and the commercial logic is obvious. But it is the second time in two months that a platform I depend on has told me, indirectly, where it is going without me.
Some context on how long this has been running, because it changes how the decision feels.
I started on FreeNAS. Not TrueNAS CORE, not SCALE. FreeNAS, on FreeBSD, back when the answer to "how do I store things properly at home" was a forum post and a lot of reading about ZFS.
That system has been renamed twice underneath me. FreeNAS became TrueNAS CORE with the 12.0 release in late 2020, when iXsystems unified the naming. Then SCALE arrived as a Debian-based sibling, first stable in early 2022, and for a while the honest answer to "which one should I run" was "whichever one you already have".
Through all of that, the actual thing I care about never changed. ZFS is ZFS. The pool I created years ago has survived two rebrands, an operating system family transition, several hardware moves, and every drive in it being replaced at least once. That is the whole argument for the filesystem in one sentence, and it is why I have never seriously considered leaving.
So this decision was never about whether to keep using TrueNAS. It was about which of its two faces to run, at a moment when the vendor had just answered that question on my behalf.
CORE against SCALE, as it actually stood in early 2024.
This is the part CORE loyalists are right about and it deserves saying plainly. ZFS on FreeBSD had a maturity and a coherence that came from being a first-class citizen rather than a port. The network stack was excellent. For a machine whose only job was serving files, CORE was arguably the more focused product, and people who preferred it were not being sentimental.
Containers, Kubernetes, the entire virtualization ecosystem, hardware support for anything recent, and the enormous pool of documentation that comes with being where everybody else is. A storage appliance that cannot easily run the workloads people want next to their storage is a narrower product, and iXsystems clearly concluded the same thing.
The historical argument for FreeBSD was that ZFS was better there. OpenZFS unified the implementations, and the practical difference for a home or small business pool narrowed to the point where it stopped deciding anything. Once the filesystem was equally good on both, the operating system question became a question about everything except storage.
You can migrate CORE to SCALE in place. There is no supported route back. That asymmetry is the tell: the vendor built a door in one direction, which tells you which side of it they expect you to end up on.
Two months earlier I had replaced ESXi with Proxmox, which is Debian underneath. Choosing SCALE meant one less operating system family in the house, one set of habits, one set of tools. That is not a technical argument so much as an operational one, and at the scale of a lab run by one person, operational simplicity is worth more than most technical arguments.
So SCALE, without much agonising. What is strange is that I did not choose Linux for storage on the merits. I chose Proxmox for the hypervisor on the merits, and the file server followed, and a vendor decision made in someone else's boardroom finished the job. Convergence by accident.
The physical machine first, and it is deliberately unglamorous. This is second-hand enterprise hardware bought for what it does rather than what it is, and that is the correct way to buy a file server. It is also a general-purpose Proxmox host rather than a dedicated appliance: TrueNAS is the most important thing on it, not the only thing.
The board choice is the quietly load-bearing one. Ten-gigabit copper is onboard, which means the PCIe slots stay free for the controller rather than being spent on a network card, and there is no transceiver anywhere in the path. Two RJ-45 ports straight off the board, into a switch, done. It is the least interesting way to get 10 gigabit and I have never once had to think about it since.
Two other things about this build are worth defending. The first is that it is old, and that is the point: a file server needs cores, memory channels, ECC and PCIe lanes, and none of those are things that improved dramatically in the years that made this hardware cheap. A previous-generation dual-socket Xeon platform gives you all four for a fraction of what current silicon costs, and the workload does not care.
The second is the memory, and the answer is less about ZFS than I expected when I started writing this down. TrueNAS is one guest among many here. The same host runs DNS, home automation, a reverse proxy, a media server, a local git server, and a rotating population of database instances and test machines that exist for a week and then do not. Half a terabyte stops looking indulgent once you count what is actually resident.
And almost all of it is allocated up front, which is not a preference. Most of these guests have some piece of hardware passed through to them, and passthrough requires the guest's memory to be statically backed. You cannot hand a VM a real PCI device and also let the hypervisor move its memory around underneath it. So ballooning is off across the board, every guest takes its full allocation at boot, and the host has to be sized for the sum of what everything might want rather than the average of what it is using.
That constraint turns out to suit ZFS perfectly, for reasons that are entirely coincidental. ARC is happiest with a fixed allocation it never has to surrender, and a host where nothing balloons is a host where the cache is never quietly taken away.
And what TrueNAS actually gets, which is a small slice of that host.
The allocation looks lopsided until you consider what each resource is for. Eight cores is plenty: file serving is an I/O problem, and the checksumming people assume is expensive is not, on hardware of this vintage or any other.
The boot disk is virtual and small on purpose. TrueNAS treats its operating system as disposable, the configuration is a file you back up, and none of the data lives there. If that disk died tomorrow the recovery is a reinstall and a config import, and the pool would not notice.
Which brings up the single mechanism that explains most of the decisions on this page, and the one thing about TrueNAS that reliably alarms people the first time they see it.
Open this virtual machine in Proxmox and the memory gauge reads 100 percent. Not occasionally, not under load. Always, and it will still read 100 percent in a year. Every person who has looked over my shoulder at that number has asked whether something is wrong.
Nothing is wrong. That is the Adaptive Replacement Cache, and filling memory is its entire job. ZFS treats free RAM as read cache, holds as much as it is allowed to hold, and hands it back the moment something else needs it. Memory sitting idle is memory doing nothing, and ZFS is unusual mainly in refusing to pretend otherwise. A TrueNAS box sitting at 40 percent memory is the one worth investigating, because it means the cache is not being allowed to do its work.
Once you accept that, a lot of this build stops looking like over-provisioning and starts looking like arithmetic.
512 GB in a machine that serves files reads as excessive until you understand that memory is the performance budget, not headroom. Everything the pool serves frequently ends up in RAM, and the difference between a read served from ARC and a read served from a spinning disk is not a percentage.
Proxmox will happily reclaim memory from a guest that appears not to be using it. ARC always appears to be using all of it, which makes ZFS and dynamic memory a bad pairing. Give the VM a fixed number and leave it alone, or watch a fast pool become a slow one for reasons the monitoring will not explain.
ARC wants the memory. A guest operating system wants the same memory. On one box those appetites compete, and the storage side has to win, because starving the cache to feed a virtual machine makes the file server slow for everyone in the building. Whatever I build next that needs serious RAM gets its own hardware.
ARC only helps with what it has seen. The first read still has to come from somewhere, and for latency-sensitive work that first read is the one you feel. A small fast pool covers the case the cache cannot, which is why capacity and speed are two pools here rather than one compromise.
And the layout, the decision you cannot walk back later. There are two pools, and neither of them uses RAIDZ.
Eight drives in four mirrored pairs, striped. That gives up half the raw capacity. It is the objection everybody raises and the reason RAIDZ is more popular. What it buys is the behaviour on the worst day. A mirror resilvers by copying from one surviving drive, which is fast and reads one disk. A RAIDZ vdev resilvers by reading every remaining drive in the vdev and reconstructing, which takes far longer and applies sustained load to a set of disks that are the same age, from the same batch, with the same hours on them, at the exact moment you least want a second failure.
The mirrors-versus-RAIDZ argument changes as drives get bigger, and most of the advice online is calibrated to drive sizes nobody buys any more. At modern capacities a RAIDZ resilver is measured in days rather than hours, and every hour of it is a window. If the drives were small this would be a closer call. They are not, so it is not.
Adding capacity to a striped mirror pool means adding two more drives. That is it. Growing RAIDZ historically meant replacing every drive in the vdev one at a time, or building a new vdev at the original width. For a lab that grows by accident rather than by plan, being able to expand in pairs matters more than the percentage on a spreadsheet.
BigPool is bulk: media, documents, archives, backup targets. Capacity per pound, and nothing on it is latency-sensitive. FastPool is the opposite, sized small and deliberately, for the things where a slow read is felt immediately. Putting both workloads in one pool means the bulk data sets the performance characteristics of everything, and one pool is a compromise between two jobs rather than the right answer to either.
Both pools have a mirrored log vdev, and a SLOG only does anything for synchronous writes. That is the tell for what this machine is actually for. Serving files to people is mostly asynchronous and would not justify it. Backing virtual machines over the network is not, and a hypervisor asking for a sync write and waiting on spinning rust is the difference between a responsive guest and a miserable one. It is mirrored because a log device failing at the wrong moment is exactly the scenario it exists to protect against.
FastPool sits close to empty. That is deliberate rather than aspirational. Pools slow down as they fill, allocation gets harder past roughly eighty percent, and a fast pool that is nearly full has stopped being a fast pool. Headroom is part of the specification, not spare capacity waiting to be used up.
All of this only works because the HBA is passed through to the virtual machine as a PCI device. ZFS is talking to actual drives, with real SMART data, real error counters and real flush semantics. Every topology decision above is meaningless if the layer underneath is lying about whether a write reached the platter, and that is precisely the failure mode people mean when they say do not virtualize your NAS. They are right about the danger and wrong to assume it is unavoidable.
Now the part that gets me argued with, so let me make the case honestly.
TrueNAS runs here as a virtual machine on a Proxmox host, with that host doing nothing else that competes with it for memory. The HBA is passed through to that VM, so TrueNAS talks to the drives directly rather than through a virtual disk layer.
A significant part of the TrueNAS community thinks this is a bad idea, and their objection is not unreasonable. ZFS wants to own the disks, know their real state, and be certain a write reached the platter. Every abstraction between ZFS and the hardware is a place for that certainty to become an assumption. The failure stories people cite are real, and almost all of them come from someone handing ZFS a virtual disk or a hardware RAID volume instead of the actual controller.
The rule that makes this defensible. The HBA goes through as a PCI device, in IT mode, so ZFS sees real drives with real SMART data and real flush semantics. Presenting individual virtual disks to a TrueNAS VM is the configuration that earns the community's warnings, and it is a different thing entirely from what I am describing.
The honest cost. IOMMU grouping, passthrough surviving a host upgrade, and a hypervisor between me and my data. An evening lost to a card that will not initialise is an annoyance on any other machine in the house. Here the stakes are the pool.
Storage stays on its own machine, with its own failure domain and its own maintenance window. Anything else I build can be rebuilt, broken or powered off for an afternoon without the house noticing, because the state that matters is not on it. That separation is the thing I would not give up, and it is worth the extra layer to get it while still running Proxmox underneath.
If you are asking whether to virtualize your NAS and you do not have a specific reason, the answer is no. Run it on the metal. I do this because both machines are already Proxmox hosts and the operational consistency is worth something to me, and I would not recommend it to someone whose first ZFS pool this is.
What actually depends on this machine, and why it gets treated more carefully than anything else here.
The lab is disposable by design. VMs get re-provisioned, hosts get rebuilt, and the AI rig in particular gets reshaped depending on what it is doing that week. None of that is frightening because the state that matters does not live there. It lives on the pool.
Media, documents, photos, the things that other people in the building notice within minutes of them being unavailable. This is the constituency that turns a hobby outage into a domestic incident, and it is most of the reason the file server is the machine I patch conservatively.
Snapshots and replication are where ZFS earns its reputation, and they are the reason a mistake on any other machine is an inconvenience rather than a loss. The rest of the lab can be careless precisely because this box is not.
No experiments run here. Nothing gets tried on the file server because it looks interesting. Every other machine in the house is a place to break things, and the value of that comes entirely from one machine where I do not.
The pattern I keep noticing across this year is that none of these decisions were really mine. Broadcom decided that free ESXi was over. iXsystems decided that FreeBSD was over. In both cases I got to choose how to respond, which is not the same as choosing.
What made both responses easy was that the thing I actually depend on was never the vendor. It was ZFS, and a pool that has outlived two product names and every drive that was originally in it. Build on the layer that persists, and the layers above it become somebody else's news cycle.