The File Server Is the Foundation

In December, iXsystems' senior VP of engineering said on Reddit that TrueNAS CORE would get a 13.1 release and that it would be "a maintenance-only type update". In March, The Register put the question directly to the company, and their marketing VP confirmed there are "no plans for a FreeBSD 14-based TrueNAS at this time", with the 13.1 release positioned as a longer-lived maintenance train "for those who want to continue running on the BSD product before migrating to SCALE later".

Read that last clause again. Not "if you choose to migrate". Later. CORE is in what the company calls a sustaining engineering phase. Sustaining engineering is the industry's politest available phrase for a product that will be kept alive but not moved forward.

No announcement, no blog post, no migration timeline. The news arrived through a Reddit comment and a journalist asking a direct question, which tells you something about how confident the company was that its users would enjoy hearing it. Mutterings about a fork are circulating, as they always are.

I do not think this is a scandal. Maintaining two operating system bases for one product is expensive, Linux is where the container and virtualization ecosystem lives, and the commercial logic is obvious. But it is the second time in two months that a platform I depend on has told me, indirectly, where it is going without me.

I started on FreeNAS, which came before either CORE or SCALE. FreeNAS, on FreeBSD, back when the answer to "how do I store things properly at home" was a forum post and a lot of reading about ZFS.

That system has been renamed twice underneath me. FreeNAS became TrueNAS CORE with the 12.0 release in late 2020, when iXsystems unified the naming. Then SCALE arrived as a Debian-based sibling, first stable in early 2022, and for a while the answer to "which one should I run" was "whichever one you already have".

Through all of that, the thing I care about never changed. ZFS is ZFS. The pool I created years ago has survived two rebrands, an operating system family transition, several hardware moves, and every drive in it being replaced at least once. Pool survival is the whole argument for the filesystem in one sentence, and it is why I have never seriously considered leaving.

So this decision was never about whether to keep using TrueNAS. It was about which of its two faces to run, at a moment when the vendor had just answered that question on my behalf.

FreeBSD was the better storage operating system, for a while. CORE loyalists are right about this, and it deserves saying plainly. ZFS on FreeBSD had a maturity and a coherence that came from being a first-class citizen rather than a port. The network stack was excellent. For a machine whose only job was serving files, CORE was arguably the more focused product, and people who preferred it were not being sentimental.

But everything else moved to Linux. Containers, Kubernetes, the entire virtualization ecosystem, hardware support for anything recent, and the enormous pool of documentation that comes with being where everybody else is. A storage appliance that cannot easily run the workloads people want next to their storage is a narrower product, and iXsystems clearly concluded the same thing.

OpenZFS closed the gap that mattered. The historical argument for FreeBSD was that ZFS was better there. OpenZFS unified the implementations, and the practical difference for a home or small business pool narrowed to the point where it stopped deciding anything. Once the filesystem was equally good on both, the operating system question became a question about everything except storage.

The migration path only runs one way. You can migrate CORE to SCALE in place. No supported route back exists. That asymmetry is the tell: the vendor built a door in one direction, which tells you which side of it they expect you to end up on.

And I had just moved the hypervisor to Debian. Two months earlier I had replaced ESXi with Proxmox, which is Debian underneath. Choosing SCALE meant one less operating system family in the house, one set of habits, one set of tools. The consolidation is not a technical argument so much as an operational one, and at the scale of a lab run by one person, operational simplicity is worth more than most technical arguments.

So SCALE, without much agonizing. What is strange is that I did not choose Linux for storage on the merits. I chose Proxmox for the hypervisor on the merits, and the file server followed, and a vendor decision made in someone else's boardroom finished the job. Convergence by accident.

The board choice is the quietly load-bearing one. Ten-gigabit copper is onboard, which means the PCIe slots go to the controller instead of a network card, and there is no transceiver anywhere in the path. Two RJ-45 ports straight off the board, into a switch, done. Onboard copper is the least interesting way to get 10 gigabit and I have never once had to think about it since.

Two other things about this build are worth defending. The first is that it is old, and that is the point: a file server needs cores, memory channels, ECC, and PCIe lanes, and none of those are things that improved dramatically in the years that made this hardware cheap. A previous-generation dual-socket Xeon platform gives you all four for a fraction of what current silicon costs, and the workload does not care.

The second is the memory, and the answer is less about ZFS than I expected when I started writing this down. TrueNAS is one guest among many here. The same host runs DNS, home automation, a reverse proxy, a media server, a local git server, and a rotating population of database instances and test machines that exist for a week and then do not. Half a terabyte stops looking indulgent once you count what is actually resident.

And almost all of it is allocated up front, which is not a preference. Most of these guests have some piece of hardware passed through to them, and passthrough requires the guest's memory to be statically backed. You cannot hand a VM a real PCI device and also let the hypervisor move its memory around underneath it. So ballooning is off across the board, every guest takes its full allocation at boot, and the host has to be sized for the sum of what everything might want, which is a good deal more than the average of what it is using.

That constraint turns out to suit ZFS perfectly, for reasons that are entirely coincidental. ARC is happiest with a fixed allocation it never has to surrender, and a host where nothing balloons is a host where the cache is never quietly taken away.

The allocation looks lopsided until you consider what each resource is for. Eight cores is plenty: file serving is an I/O problem, and the checksumming people assume is expensive is not, on hardware of this vintage or any other.

The boot disk is virtual and small on purpose. TrueNAS treats its operating system as disposable, the configuration is a file you back up, and none of the data lives there. If that disk died tomorrow the recovery is a reinstall and a config import, and the pool would not notice.

Open this virtual machine in Proxmox and the memory gauge reads 100 percent. Always, at idle as much as under load. It will still read 100 percent in a year. Every person who has looked over my shoulder at that number has asked whether something is wrong.

The system is healthy. The gauge is showing the Adaptive Replacement Cache, and filling memory is its entire job. ZFS treats free RAM as read cache, holds as much as it is allowed to hold, and hands it back the moment something else needs it. Memory sitting idle is memory doing nothing, and ZFS is unusual mainly in refusing to pretend otherwise. A TrueNAS box sitting at 40 percent memory is the one worth investigating, because it means the cache is not being allowed to do its work.

Once you accept that, a lot of this build stops looking like over-provisioning and starts looking like arithmetic.

It is why the host has half a terabyte. 512 GB in a machine that serves files reads as excessive until you understand that memory is the performance budget, not headroom. Everything the pool serves frequently ends up in RAM, and the difference between a read served from ARC and a read served from a spinning disk is orders of magnitude.

It is why the allocation is fixed and never ballooned. Proxmox will happily reclaim memory from a guest that appears not to be using it. ARC always appears to be using all of it. ZFS and dynamic memory are a bad pairing. Give the VM a fixed number and leave it alone, or watch a fast pool become a slow one for reasons the monitoring will not explain.

It is why anything memory-hungry lives somewhere else. ARC wants the memory. A guest operating system wants the same memory. On one box those appetites compete, and the storage side has to win, because starving the cache to feed a virtual machine makes the file server slow for everyone in the building. Whatever I build next that needs serious RAM gets its own hardware.

And it is why a second, faster pool exists at all. ARC only helps with what it has seen. The first read still has to come from somewhere, and for latency-sensitive work that first read is the one you feel. A small fast pool covers the case the cache cannot, which is why capacity and speed get two pools here instead of one compromise.

Mirrors are everywhere, and it is not a capacity decision. Eight drives in four mirrored pairs, striped. That gives up half the raw capacity. Capacity is the objection everybody raises and the reason RAIDZ is more popular. What it buys is the behavior on the worst day. A mirror resilvers by copying from one surviving drive, which is fast and reads one disk. A RAIDZ vdev resilvers by reading every remaining drive in the vdev and reconstructing, which takes far longer and applies sustained load to a set of disks that are the same age, from the same batch, with the same hours on them, at the exact moment you least want a second failure.

Drive size is what actually decides this. The mirrors-versus-RAIDZ argument changes as drives get bigger, and most of the advice online is calibrated to drive sizes nobody buys any more. At modern capacities a RAIDZ resilver is measured in days rather than hours, and every hour of it is a window. If the drives were small this would be a closer call. They are not, so it is not.

Expansion is the quiet argument. Adding capacity to a striped mirror pool means adding two more drives. That is all it takes. Growing RAIDZ historically meant replacing every drive in the vdev one at a time, or building a new vdev at the original width. For a lab that grows by accident, being able to expand in pairs matters more than the percentage on a spreadsheet.

Two pools, because the workloads are different. BigPool is bulk: media, documents, archives, backup targets. Capacity per pound, and nothing on it is latency-sensitive. FastPool is the opposite, sized small and deliberately, for the things where a slow read is felt immediately. Putting both workloads in one pool means the bulk data sets the performance characteristics of everything, and one pool compromises between two jobs and is the right answer to neither.

The SLOG is there because of what this serves. Both pools have a mirrored log vdev, and a SLOG only does anything for synchronous writes. The SLOG is the tell for what this machine is actually for. Serving files to people is mostly asynchronous and would not justify it. Backing virtual machines over the network is not, and a hypervisor asking for a sync write and waiting on spinning rust is the difference between a responsive guest and a miserable one. The log is mirrored because a log device failing at the wrong moment is exactly the scenario it exists to protect against.

The empty pool is not a mistake. FastPool sits close to empty. That is deliberate rather than aspirational. Pools slow down as they fill, allocation gets harder past roughly eighty percent, and a fast pool that is nearly full has stopped being a fast pool. Headroom is part of the specification, not spare capacity waiting to be used up.

And ZFS sees real disks, which is why the passthrough exists. All of this only works because the HBA is passed through to the virtual machine as a PCI device. ZFS is talking to actual drives, with real SMART data, real error counters, and real flush semantics. Every topology decision above is meaningless if the layer underneath is lying about whether a write reached the platter, and that is precisely the failure mode people mean when they say do not virtualize your NAS. They are right about the danger and wrong to assume it is unavoidable.

TrueNAS runs here as a virtual machine on a Proxmox host, with that host doing nothing else that competes with it for memory. The HBA is passed through to that VM, so TrueNAS talks to the drives directly, with no virtual disk layer in between.

A significant part of the TrueNAS community thinks this is a bad idea. Their objection is not unreasonable. ZFS wants to own the disks, know their real state, and be certain a write reached the platter. Every abstraction between ZFS and the hardware is a place for that certainty to become an assumption. The failure stories people cite are real, and almost all of them come from someone handing ZFS a virtual disk or a hardware RAID volume instead of the actual controller.

Pass the controller, never the disks. The rule that makes this defensible. The HBA goes through as a PCI device, in IT mode, so ZFS sees real drives with real SMART data and real flush semantics. Presenting individual virtual disks to a TrueNAS VM is the configuration that earns the community's warnings, and it is a different thing entirely from what I am describing.

It is one more layer that can be wrong. IOMMU grouping, passthrough surviving a host upgrade, and a hypervisor between me and my data. An evening lost to a card that will not initialize is an annoyance on any other machine in the house. Here the stakes are the pool.

What it buys is a boundary. Storage stays on its own machine. Its own failure domain, its own maintenance window. Anything else I build can be rebuilt, broken, or powered off for an afternoon without the house noticing, because the state that matters is not on it. That separation is the thing I would not give up, and it is worth the extra layer to get it while still running Proxmox underneath.

Bare metal is still the right default. If you are asking whether to virtualize your NAS and you do not have a specific reason, the answer is no. Run it on the metal. I do this because both machines are already Proxmox hosts and the operational consistency is worth something to me, and I would not recommend it to someone whose first ZFS pool this is.

It holds everything that must survive a rebuild. The lab is disposable by design. VMs get re-provisioned, hosts get rebuilt, and a machine gets reshaped depending on what it is doing that week. None of that is frightening because the state that matters does not live there. It lives on the pool.

It serves the household half of the network. Media, documents, photos, the things that other people in the building notice within minutes of them being unavailable. The household is the constituency that turns a hobby outage into a domestic incident, and it is most of the reason the file server is the machine I patch conservatively.

Backup targets and snapshots land here too. Snapshots and replication are where ZFS earns its reputation, and they are the reason a mistake on any other machine is an inconvenience rather than a loss. The rest of the lab can be careless precisely because this box is not.

And it stays boring on purpose. No experiments run here. I do not try things on the file server because they look interesting. Every other machine in the house is a place to break things, and the value of that comes entirely from one machine where I do not.

The pattern I keep noticing across this year is that none of these decisions were really mine. Broadcom decided that free ESXi was over. iXsystems decided that FreeBSD was over. In both cases I got to choose how to respond, which is not the same as choosing.

What made both responses easy was that the thing I actually depend on was never the vendor. It was ZFS, and a pool that has outlived two product names and every drive that was originally in it.