Splitting one model across all four of my RTX 3090s only became worth doing recently, once local models and the engines that serve them caught up with the hardware. The cards have no NVLink bridges, so every exchange between them went through system memory. A firmware flash, a BIOS setting, a patched driver, and one environment variable opened a direct path over ordinary PCIe, and prompt processing went from about 1,840 tokens a second to about 3,950. Most of the tests along the way moved nothing.
15 October 2026·12 min read·homelabai
The box is the one from the original lab build: four EVGA RTX 3090s on a 32-core EPYC board, 96 GB of VRAM between them, each card in its own PCIe 4.0 x16 slot. It serves one model, Qwen3.8-Flash-Next, a mixture-of-experts model with recurrent layers mixed in among its attention layers, split across all four cards through vLLM. Its clients are a pair of coding agents and a chat window.
Agents are hard on a local server. Every turn resends the whole conversation, and by the middle of a session that is fifteen or twenty thousand tokens of prompt to read before the first word of a reply. A chat user asks a question and waits once. An agent waits on every tool call, and a session can make dozens of them.
The first fix was in a client. The chat window sent no reasoning setting, and the model thought at its maximum on every message. The same test question took 22.6 seconds there and about 4 at medium.
A prefix cache lets a repeated prompt skip work already done. For an agent, each turn is the previous turn plus a little more, and on this model the cache was keeping almost none of it. An 18,000-token prompt took 10.25 seconds to start answering. Sending the identical prompt again took 10.26, with zero tokens reused.
The cause is in how vLLM caches a hybrid model. Attention layers keep an entry per token. The recurrent layers keep a running state instead, and vLLM can reuse that state only at block boundaries, here every 1,616 tokens. This build defaults the prefix cache retention interval to zero, which keeps a recurrent checkpoint only where two different requests have already shared a prefix. A new prompt saves nothing the first time. The second request marks the junction, and the third gets the reuse. Sending one prefix three times in a row showed it: 10.20 seconds, 10.25, then 1.45, with 16,160 tokens reused on the third.
Setting the interval to one block, 1,616 tokens, keeps a checkpoint at every boundary. The identical resend came back in 1.45 seconds on the second try. A 125,000-token prompt took 70.9 seconds the first time and 1.9 seconds when resent, with 122,816 tokens reused and nothing evicted.
That fixed repeat visits. A prompt the server had never seen still read at about 1,840 tokens a second, and a fresh 16,000-token turn sat for eight and a half seconds before anything came back.
Bigger prompt chunks changed nothing and cost cache. The server reads a prompt 1,024 tokens at a time. At 2,048 the 16,000-token prompt read at 1,739 tokens a second and the 64,000-token one at 1,819, against 1,838 and 1,836 at 1,024. The larger working buffers shrank the cache from 155,571 tokens to 142,096. Two 98,000-token prompts sent together no longer both fit. The second waited behind the first and finished in 112 seconds.
Drafting three tokens ahead instead of four made generation slower. The model carries its own small draft head, which guesses several tokens ahead for the main model to check in one pass. At three, more guesses were accepted, 69% on code against 59%. Each step still produced fewer tokens, and code and agent-style output both came out slower.
Generating, the cards drew 262 to 276 W against the 280 W cap and held 1,920 to 1,965 MHz of a possible 2,100. The reference build of this model runs at 220 W and generates prose at 102 to 109 tokens a second, the same as mine at 280. Its code reply was faster, 186 to 188 tokens a second on a 3,000-token prompt against my 173 to 175 on the same one. Its prompt processing was half again faster. Its cards have two NVLink bridges.
Watching the cards read a long prompt pointed somewhere else. All four reported full utilization and drew only about 231 W. The PCIe links carried 3 to 4 GB/s each, and the CPU threads driving the cards sat pinned at 100%. Busy cards draw more than that. Cards waiting on each other look exactly like it.
Splitting one model across four cards means splitting every layer. Each card computes its slice, and all four combine their results before the next layer can start, for every token and far more often across a long prompt. That exchange sets a floor under everything else.
NVIDIA's answer is NVLink, a bridge that clips across the top of two cards and gives them a private link faster than the slot. I went looking for 3090 bridges and came up empty. A pair would have been a partial answer anyway. A 3090 bridge joins exactly two cards and comes in fixed widths that have to match the gap between them, and four cards would still cross PCIe between the pairs. The reference build does exactly that.
Without a bridge the cards can still reach each other over the PCIe bus, through the processor's I/O die. NVIDIA's standard Linux driver refuses. On a GeForce card it reports peer-to-peer access as unsupported, and every exchange goes the long way: out of one card into system memory, then into the next. nvidia-smi topo -p2p r printed CNS, chipset not supported, for every pair.
The restriction lives in the driver. The slots are ordinary PCIe 4.0 with sixteen lanes each, and that hardware handles peer transfers fine. In 2024 the tinygrad project published a patch to NVIDIA's open-source kernel modules that switches them back on for the 4090. Forks carry it forward release by release, 3090s included. Reported gains range from a few percent on a pair of 5090s to around half again on four 3090s running a different model. On my cards the patch needed a bigger window into each card's memory first.
For one card to write into another, the target card's memory has to be addressable on the PCIe bus. A GPU exposes it through BAR1, a window that maps card memory into the system's address space. Out of the box the window is 256 MB on a card with 24 GB. The patched driver maps the whole card through it.
Growing the window is Resizable BAR, and it needs two things: the motherboard has to allow it, and the card's firmware has to offer the larger sizes. All four cards are EVGA RTX 3090 FTW3 Ultras, bought at different times, and the PCI capability on each split them into two production runs:
3842:3982, two cards, firmware 94.02.42.C0.05: BAR1 sizes up to 32 GB.3842:3987, two cards, firmware 94.02.26.48.15 from October 2020: BAR1 capped at 256 MB.The older pair still had launch firmware from before EVGA's Resizable BAR updates in March 2021. The newer pair's image was the wrong file for them. A card's firmware carries its own board's power tables, voltage tables, and fan wiring. Only the leading 94.02.42 is shared across NVIDIA's range, and the suffix is one board's build.
EVGA published two images for 3842:3987. The first, 94.02.42.80.AD, drew owner reports of a lowered power limit, and EVGA replaced it a week later with 94.02.42.80.27. I used that one, after checking its MD5 against the published hash and matching the subsystem and board ID in its header to the cards. With the NVIDIA driver unloaded and the model offline, I saved the original firmware from all four cards and wrote the new image to the older two with nvflash. Each FTW3 has a second firmware chip behind a switch that the flash leaves alone, a ready fallback. After a reboot both cards reported 94.02.42.80.27, with the same 420 W default power limit as before.
The motherboard setting, Re-Size BAR Support, sat one row below Above 4G Decoding and was switched off. The remote console kept missing the setup prompt, so systemctl reboot --firmware-setup booted straight into it. After one more boot every card reported a 32 GB BAR1.
On its own, Resizable BAR changed nothing measurable, and the stock driver still reported CNS for every pair.
The patched driver comes as one branch per NVIDIA release, and the user-space driver has to match it exactly, on the host and inside the container that runs the model. My version had no branch. The host moved to the nearest one, 595.71.05, with kernel modules built from the fork. The IOMMU, which normally translates every device's memory access, went into passthrough mode; the fork's direct writes fail through translation. After a reboot nvidia-smi reported OK for every pair, and copying a gigabyte between any two cards ran at 24.6 GB/s in every direction, close to the limit of sixteen PCIe 4.0 lanes.
Then I ran the benchmark, and prompts still read at 1,830 tokens a second.
NCCL, the library that runs those exchanges for the model server, had declined the new path. By default it goes peer-to-peer only between cards that sit close together on the PCIe tree, behind a shared switch. Mine meet at the processor, which nvidia-smi topo -m labels NODE or PHB. NCCL kept staging everything in host memory and logged nothing about it. One environment variable, NCCL_P2P_LEVEL=SYS, allows peer-to-peer at any distance. With debug logging on for one start, NCCL reported P2P/CUMEM on all sixteen channels.
Prompt processing more than doubled. A 16,000-token prompt the server had never seen went from 8.6 seconds to 4.0, and a 64,000-token one from 34.5 to 16.1, both at about 3,950 tokens a second. The reference build publishes roughly 2,640 to 3,190 tokens a second, on its own software at other prompt lengths; the two land in the same range, and the comparison supports no more than that.
Generation gained 11 to 16 percent. Producing a token sends far less data between cards than reading a long prompt does, and a faster link has less to work on. On the reference build's own 3,000-token code prompt, mine went from 173 to 175 tokens a second before P2P to 185 to 193 after, against the 186 to 188 it publishes with bridges.
For an agent, the number that matters is the wait before a reply starts, and the two fixes stack. An 18,000-token prompt took a little over ten seconds every time it was sent. The cache fix brought the resend down to 1.45 seconds and left the first send alone. P2P cut the first send to 4.45 seconds and the resend to 0.76.
With the cards no longer waiting on each other, they ran into the power cap. Reading a prompt, they had drawn about 231 W before P2P. Afterward they drew 271 to 278 W, and the cap was holding their clocks down on all four. Generating, they sat at 277 to 279 W, also capped.
Raising the cap to 350 W made a 64,000-token prompt read 4% faster and moved nothing else. Generation is limited by how fast each card reads its own memory, which seventy more watts per card does not change. Four percent on long prompts did not justify the extra power and heat, and the cap went back to 280.
The last question was how many agents the box can carry. The server ran at most two requests at once, and a third waited in a queue. Raising that to four cost 4,900 tokens of cache. Four 35,000-token prompts sent together, filling most of the cache, ran without an error or an eviction.
Concurrent agents got no faster. Four 16,000-token requests sent together got their first tokens at an average of 13.6 seconds with four slots and 12.5 with two, and all four finished in 25.2 seconds against 23.9. Prompt reading is the shared bottleneck now, about 4,000 tokens a second for everyone combined, and four 16,000-token turns are 64,000 tokens of reading however the slots divide them.
A short chat arriving two seconds after two agents had each sent a 30,000-token prompt still waited 18.2 seconds for its first token at four slots. The scheduler spends each step's budget on prompts it is already reading, first come first served, and a newcomer waits until they finish.
The server stayed at four anyway. A lone request runs as fast with four slots as with two. Under load every agent makes progress and none sits in a queue, for about 3% of the cache. Agents also spend much of their time running tools on the client, and three or four sessions rarely collide on the server. The cache is the softer ceiling. At 15,000 to 20,000 tokens a session, about seven fit before they start evicting each other, and an evicted agent now rereads a 20,000-token conversation in about five seconds instead of eleven.
The IOMMU no longer polices device memory access. Passthrough mode lets a device write wherever it is told to. On a host that hands hardware to virtual machines, that would be a real loss. This one hands nothing through, and the GPUs reach the model's container through the host's own driver, so the protection had nothing to guard.
The driver has to be rebuilt for every kernel. DKMS, the usual tool, would rebuild NVIDIA's stock modules and silently undo the patch. A short hook in the Proxmox kernel install rebuilds the patched ones instead, and a boot-time service catches any kernel whose headers arrived after the hook ran.
Driver upgrades wait on the fork. A new NVIDIA release is usable only once somebody ports the patch to it, and the host and the container move to it together.
One faster option stays switched off. vLLM has its own all-reduce that beats NCCL on paper. It assumes NVLink between every card, and people who have forced it on four 3090s over PCIe report outputs that drift from the correct answer. NCCL is slower and gets the right answer: a spot check after the change still answered 17 times 23 with 391.
The prompts were random words sized by token count. Each prompt was unique, and no run borrowed another's cache unless the test was about the cache. Real conversations tokenize differently. I would trust the ratios between configurations over the absolute speeds.
Speeds exclude the wait for the first token. Prompt processing is prompt tokens divided by time to first token. Generation is reply tokens divided by the time from the first token to the last, at medium reasoning effort, averaged over three runs per workload.
One user, one idle server, except in the concurrency tests. A busy server divides the same capacity between requests. The single-user numbers are a ceiling, and the concurrency section shows what sharing costs.
The first request after a restart pays for kernel compilation. That run shows up in places as an outlier, 2,491 tokens a second against 3,961 on the next. The comparisons use the warm runs.
I treated the missing bridges as a hardware limit from the day the cards went in, and for most of that time it barely mattered: services were told to disable peer-to-peer, and the models worth running fit on one card or on the pair that shares a corner of the processor. That changed when a model and an engine arrived that could use all four at once. The cards were connected the whole time. Between them sat a policy in a driver, firmware from 2020, an option one row down in a setup menu, and a library being careful about a topology nobody had told it was fine. Bigger prompt chunks and a bigger memory window each left the number alone, and each one crossed off a place the time could have been going. The answer sat behind three letters, CNS, that I had read many times without asking what they were protecting.