The Hunt for Local Frontier AI

This build assumes the host already has AMD IOMMU enabled and the four 3090s bound to vfio-pci. Confirm that before creating the VM. If a GPU is still bound to nvidia/nouveau on the host, passthrough won't work. First, that IOMMU is active and groups populate:

dmesg | grep -i -e DMAR -e IOMMU | head
ls /sys/kernel/iommu_groups | wc -l      # > 0 → IOMMU is on

Find the four GPUs (NVIDIA vendor id 10de) and confirm each is bound to vfio-pci:

lspci -nnk -d 10de:
# expect c1:00, c2:00, 01:00, 81:00
# each → "Kernel driver in use: vfio-pci"  (not nvidia/nouveau)

Confirm the IOMMU groups are clean, each GPU isolated:

for d in /sys/kernel/iommu_groups/*/devices/*; do
  g=${d#*/iommu_groups/}; g=${g%%/*}
  printf 'group %s: ' "$g"; lspci -nns "${d##*/}"
done | grep -i nvidia

Each 3090 should sit alone in its group (at most its own audio function and root-port bridge), so no ACS-override hack is needed. Link-width check on a card: lspci -vvs c1:00.0 | grep -i lnksta → Speed 16GT/s, Width x16 (PCIe 4.0 x16).

VM 100 (ubuntu-vllm). Confirmed-good settings:

Machine     q35
BIOS        OVMF (UEFI)  + add EFI disk
SCSI        VirtIO SCSI single
Disk        scsi0 · 1000 GB on NVMe · discard + iothread
CPU         host · 24 cores
Memory      90112 MiB (88 GiB) · ballooning OFF
Network     vmbr0 · VirtIO

Attach the four GPUs: All Functions + PCI-Express, and Primary GPU unchecked on all of them (that box sets x-vga=1, which fights the headless console). Add the c1/c2 pair first so the priority pair lands on CUDA 0,1:

qm set 100 -hostpci0 0000:c1:00,pcie=1   # → CUDA 0  (priority pair)
qm set 100 -hostpci1 0000:c2:00,pcie=1   # → CUDA 1
qm set 100 -hostpci2 0000:01:00,pcie=1   # → CUDA 2
qm set 100 -hostpci3 0000:81:00,pcie=1   # → CUDA 3

Sanity-check the config: qm config 100 → expect machine: q35, bios: ovmf, balloon: 0, the four hostpci lines, and no x-vga.

Secure Boot: OVMF ships with pre-enrolled keys, so the proprietary driver won't load without MOK enrollment. Easiest headless path: disable Secure Boot at first boot (Esc at the TianoCore splash → Device Manager → Secure Boot Configuration → disable), or complete MOK enrollment during the driver install.

Minimal Ubuntu Server 24.04 install with Install OpenSSH server checked and a static IP / DHCP reservation. On the storage summary set ubuntu-lv to max. Guided "use entire disk + LVM" only allocates ~100 GB and strands the rest. Choose "Do not install third-party drivers now." If you skipped the LV resize, fix it after boot:

sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
sudo resize2fs /dev/ubuntu-vg/ubuntu-lv
df -h /

Update, then install the driver (595-server, driver only, no CUDA toolkit):

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential python3-venv python3-pip tmux
sudo ubuntu-drivers install
sudo reboot

Confirm cards + topology, and lock device order (also set in the service later, since /etc/environment only covers interactive shells):

nvidia-smi                    # 4× RTX 3090, 24576 MiB each
nvidia-smi topo -m            # all PHB, NUMA node 0
nvidia-smi topo -p2p r        # all NS (no P2P), expected on Zen 2
echo 'CUDA_DEVICE_ORDER=PCI_BUS_ID' | sudo tee -a /etc/environment

If Secure Boot is still enabled, you set a MOK password here and must choose Enroll MOK on the blue screen at the next reboot. Otherwise nvidia-smi comes up empty.

Four cards at 420 W is ~1.7 kW; 280 W barely affects inference throughput while running far cooler. Create /etc/systemd/system/nvidia-powerlimit.service:

[Unit]
Description=NVIDIA persistence mode and power limit

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 280

[Install]
WantedBy=multi-user.target

Enable and verify:

sudo systemctl daemon-reload
sudo systemctl enable --now nvidia-powerlimit.service
nvidia-smi --query-gpu=index,power.limit,persistence_mode --format=csv

This 280 W cap is for inference, which is memory-bandwidth bound, so the throughput hit is barely measurable. Training is different: it's compute-bound, so for fine-tuning runs the limit comes off (nvidia-smi -pl 350 and up) and the extra watts translate into real training throughput.

vLLM in a venv (needs >=0.19.0 for the Qwen3.6 models):

sudo mkdir -p /opt/vllm && sudo chown $USER /opt/vllm
python3 -m venv /opt/vllm/venv && source /opt/vllm/venv/bin/activate
pip install --upgrade pip && pip install vllm
vllm --version

Pull AWQ quants (vLLM can't load GGUF) from the cyankiwi quant team. Authenticate with a free HF Read token, then download inside tmux with the Xet backend disabled:

pip install -U huggingface_hub
export HF_TOKEN=hf_your_read_token
sudo mkdir -p /opt/models && sudo chown $USER /opt/models

tmux new -s dl
export HF_HUB_DISABLE_XET=1      # hf-xet wedged the VM; force classic HTTPS
hf download cyankiwi/Qwen3-Coder-Next-AWQ-4bit  --local-dir /opt/models/Qwen3-Coder-Next-AWQ-4bit
hf download cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4  --local-dir /opt/models/Qwen3.6-27B-AWQ-BF16-INT4
# optional: bundled coder:
hf download cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit   --local-dir /opt/models/Qwen3.6-35B-A3B-AWQ-4bit

Detach tmux with Ctrl+B then D; downloads resume on re-run. The core set is Coder-Next 80B (~45 GB, best coder) and Qwen3.6-27B (~28 GB, planner); the 35B-A3B (~20 GB) is only for the both-live bundle. If hf-xet hangs anyway (unkillable D-state), reboot the VM from Proxmox; partial files are preserved.

Default daily driver: Coder-Next, all four GPUs, pipeline-parallel (required, P2P is NS). VLLM_USE_FLASHINFER_SAMPLER=0 is mandatory on a driver-only box, or FlashInfer JIT-compiles a sampler kernel at startup and crashes with "Could not find nvcc":

NCCL_P2P_DISABLE=1 CUDA_DEVICE_ORDER=PCI_BUS_ID VLLM_USE_FLASHINFER_SAMPLER=0 \
/opt/vllm/venv/bin/vllm serve /opt/models/Qwen3-Coder-Next-AWQ-4bit \
  --pipeline-parallel-size 4 --gpu-memory-utilization 0.92 --max-model-len 262144 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --host 0.0.0.0 --port 8000

First start takes a few minutes (weight load + torch.compile + CUDA-graph capture, then cached). Watch for Application startup complete, then confirm in another shell:

curl http://localhost:8000/v1/models

Benign startup warnings to ignore: Triton kernel JIT on the first request, unbatched P2P op / new 2-rank NCCL communicator (expected with NCCL_P2P_DISABLE=1), and a cosmetic torch.frombuffer "buffer is not writable" UserWarning. A healthy log shows ~90–100 tokens/s at Running: 1 reqs. If it OOMs on the KV cache, dial --max-model-len back toward 200000.

Three unit files, one per model, each on its own permanent port so a saved client chat always maps to one known model:

Unit                          Model                GPUs        Port  Served name
vllm-qwen-coder-next.service  Qwen3-Coder-Next 80B  all 4 (PP4)  8000  qwen-coder-next
vllm-qwen-planner.service     Qwen3.6-27B           0,1 (TP2)    8001  qwen-planner
vllm-qwen-bundle.service      Qwen3.6-35B-A3B       2,3 (TP2)    8002  qwen-bundle

The coder-next unit, /etc/systemd/system/vllm-qwen-coder-next.service:

[Unit]
Description=vLLM coder-next: Qwen3-Coder-Next 80B (all 4 GPUs)
After=network-online.target nvidia-powerlimit.service
Wants=network-online.target nvidia-powerlimit.service

[Service]
User=anandpatel
Environment=CUDA_DEVICE_ORDER=PCI_BUS_ID
Environment=NCCL_P2P_DISABLE=1
Environment=VLLM_USE_FLASHINFER_SAMPLER=0
ExecStart=/opt/vllm/venv/bin/vllm serve /opt/models/Qwen3-Coder-Next-AWQ-4bit --pipeline-parallel-size 4 --gpu-memory-utilization 0.92 --max-model-len 262144 --enable-auto-tool-choice --tool-call-parser qwen3_coder --served-model-name qwen-coder-next --host 0.0.0.0 --port 8000
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

The planner and bundle units are the same shape, each pinned to a GPU pair. They add CUDA_VISIBLE_DEVICES=0,1 (or 2,3), --tensor-parallel-size 2, --reasoning-parser qwen3, and --gpu-memory-utilization 0.85 --enforce-eager --max-num-seqs 4. Those last three are what let the hybrid Qwen3.6 models fit and run cleanly on a 24 GB card at TP2:

# vllm-qwen-planner.service  (27B · GPUs 0,1 · port 8001)
Environment=CUDA_VISIBLE_DEVICES=0,1
ExecStart=... /opt/models/Qwen3.6-27B-AWQ-BF16-INT4 --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 --enforce-eager --max-num-seqs 4 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --served-model-name qwen-planner --port 8001

Enable and watch; switch layouts by disabling the side that shares the GPUs first:

sudo systemctl daemon-reload
sudo systemctl enable --now vllm-qwen-coder-next
journalctl -u vllm-qwen-coder-next -f

# handoff: plan on the 27B, swap, implement on the 80B
sudo systemctl stop  vllm-qwen-planner
sudo systemctl start vllm-qwen-coder-next   # ~1–2 min to load

Replace User= with your login (whoami). A literal <user> makes systemd fail 217/USER and crash-loop every 10 s. Three ways to work: best coder (80B solo), best planner (27B solo), best plan + code via a handoff (the plan is a .md that survives the model swap), or the both-live bundle (27B + 35B, instant Plan/Act toggle, weaker coder).

Every client lives on a separate CPU-only apps VM and reaches vLLM at :8000 / :8001 / :8002 as a plain OpenAI-compatible backend (the API key is ignored unless you launch with --api-key). Open WebUI: Admin → Settings → Connections, one OpenAI connection per port (:8000→qwen-coder-next, :8001→qwen-planner, :8002→qwen-bundle), key any non-empty string.

opencode: ~/.config/opencode/opencode.json, a custom provider per endpoint so Plan/Act can plan on the 27B and act on the coder:

{
  "provider": {
    "vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://<vllm-ip>:8000/v1", "apiKey": "sk-local" },
      "models": { "qwen-coder-next": { "name": "Qwen3-Coder-Next" } }
    }
  },
  "model": "vllm/qwen-coder-next"
}

Hermes: point the default profile at the 80B coder (hermes model → Custom endpoint → :8000) and add a planner profile on the 27B (hermes -p planner model → :8001). The big local-only gotcha: auxiliary side-jobs default to a cloud chain and silently fail with no key, so pin them to the local model in ~/.hermes/config.yaml:

auxiliary:
  compression: { provider: main }
  web_extract:  { provider: main }
  vision:       { provider: main }

Each vLLM port serves exactly one model, so a saved chat always maps to the same one. curl -s http://<vllm-ip>:8000/v1/models shows the served name to reference in every client.

vLLM exposes Prometheus metrics at /metrics on each API port. Scrape all three, plus the GPU and host exporters (a port reads DOWN when its model isn't running, which is expected; the per-model split comes from the model_name label, not the port):

# prometheus.yaml
global: { scrape_interval: 5s }
scrape_configs:
  - job_name: vllm
    static_configs:
      - targets: ['<vllm-ip>:8000','<vllm-ip>:8001','<vllm-ip>:8002']
  - job_name: dcgm
    static_configs: [{ targets: ['<vllm-ip>:9400'] }]
  - job_name: node
    static_configs: [{ targets: ['<vllm-ip>:9100'] }]

On the vLLM VM, add the GPU + host exporters (the DCGM snap bundles everything, node_exporter comes from apt):

sudo snap install dcgm && sudo snap start dcgm.dcgm-exporter   # GPU telemetry :9400
sudo apt install -y prometheus-node-exporter                    # host metrics  :9100
sudo ufw allow 9400/tcp; sudo ufw allow 9100/tcp

tokens/sec, split per model by label (not by port):

sum by (model_name) (rate(vllm:generation_tokens_total[1m]))

Import three Grafana dashboards for the full picture: the official vLLM board (app-level tokens/queue/KV), DCGM ID 12239 (per-GPU temp/power/VRAM/clocks), and Node Exporter Full ID 1860 (host CPU/RAM/disk). On consumer 3090s the DCGM profiling fields may stay empty, and the core panels don't need them.