Every command, from a bare virtual machine to a saved adapter. The surprise, if you have read the eight parts before it, is how little of the evening the training itself takes. The run is minutes. The setup around it is the whole job, and so is knowing what the loss curve is telling you while it goes.
Part 09 of 10·14 min read
What comes out the far end of this part is a file of a few tens of megabytes: a LoRA adapter that, when loaded alongside the base model, makes it write like Doc Quatrain. The base weights are untouched and the adapter is a separate artifact, which means a bad run costs nothing but the time and I can keep several and compare them.
There are two tracks through this part, and they differ only in which model you point the configuration at. The main track is what I ran. The small track exists because nothing in this course should require the machine in my office, and the honest truth is that a style transfer this narrow does not need it.
The main track is Qwen3-8B on one 24 GB card. Dense, in its instruct form, quantized to four bits for the run. Small enough to train comfortably on a single 3090, capable enough that the prose underneath the voice is worth reading, and open under a license that permits this. The run uses about eleven gigabytes, which leaves room to be careless. This is what every command and every configuration value below is written for.
The small track is Qwen3-1.7B, and it fits in about six gigabytes. Same family, same tokenizer, same chat template, same configuration file with one line changed. Six gigabytes means an 8 GB consumer card, a laptop GPU, or a free hosted notebook. If you do not have a 3090 and want to follow along on the same corpus and produce a real adapter of your own, this is the track to take, and it is not a toy version of the exercise.
The smaller model learns the rhyme nearly as well, and that is the point. This surprised me less than it should have, because it is exactly what part seven argued. Form is cheap to install: every token of every example is evidence about it, so a 1.7B model picks up the couplets and the register about as reliably as the 8B does. What the small model gives up is underneath the voice. Its technical content is thinner and its arguments are flatter, because training moved the style and could not add capability that pretraining never put there.
Do not go below about a billion parameters for this. Qwen3 goes down to 0.6B and it will run almost anywhere, and at that size the model struggles to hold a coherent argument across three hundred words at all. You get rhyming couplets attached to nothing. That is still an instructive result, and it is a poor place to learn what a good one looks like.
The trainer is the same either way. One short Python script against the libraries directly, which is the version the steps below walk through because nothing in it is hidden. The last step carries the same run as an Axolotl configuration file for anybody who would rather drive it from YAML, and the values map one to one. Neither is better. The script is easier to read and the config is easier to sweep.
What you actually need, before any of it. None of this is exotic and the smaller column is a laptop.
The card has to be NVIDIA. Not a preference. The whole stack below is built on CUDA, and while AMD cards can train through ROCm, none of these commands would be right for one and I have not tested it. Apple Silicon can run inference well and is not a realistic training target for this. If you do not have an NVIDIA card, a rented hour on a cloud GPU is the cheaper path than fighting the toolchain.
The disk figure is mostly not the model. The 8B weights are about 16 GB, and that is the part people budget for. The rest is the toolchain: PyTorch and its bundled CUDA libraries come to roughly 10 GB in the virtual environment, the toolkit adds several more, and the download tool keeps a cache that can hold a second copy unless you pass a local directory as step five does. Sixty gigabytes free is comfortable, forty is tight, and running out mid-download leaves a partial file that looks like a corrupt model.
System RAM matters more than you would expect. The weights are staged through main memory on their way to the card, so loading an 8B model briefly wants noticeably more system RAM than the file size suggests. Thirty-two gigabytes is comfortable and I would not attempt the 8B track below sixteen. The dataset itself is irrelevant here, at four thousand tokens, but that is a property of this corpus rather than of training generally.
The operating system is Ubuntu 24.04, and any modern Linux will do. That is what everything below was run on, and the package commands are Debian-flavored, so on Fedora or Arch the driver and toolkit steps differ while nothing else does. Windows works through WSL2 with the CUDA-enabled driver and I have not tested it, so treat that route as plausible rather than verified. A rented cloud instance is the same as bare metal for every step after the first.
Three things about my own hardware are worth stating before the commands, because they are the parts a reader is most likely to want to change.
This runs on one GPU, and on my machine three cards sit idle. That looks wasteful and it is correct. The cards here have no fast link between them, so data-parallel training has to push gradients over the bus on every step, and for a job that fits comfortably on a single card the coordination costs more than the parallelism returns. I have written about this constraint elsewhere on the site. The short version is that this rig is an inference machine that can train small things, and a QLoRA on an 8B model is exactly the small thing it is good at.
The power limit comes off for training and goes back on after. The cards here are capped at 280 W for serving, where the throughput cost is barely measurable because inference is bound by memory bandwidth. Training is compute-bound and those watts buy real speed, so the training card goes back to full board power for the run. On a job this short it hardly matters; on a long one it does, and the habit is worth having.
Nothing here needs the network once the weights are down. Worth noting for anybody whose interest in local training is that the data never leaves. The base model is a download, the corpus is local, and the run itself makes no outbound calls. Set the trainer to offline mode and you can verify that rather than trust it.
From a fresh virtual machine to a saved adapter. Tap any step to expand it. The commands are what I actually ran; where a flag is likely to be renamed by a future release, the note underneath says what it does so you can find the current spelling.
This step has one job: make sure the training process sees a single GPU. If you have one graphics card and an ordinary Linux install, it is already done and you can go to step two. Everything below is for machines with several cards, and the Proxmox half is for my particular arrangement rather than a requirement of the exercise.
On any machine with more than one card, pin the run to one with an environment variable. This is the whole of it, and it works identically on bare metal, in a cloud instance, and inside a virtual machine:
nvidia-smi -L # list the cards and their indices
export CUDA_VISIBLE_DEVICES=0 # the run now sees card 0 only
# confirm the process really sees one
python3 -c "import torch; print(torch.cuda.device_count())" # -> 1Why bother, when the configuration never asks for more than one card: because on a multi-GPU box one mistyped setting starts a distributed run instead of failing, and the symptom is not an error. It is a job that takes twenty times longer while every card sits at low utilization. Pinning makes that accident impossible rather than unlikely.
The rest of this step is Proxmox only. My cards live in a virtualized inference host, so the training job gets its own VM shape: the serving shape carries the driver alone, no compiler, and holds all four cards to serve a large model across them. Training wants the opposite. If you are not running Proxmox, none of this applies and nothing later depends on it.
# on the Proxmox host, from a shut-down VM
qm set 100 -memory 65536 # 64 GiB is ample for a corpus this size
qm set 100 -cores 16
# detach three of the four cards; keep hostpci0 only
qm set 100 -delete hostpci1
qm set 100 -delete hostpci2
qm set 100 -delete hostpci3
qm start 100Detaching is cleaner than the environment variable for the same reason a constraint beats a convention: with all four passed through, the wrong setting can still find them. With one attached it cannot. That is worth the reboot on a host I control, and it is not worth restructuring your machine for if you do not already work this way.
Serving needs the driver alone. Training needs the CUDA toolkit as well, because parts of the stack compile kernels at install time and will fail with a missing compiler in a way that is not obvious from the error.
sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential python3-venv python3-pip git tmux
sudo ubuntu-drivers install
sudo apt install -y nvidia-cuda-toolkit
sudo rebootAfter the reboot, confirm the card is present, the compiler exists, and the two agree:
nvidia-smi # one RTX 3090, 24576 MiB
nvcc --version # the toolkit, not just the driver
python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"That last line is the check worth running before anything else, and it is the one people skip. A stack that reports False here will still install everything successfully and still start a training run, on the CPU, at roughly a thousandth of the speed. The symptom is a job that never seems to finish rather than a job that fails.
The serving cap is 280 W and it is set by a boot service on this machine. For training, put the card back to full board power. This is a runtime change and the boot service will reassert the cap on the next reboot, which is the behavior I want.
sudo nvidia-smi -i 0 -pl 350
nvidia-smi --query-gpu=index,power.limit --format=csvCheck the room before the card. Four cards at full power is about 1.7 kW in this frame and the cooling here was sized for that, but one card at 350 W in a closed office is still a space heater running flat out. On a run this short it is academic. It stops being academic the first time you queue up something that runs overnight.
Pull whichever track you are taking, or both, since together they are under twenty gigabytes and having the small one on hand is useful for testing a configuration change quickly. The weights get quantized to four bits when they load, so these full-precision sizes matter only to your disk.
pip install -U huggingface_hub
export HF_TOKEN=hf_your_read_token
sudo mkdir -p /opt/models && sudo chown $USER /opt/models
tmux new -s dl
# main track: 8B, about 16 GB on disk, ~11 GB in use when training
hf download Qwen/Qwen3-8B --local-dir /opt/models/Qwen3-8B
# small track: 1.7B, about 3.4 GB on disk, ~6 GB in use when training
hf download Qwen/Qwen3-1.7B --local-dir /opt/models/Qwen3-1.7BConfirm the tokenizer and the chat template came down with it, because the template is what part three warned about and its absence is a silent failure rather than a loud one:
ls /opt/models/Qwen3-8B | head
python3 -c "from transformers import AutoTokenizer; \
t=AutoTokenizer.from_pretrained('/opt/models/Qwen3-8B'); \
print(bool(t.chat_template)); print(len(t))"Do this inside tmux so a dropped connection does not cost you the download; detach with Ctrl+B then D, and re-running resumes. The second command printing True is what you want. A model whose tokenizer has no chat template can still be trained, but you are then responsible for formatting the conversation yourself, and getting that wrong produces a model that is quietly worse with no error anywhere.
A virtual environment, the packages, and the data. Everything from here is Python, which is what the rest of the ecosystem is: the tokenizer checks earlier in this series are Python, the trainer is Python, and anyone adapting this to their own writing will be working in Python whether they wanted to or not.
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# the ten poems, the builder, the trainer, and this requirements file
# are all downloadable from the corpus folder linked in part eight
python3 build_corpus.py
# train.jsonl 8 examples ~2127 words
# holdout.jsonl 2 examples ~621 wordsIf pip resolves a torch build without CUDA for your setup, install it explicitly first: pip install torch --index-url https://download.pytorch.org/whl/cu124. The requirements file is deliberately unpinned, because these packages move fast enough that a version pinned in a write-up is usually wrong within months.
Thirty seconds, and it sets a value in the script that silently truncates your data if you guess it wrong. Run the corpus through the model's own tokenizer and take the longest:
python3 - <<'PY'
import json
from transformers import AutoTokenizer
t = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
n = []
for line in open("train.jsonl"):
msgs = json.loads(line)["messages"]
n.append(len(t.apply_chat_template(msgs, tokenize=True)))
print("examples:", len(n), "longest:", max(n), "total:", sum(n))
PYOn this corpus the longest lands a little over five hundred tokens, so max_length=1024 in the script has comfortable headroom. Set it below the longest example and the trainer truncates from the end without complaining, which for a form whose whole point is its closing stanza means training the model to stop before it makes its argument. This is the cheapest check in the whole process.
The whole trainer is one file, and this is the part that decides whether it fits on your card. The base weights are frozen, so storing them at four bits costs very little and is what brings an 8B model onto a 24 GB card with room left for the activations.
# QLoRA: the frozen base is stored at 4 bits, which is what brings an 8B model onto a
# 24 GB card with room for the activations. The adapter itself stays at full precision,
# because it is the part being trained.
quant = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
Then the tokenizer, with the check part three argued for. A model with no chat template can still be trained, and the result is quietly worse for a reason nothing will report:
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
if tokenizer.chat_template is None:
raise SystemExit(
f"{BASE_MODEL} ships no chat template, so the turn markers the model was trained "
"to recognize cannot be applied. Part three explains why that matters."
)
model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
quantization_config=quant,
dtype=torch.bfloat16,
device_map={"": 0},
)
model.config.use_cache = False
device_map={"": 0} puts the whole model on card 0 rather than letting accelerate spread it. On a single-GPU machine this changes nothing; on mine it is the difference between a run that trains and a run that shards itself across four cards over a slow bus for no benefit.
Low-rank adaptation, configured. target_modules="all-linear" attaches an adapter to every linear layer, which is the sane default and avoids the failure in the last section of this part, where a hand-written module list matched nothing and the run trained air.
# Low-rank adaptation. Instead of updating all 8 billion parameters, freeze them and learn
# two small matrices beside each weight table. r=16 trains well under 1% of the model.
peft_config = LoraConfig(
r=16, # MATTERS: adapter capacity. Higher memorizes sooner.
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules="all-linear",
)
Then the training settings. Four of them matter and the rest are either forced by the hardware or defaults I have not had reason to move:
config = SFTConfig(
output_dir=OUT,
num_train_epochs=15, # MATTERS: passes over very few examples.
learning_rate=1e-4, # MATTERS: step size. Roughly 10x a full fine-tune's.
save_strategy="epoch", # MATTERS: keep every pass, so you can go back to the best.
lr_scheduler_type="cosine",
warmup_ratio=0.1,
per_device_train_batch_size=1,
gradient_accumulation_steps=4,
max_length=1024, # Above the longest example. Below it, pieces get truncated.
gradient_checkpointing=True,
bf16=True,
optim="paged_adamw_8bit",
logging_steps=1,
report_to="none",
# Train on the response only, so the model does not learn to generate prompts as well
# as answers. This relies on the tokenizer's chat template marking the assistant turn;
# if your version of TRL or the template rejects it, drop this line. Loss then covers
# the whole conversation, which at this corpus size is a small effect but not nothing.
assistant_only_loss=True,
)
r=16 is how much the adapter can absorb, and turning it up is how you get memorization rather than a voice. 15 epochs is high because eight examples is few, and it is the setting most likely to need changing. 1e-4 is roughly ten times what you would use updating every parameter. save_strategy="epoch" is not a performance setting at all: it writes a checkpoint after every pass, so when the last one turns out to be overcooked the sixth is still on disk.
The trainer itself is four lines, because everything interesting was decided above it.
trainer = SFTTrainer(
model=model,
args=config,
train_dataset=dataset,
peft_config=peft_config,
processing_class=tokenizer,
)
result = trainer.train()
trainer.save_model(OUT)
tokenizer.save_pretrained(OUT)
Launch it inside tmux, with the output captured so the log survives the terminal:
tmux new -s train
python3 train.py 2>&1 | tee train.log
# the smaller track is one environment variable
BASE_MODEL=Qwen/Qwen3-1.7B python3 train.pyIn a second shell, watch what the card is actually doing:
watch -n 2 nvidia-smi --query-gpu=utilization.gpu,memory.used,power.draw \
--format=csvExpect roughly eleven gigabytes in use on the 8B model and utilization well up during the steps. If memory sits near the ceiling, lower max_length before anything else, then r. If utilization is low and the run crawls, it is on the CPU, and the script refuses to start in that case for exactly this reason.
The script above uses the libraries directly, which is the version worth reading because nothing is hidden. If you would rather drive a trainer from a configuration file, Axolotl takes the same run as YAML, and it is downloadable alongside everything else:
base_model: /opt/models/Qwen3-8B
load_in_4bit: true # QLoRA: frozen base at 4 bits
adapter: qlora
bf16: true
datasets:
- path: /opt/ft/data/train.jsonl
type: chat_template # use the tokenizer's own template
field_messages: messages
val_set_size: 0 # the holdout is judged by hand, not scored
sequence_len: 1024 # > the longest example, measured above
sample_packing: false
train_on_inputs: false # loss on the response only
lora_r: 16 # MATTERS: adapter capacity
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
num_epochs: 15 # MATTERS: passes over 8 examples
learning_rate: 0.0001 # MATTERS: step size
lr_scheduler: cosine
warmup_ratio: 0.1
micro_batch_size: 1
gradient_accumulation_steps: 4
optimizer: adamw_bnb_8bit # 8-bit moments, less memory
gradient_checkpointing: true # recompute instead of storing
flash_attention: true # drop this if flash-attn would not build
output_dir: /opt/ft/out/doc-quatrain
save_strategy: epoch # MATTERS: keep every pass to compare
logging_steps: 1axolotl train qwen3-8b-quatrain.ymlSame settings, same result, and the values map one to one onto the script. Older releases of the trainer are launched as accelerate launch -m axolotl.cli.train <config>; if axolotl train is not a command, that is the spelling to try. Neither path is better. The script is easier to read and the config is easier to sweep.
This is the part that surprised me, and it is why the series spends a whole part on the corpus and only this one on the run.
# 8 examples x 15 epochs = 120 sample passes
# at gradient_accumulation_steps: 4 -> 30 optimizer steps
loading base model (4-bit) ......... ~2 min
training ........................... ~6 min
writing adapters ................... ~10 s
ls /opt/ft/out/doc-quatrain/
checkpoint-2 checkpoint-4 ... checkpoint-30 adapter_model.safetensorsUnder ten minutes end to end, most of it spent loading the model. The evening was long; the training was not. Nearly all of the wall-clock went into the environment above and the evaluation in the next part.
Thirty optimizer steps is a very small number and it is worth letting that land. This is not a scaled-down version of what a lab does, it is a different activity that happens to use the same machinery. Anybody promising that an afternoon on a consumer card produces a meaningfully new capability is describing something other than this.
The log prints a loss after every step and it is the only live signal you get, so it is worth knowing what it can and cannot tell you.
It should fall quickly at first and then flatten. On this run it started somewhere above two, dropped hard over the first pass or two as the model registered that everything it is being shown rhymes, and then settled into a slow decline. That shape is the normal one.
Two shapes are wrong and both are unmistakable. A loss that climbs, or oscillates violently, means the learning rate is too high, and the fix is to cut it and start again rather than to wait and hope. A loss that barely moves means the opposite, or that the adapter is not attached to anything, and the second is more common than the first: a configuration that targets no layers trains nothing at all while reporting success at every step.
What the loss cannot tell you is the thing you actually want to know. It measures how well the model predicts the eight poems it is being trained on, so it falls fastest exactly when the model is memorizing them. A training loss approaching zero is not a triumph, it is the specific symptom of the failure part seven described. There is no number in the log that distinguishes a model that learned a voice from one that learned eight poems, which is why the holdout exists and why the next part is done by reading rather than by scoring.
The first run trained nothing, successfully. A misconfigured target setting meant the adapter attached to no layers. The run completed, the loss printed, the checkpoints were written, and the resulting model was byte-identical in behavior to the base. Nothing anywhere said so. The check that catches it is to compare the loss at the first step against the loss at the last: if they match to several decimal places, no learning happened regardless of what the progress bar did.
The second run was overcooked and I nearly shipped it. At thirty epochs the training loss looked wonderful and the model recited. Ask for a field note about certificates and back came the certificate poem with three words changed. That is what the failure looks like from the inside: not obviously broken, just suspiciously good. Comparing against a held-out topic is what exposed it, and nothing else would have.
The compiler was missing and the error did not say so. An install that fails while building an attention kernel produces a long trace whose actual cause is one line near the top about a missing compiler. Installing the toolkit fixed it in a minute after I had spent forty on the wrong end of the message.
A truncated example taught the model to stop early. An early configuration had a sequence length below the longest poem, so that piece lost its closing stanza. The model then produced pieces that ended one stanza short of their point, which read as a subtle stylistic quirk rather than as data loss. Measuring the longest example first is the fix and it takes thirty seconds.
Keeping every checkpoint cost nothing and saved the run. Adapters are tens of megabytes. Writing one after every pass meant that when the final one turned out to be memorizing, six earlier versions were sitting on disk to compare against, and the one I ended up using was not the last. This is the setting I would argue hardest for and it is the one most tutorials leave at its default.
The run is the part of this project that photographs well and it is the part that matters least. Thirty optimizer steps, six minutes, four settings worth touching. Everything that determined whether those six minutes produced anything happened in the previous part, in a text editor, deciding what to put on the prompt side of eight examples.
That ratio is not specific to this project. It is the general shape of small fine-tuning work, and the reason so much of the published advice is about hyperparameters is that hyperparameters are easy to write about and datasets are not.
What sits on disk now is a set of adapters, one per pass, and no way yet to say which of them is any good. The training loss prefers the last one and the training loss is exactly the wrong judge.
Next is merging an adapter into the base weights, serving the result, and putting it in front of the two poems it was never shown. That last part is the one I was actually curious about: the rhyme was always going to transfer, and the question is whether the model picked up the habit of opening on its own mistake and closing on what bites you next, which nobody ever told it to do.