Fine-Tuning at Home, Part Nine: Reading the Scripts

Part eight built the dataset. The next part runs the training, and this one sits between them because reading a script before you run it is the difference between debugging and guessing.

That is not a general principle about diligence. It is specific to this: when a run finishes and the output is wrong, almost every question you will have is answered by four or five lines in these files. Which model went on the card and at what precision. What was actually trained, as opposed to what you meant to train. How many times it saw your examples. Whether it wrote a checkpoint you can go back to. Nothing in the log tells you any of that, and the files are short enough to read in the time it takes the model to download.

Everything below is the real file. The listings on this page are cut out of the scripts at build time rather than copied here, so a page that disagrees with the code you downloaded is not a failure this course can have.

The dataset builder first, because it is the shortest and it is the one whose output everything else depends on. It reads ten markdown files and writes two JSONL files. That is the whole job.

The top of it is the part worth arguing about, and part eight argues about it at length: ten instructions written by hand, one per poem, none of which mentions rhyme or verse.

# The poems are output with no record of what was asked for, so the instruction side of
# each pair is written by hand. Note that none of these mentions rhyme or verse: the voice
# has to come from training rather than from the request.
PROMPTS = {
    "01": "Write a field note about backups that are never restored.",
    "02": "Write a field note about a stale DNS record and a resolver cache.",
    "03": "Write a field note about trusting a UPS battery you never load tested.",
    "04": "Write a field note about a TLS certificate that expired unnoticed.",
    "05": "Write a field note about dust, cooling, and a machine that quietly throttled.",
    "06": "Write a field note about debug logging that filled a disk.",
    "07": "Write a field note about the spare part you decided not to buy.",
    "08": "Write a field note about a firmware update that reset a configuration.",
    "09": "Write a field note about the two cables you never got around to labeling.",
    "10": "Write a field note about monitoring that failed silently.",
}

# Kept back so the finished model can be tested on subjects it has never seen. These two
# sit furthest from the rest, so passing the test needs actual generalization.

Then the loop. Note what it does to each file: strips the markdown heading, pairs the body with its instruction, and routes two of the ten into the holdout rather than the training set.

files = sorted(p for p in POEMS.iterdir() if re.fullmatch(r"\d\d-.*\.md", p.name))
if len(files) != 10:
    raise SystemExit(f"expected 10 poems in {POEMS}, found {len(files)}")

train, holdout = [], []

for path in files:
    key = path.name[:2]
    if key not in PROMPTS:
        raise SystemExit(f"no prompt written for {path.name}")

    # Drop the markdown heading: leaving it in teaches the model that a title is part of
    # the voice, and it then appears in everything the model generates.
    body = re.sub(r"\A#[^\n]*\n+", "", path.read_text(encoding="utf-8")).strip()

    example = {
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": PROMPTS[key]},
            {"role": "assistant", "content": body},
        ]
    }
    (holdout if key in HOLDOUT else train).append(example)

The heading is stripped, and that is not tidiness. Leaving # Title at the top of every example teaches the model that a title is part of the voice, and it then emits one in everything it generates afterwards. This is a one-line change that quietly shapes every sample you will read in part eleven.

The holdout is decided here, before anything is trained. Two ids in a set. It is the least impressive line in the file and the one the entire evaluation rests on, because a model that has seen all ten can recite, and reciting is indistinguishable from style transfer unless something was kept back.

The system message is one short line on every example. Including the held-out ones. A richer persona prompt here would produce better output and would also do the job the training is supposed to do, leaving no way to attribute the result. Part eight calls this keeping the prompt side deliberately weak.

It has no dependencies at all. Standard library only, which is why it runs before you have installed anything. That is deliberate: the dataset is the part you will iterate on, and a builder that needs a working CUDA stack to run is a builder you will avoid touching.

Now the trainer. The first block is the one that decides whether any of this fits on your card.

# QLoRA: the frozen base is stored at 4 bits, which is what brings an 8B model onto a
# 24 GB card with room for the activations. The adapter itself stays at full precision,
# because it is the part being trained.
quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

Four bits, and the base model never changes, which is the whole trick. Quantization is lossy, and it is lossy on weights that are frozen anyway, so the cost lands somewhere it does comparatively little harm. An 8B model at 16 bits is roughly 16 GB before you have made room for anything else; at four it is under five.

Then the tokenizer, and a check that part four earned:

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
if tokenizer.chat_template is None:
    raise SystemExit(
        f"{BASE_MODEL} ships no chat template, so the turn markers the model was trained "
        "to recognize cannot be applied."
    )

model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=quant,
    dtype=torch.bfloat16,
    device_map={"": 0},
)
model.config.use_cache = False

A missing chat template is a silent failure, so it is a loud one here. The markers that delimit a turn are what the model was trained to key on. Without a template you would be responsible for producing them yourself, and getting it subtly wrong gives you a model that is quietly worse with nothing in the log to say why. The script refuses to start instead.

device_map pins the model to one card. {"": 0} puts the whole model on device 0 rather than letting the loader spread it across whatever it finds. On a single-GPU machine this changes nothing. On a multi-GPU machine it is the difference between a run that trains and a run that shards itself across a slow bus for no benefit.

use_cache is turned off. The key-value cache from part five is what makes generation fast and it is useless during training, where every position is processed at once rather than one token at a time. Leaving it on wastes memory and conflicts with gradient checkpointing.

The adapter. This is low-rank adaptation from part seven, expressed as about six lines of configuration.

# Low-rank adaptation. Instead of updating all 8 billion parameters, freeze them and learn
# two small matrices beside each weight table. r=16 trains well under 1% of the model.
peft_config = LoraConfig(
    r=16,                       # MATTERS: adapter capacity. Higher memorizes sooner.
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules="all-linear",
)

target_modules="all-linear" is the line to leave alone. It attaches an adapter to every linear layer in the model, which is the sane default and, more usefully, is impossible to get wrong. A hand-written list of module names that matches nothing produces a run that completes, writes checkpoints, reports a falling loss and trains absolutely nothing, which is the first failure in the list at the end of part ten.

Then the settings, which is the part people actually come to a script like this for:

config = SFTConfig(
    output_dir=OUT,
    num_train_epochs=15,        # MATTERS: passes over very few examples.
    learning_rate=1e-4,         # MATTERS: step size. Roughly 10x a full fine-tune's.
    save_strategy="epoch",      # MATTERS: keep every pass, so you can go back to the best.
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    max_length=1024,            # Above the longest example. Below it, pieces get truncated.
    gradient_checkpointing=True,
    bf16=True,
    optim="paged_adamw_8bit",
    logging_steps=1,
    report_to="none",
    # Train on the response only, so the model does not learn to generate prompts as well
    # as answers. This relies on the tokenizer's chat template marking the assistant turn;
    # if your version of TRL or the template rejects it, drop this line. Loss then covers
    # the whole conversation, which at this corpus size is a small effect but not nothing.
    assistant_only_loss=True,
)

Four of them matter and the rest are scaffolding. lora_r is how much the adapter can absorb. num_epochs is how many times it sees your handful of examples. learning_rate is how far each step moves. save_strategy keeps every pass so you can go back to the one that read best. Everything else is either forced by the hardware or a default with no reason to move.

They trade against each other rather than being independent. Raising the rank and the epochs together is how a small corpus gets memorized. If the output in part eleven turns out to be reciting, the fix is fewer passes before it is anything else, and the checkpoints written along the way are what let you find out without training again.

max_length has to clear your longest example. Set it below and the trainer truncates from the end without complaining. For a form whose entire point is the closing stanza, that means training the model to stop before it makes its argument. Part ten measures the longest example before the run, which takes thirty seconds.

The optimizer is the paged 8-bit one for a reason. Optimizer state is per-trained-parameter, and only the adapter is training, so this is a small band in the memory chart either way. The paged variant survives the momentary spikes that otherwise end a run with an out-of-memory error several minutes in.

The trainer itself is four lines, because everything interesting was decided above it.

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    peft_config=peft_config,
    processing_class=tokenizer,
)

result = trainer.train()
trainer.save_model(OUT)
tokenizer.save_pretrained(OUT)

And then the part I would not skip, which is the script telling you whether it did anything at all:

first = trainer.state.log_history[0].get("loss") if trainer.state.log_history else None
print(f"\nadapter    : {OUT}")
print(f"loss       : {first} -> {result.training_loss:.4f}")
print(

It prints the first loss beside the last. If those two numbers match to several decimal places, nothing trained. That is the failure that looks exactly like success from every other angle: the progress bar advances, the checkpoints appear, the run exits zero. Comparing the ends of the curve is the cheapest check there is and nothing else in the output reveals it.

It refuses to run on the CPU. Not out of strictness. A CPU run of this does not fail, it succeeds at roughly a thousandth of the speed, so the symptom is a job that never seems to finish rather than an error. The check at the top costs one line and saves an evening.

The last file exists to answer whether it worked, and part eleven is about doing that honestly. Mechanically it is short: load the same four-bit base, put the adapter on top, and generate.

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, quantization_config=quant, dtype=torch.bfloat16, device_map={"": 0}
)
if not args.base:
    model = PeftModel.from_pretrained(model, args.adapter)
model.eval()

The adapter is loaded over the base rather than merged into it, which is what makes the comparison cheap. The same base stays in memory and --base simply skips one line, so the trained and untrained models are one flag apart rather than two separate loads.

And the generation itself, with everything that could vary pinned:

messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": args.prompt}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

with torch.no_grad():
    out = model.generate(
        inputs,
        max_new_tokens=600,
        do_sample=True,
        temperature=args.temperature,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
    )

The seed and the temperature are arguments with defaults, not constants. They are exposed so they can be held identical across runs, which is the only thing that makes two outputs comparable. Part six is why: sampling lives outside the model, so two systems compared at different settings are not being compared at all.

It prints what produced the output. The header line names the adapter, the temperature and the seed. Reading twenty generations later and being unable to say which came from where is how an evaluation quietly becomes an opinion, and part eleven has a protocol built on the assumption that this line is there.

Four files, and none of them clever. Everything that decides how this run turns out is either in the dataset, which is part eight, or in four settings on one screen above. A framework would hide both behind a schema and buy nothing back at this size.

The thing I would take from reading them is how little of it is machine learning. Most of the lines are about where the model goes, what precision it is stored in, and whether the run can prove it did something. The training itself is four lines and it is the part you will think about least.

Next is running it: the machine, the driver, the install, and the ten minutes it actually takes.