Fine-Tuning at Home, Part Nine: Reading the Scripts

Part eight built the dataset. The next part runs the training, and this one sits between them because reading a script before you run it is the difference between debugging and guessing.

Read a script first for a specific reason: when a run finishes and the output is wrong, almost every question you will have is answered by four or five lines in these files. Which model went on the card and at what precision. What was actually trained, as opposed to what you meant to train. How many times it saw your examples. Whether it wrote a checkpoint you can go back to. The log tells you none of that, and the files are short enough to read in the time it takes the model to download.

A second reason exists, and I did not expect it. Three of the ways this goes wrong produce no error at all: a torch that quietly fell back to the processor, a sequence length that cuts the end off your longest example, and an adapter attached to no layers. All three complete, print a progress bar, and write checkpoints. The scripts check for them before training starts, and those checks are the most useful thing in the files.

Everything below is the real file. The listings on this page are cut out of the scripts at build time instead of copied here, so a page that disagrees with the code you downloaded is not a failure this course can have. Each one also names the figure earlier in the series that explains what it is doing, because every block here is a mechanism one of those pictures already showed you.

The dataset builder first, because it is the shortest, it needs no GPU and nothing installed, and everything else depends on its output. It reads a folder of markdown files and writes two JSONL files, one to train on and one held back, which is the whole job.

The instruction side is not in the script. It lives with the corpus, because it is a fact about one set of poems rather than about the tool, and it is the part worth arguing about: part eight spends its longest section on it. Fourteen instructions written by hand, one per poem, and not one of them mentions rhyme or verse.

{
  "system": "You are Doc Quatrain. You write short technical field notes.",
  "holdout": [
    "05",
    "10",
    "11",
    "12",
    "13",
    "14"
  ],
  "prompts": {
    "01": "Write a field note about backups that are never restored.",
    "02": "Write a field note about a stale DNS record and a resolver cache.",
    "03": "Write a field note about trusting a UPS battery you never load tested.",
    "04": "Write a field note about a TLS certificate that expired unnoticed.",
    "05": "Write a field note about dust, cooling, and a machine that quietly throttled.",
    "06": "Write a field note about debug logging that filled a disk.",
    "07": "Write a field note about the spare part you decided not to buy.",
    "08": "Write a field note about a firmware update that reset a configuration.",
    "09": "Write a field note about the two cables you never got around to labeling.",
    "10": "Write a field note about monitoring that failed silently.",
    "11": "Write a field note about access that was granted and never taken away.",
    "12": "Write a field note about a retry that made an outage worse.",
    "13": "Write a field note about a dependency that changed without being named.",
    "14": "Write a field note about a configuration default nobody revisited."
  }
}

Then the loop, which is the shortest interesting code in the project. Note what it does to each file: strips the markdown heading, pairs the body with its instruction, and routes the held-out pieces away from the training set before anything else happens.

for path in files:
    key = path.name[:2]

    # Drop the markdown heading: leaving it in teaches the model that a title is part of
    # the voice, and it then appears in everything the model generates.
    body = re.sub(r"\A#[^\n]*\n+", "", path.read_text(encoding="utf-8")).strip()
    if not body:
        raise SystemExit(f"{path.name} is empty once its heading is removed")

    example = {
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": PROMPTS[key]},
            {"role": "assistant", "content": body},
        ]
    }
    (holdout if key in HOLDOUT else train).append(example)

The heading is stripped, and that is not tidiness. Leaving # Title at the top of every example teaches the model that a title is part of the voice, and it then emits one in everything it generates afterwards. Stripping the heading is a one-line change that quietly shapes every sample you will read in part eleven.

The holdout is decided here, before anything is trained. Two names in a set, the least impressive line in the file and the one the entire evaluation rests on, because a model that has seen every piece can recite, and reciting is indistinguishable from style transfer unless something was kept back.

The system message is one short line on every example. Including the held-out ones. A richer persona prompt here would produce better output and would also do the job the training is supposed to do, leaving no way to attribute the result. Part eight calls this keeping the prompt side deliberately weak.

It checks that the poems and the instructions agree. Every poem needs an instruction and every instruction needs a poem, and a mismatch is reported by name instead of surfacing later as a training set that is quietly one example short. The same check catches a holdout naming a piece that is not there, which is how a renamed file turns into an evaluation with nothing held back.

It has no dependencies at all. Standard library only, which is why it runs before you have installed anything. The restriction is deliberate: the dataset is the part you will iterate on, and a builder that needs a working CUDA stack to run is a builder you will avoid touching.

Now the trainer. The first block is the one that decides whether any of this fits on your card, and it is the memory arithmetic from part one expressed as four arguments.

# QLoRA: the frozen base is stored at 4 bits, which is what brings an 8B model onto a
# 24 GB card with room for the activations. The adapter itself stays at full precision,
# because it is the part being trained.
quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

Four bits, and the base model never changes, which is the trick. Quantization is lossy, and it is lossy on weights that are frozen anyway, so the cost lands somewhere it does comparatively little harm. An 8B model at 16 bits is roughly 16 GB before you have made room for anything else; at four it is under five. The LoRA figure in part seven is the other half of why this fits.

Then the tokenizer, and a check that part three earned:

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
if tokenizer.chat_template is None:
    raise SystemExit(
        f"{BASE_MODEL} ships no chat template, so the turn markers the model was trained "
        "to recognize cannot be applied."
    )

And the check I would least want to lose, because the failure it catches is the one that looks most like success. Everything is measured in tokens, and the sequence length is a number you set by hand, so the script measures the longest example against it instead of trusting either.

# Measure the longest example before training rather than after. A sequence length below it
# truncates from the end, silently, and for a form that closes on its most important stanza
# that means training the model to stop before it makes its point. Thirty seconds here
# against a run you would otherwise throw away.
lengths = [len(tokenizer.apply_chat_template(row["messages"], tokenize=True))
           for row in dataset]
longest = max(lengths)
print(f"examples   : {len(dataset)}, longest {longest} tokens, max_length {MAX_LENGTH}")
if longest > MAX_LENGTH:
    raise SystemExit(
        f"the longest example is {longest} tokens and max_length is {MAX_LENGTH}, so it "
        f"would be cut off.\nRaise MAX_LENGTH to at least {longest} (more memory), or "
        "shorten that example."
    )

A missing chat template is a silent failure, so it is a loud one here. The markers that delimit a turn are what the model was trained to key on. Without a template you would be responsible for producing them yourself, and getting it subtly wrong gives you a model that is quietly worse with nothing in the log to say why. The script refuses to start instead.

Truncation is measured, never assumed. Set the sequence length below the longest example and the trainer cuts the end off it without complaining. For a form whose entire point is its closing stanza, that means training the model to stop before it makes its argument, and the symptom is output that reads as a stylistic quirk instead of as data loss. The measurement costs seconds and it names the number to raise.

device_map pins the model to one card. {"": 0} puts the whole model on device 0 instead of letting the loader spread it across whatever it finds. On a single-GPU machine this changes nothing. On a multi-GPU machine it is the difference between a run that trains and a run that shards itself across a slow bus for no benefit.

use_cache is turned off. The key-value cache from part five is what makes generation fast and it is useless during training, where every position is processed at once instead of one token at a time. Leaving it on wastes memory and conflicts with gradient checkpointing.

The adapter. LoRA is the low-rank trick drawn to scale in part seven, expressed as about six lines of configuration.

# Low-rank adaptation. Instead of updating all 8 billion parameters, freeze them and learn
# two small matrices beside each weight table. r=16 trains well under 1% of the model.
peft_config = LoraConfig(
    r=16,                       # MATTERS: adapter capacity. Higher memorizes sooner.
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules="all-linear",
)

target_modules="all-linear" is the line to leave alone. It attaches an adapter to every linear layer in the model, which is the sane default and, more usefully, is impossible to get wrong. A hand-written list of module names that matches nothing produces a run that completes, writes checkpoints, and reports a falling loss while training absolutely nothing, and it is the first failure in the list at the end of part ten.

Then the settings, which is the part people actually come to a script like this for:

config = SFTConfig(
    output_dir=OUT,
    num_train_epochs=15,        # MATTERS: passes over very few examples.
    learning_rate=1e-4,         # MATTERS: step size. Roughly 10x a full fine-tune's.
    save_strategy="epoch",      # MATTERS: keep every pass, so you can go back to the best.
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    max_length=MAX_LENGTH,
    gradient_checkpointing=True,
    bf16=True,
    optim="paged_adamw_8bit",
    logging_steps=1,
    report_to="none",
    # Train on the response only, so the model does not learn to generate prompts as well
    # as answers. This relies on the tokenizer's chat template marking the assistant turn;
    # if your version of TRL or the template rejects it, drop this line. Loss then covers
    # the whole conversation, which at this corpus size is a small effect but not nothing.
    assistant_only_loss=True,
)

Four of them matter and the rest are scaffolding. r is how much the adapter can absorb. num_train_epochs is how many times it sees your handful of examples. learning_rate is how far each step moves. save_strategy keeps every pass so you can go back to the one that read best. Everything else is either forced by the hardware or a default with no reason to move.

They trade against each other instead of moving independently. Raising the rank and the epochs together is how a small corpus gets memorized, which is the curve part seven draws: training loss falls fastest exactly when the model is learning the examples themselves. If the output in part eleven turns out to be reciting, the fix is fewer passes before it is anything else, and the checkpoints written along the way are what let you find out without training again.

The sequence length is one constant, used twice. It sets the trainer's max_length and it is the number the measurement above compares against, so the check and the setting cannot drift apart. Lowering it is also the first thing to try when a run will not fit, ahead of the rank.

The optimizer is the paged 8-bit one for a reason. Optimizer state is per-trained-parameter, and only the adapter is training, so this is a small band in the memory chart either way. The paged variant survives the momentary spikes that otherwise end a run with an out-of-memory error several minutes in.

The trainer itself is four lines, because everything interesting was decided above it.

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    peft_config=peft_config,
    processing_class=tokenizer,
)

And then the check that earns its place more than anything else in these files. It counts what is actually going to be trained, before a single step runs.

# What is actually going to be trained. A target_modules that matches nothing produces a
# run that completes, writes checkpoints and reports a falling loss while changing no
# weights at all, and this is the only place that failure is visible before it wastes the
# run rather than after.
trainable = sum(p.numel() for p in trainer.model.parameters() if p.requires_grad)
total = sum(p.numel() for p in trainer.model.parameters())
print(f"trainable  : {trainable:,} of {total:,} parameters ({100 * trainable / total:.3f}%)")
if trainable == 0:
    raise SystemExit(
        "the adapter attached to no layers, so this run would train nothing while looking "
        "like it worked. Check target_modules in the LoraConfig above."
    )

That number is the answer to a question nothing else in the output can answer. An adapter that attached to nothing reports zero here and stops; a healthy QLoRA on an 8B model reports something under one percent, which is the claim part seven makes about low-rank adaptation turned into a figure you can read off your own run.

Afterwards, the two numbers that say whether the run did anything:

# The first logged loss beside the last one. Not the average the trainer returns: on a run
# that trained nothing every step reports the same number, and an average hides that by
# looking like a plausible loss.
losses = [e["loss"] for e in trainer.state.log_history if "loss" in e]
print(f"\nadapter    : {OUT}")
if len(losses) >= 2:
    print(f"loss       : {losses[0]:.4f} -> {losses[-1]:.4f}  (mean {result.training_loss:.4f})")
    if abs(losses[0] - losses[-1]) < 1e-4:
        print("\nThose two are the same to four decimal places, which means nothing trained.")
else:
    print(f"loss       : mean {result.training_loss:.4f}")

The trainable count is the cheapest bug in the project to catch. A run that trains nothing takes the same six minutes, writes the same checkpoints, and exits zero. Comparing the first loss against the last catches it afterwards. Counting the parameters catches it in the second before the run starts, which is the difference between reading a log and losing an evening.

It prints the first logged loss beside the last, not the average. The average is what the trainer hands back and it is the wrong number for this: on a run that changed nothing every step reports the same value, and averaging them produces a plausible-looking loss that hides it. The ends of the curve are what the three curve shapes in part ten are about.

It refuses to run on the CPU. Not out of strictness. A CPU run of this does not fail, it succeeds at roughly a thousandth of the speed, so the symptom is a job that never seems to finish rather than an error. The check at the top costs one line and saves an evening, and it names the likeliest cause, which is pip resolving a build of torch without CUDA in it.

It says what it is about to do before it does it. The examples, the longest one in tokens, the sequence length, the trainable fraction, and the number of optimizer steps to expect. Thirty steps is a very small number and seeing it printed before the progress bar starts is what stops you reading a two-minute run as a failure.

The last file exists to answer whether it worked, and part eleven is about doing that honestly. Mechanically it is short: load the same four-bit base, put the adapter on top, and generate.

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, quantization_config=quant, dtype=torch.bfloat16, device_map={"": 0}
)
if not args.base:
    model = PeftModel.from_pretrained(model, args.adapter)
model.eval()

The adapter is loaded over the base rather than merged into it, which is what makes the comparison cheap. The same base stays in memory and --base simply skips one line, so the trained and untrained models are one flag apart instead of two separate loads.

Then the generation itself, with everything that could vary pinned:

messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=args.max_new_tokens,
        do_sample=True,
        temperature=args.temperature,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
    )

And the line that makes twenty of these readable a week later:

# The header says what produced the text. Reading twenty of these later and being unable to
# say which came from where is how an evaluation quietly turns into an opinion.
new = out[0][inputs["input_ids"].shape[-1]:]
finished = "stop" if new[-1].item() == tokenizer.eos_token_id else f"length ({args.max_new_tokens})"
print(f"# {'base model' if args.base else args.adapter}")
print(f"# temperature {args.temperature}, seed {args.seed}, {len(new)} tokens, ended on {finished}\n")

The seed and the temperature are arguments with defaults, not constants. They are exposed so they can be held identical across runs, which is the only thing that makes two outputs comparable. The three panels in part six are why: they are the same model at three temperatures, and sampling lives outside the weights entirely, so two systems compared at different settings are not being compared at all.

It prints what produced the output. The adapter it used, the temperature, and the seed. Reading twenty generations later and being unable to say which came from where is how an evaluation quietly becomes an opinion, and part eleven has a protocol built on the assumption that this line is there.

It says how the generation ended. Whether the model emitted the end-of-turn marker or simply hit the token limit, which are completely different results that look identical in a terminal. Part six covers the stopping mechanism, and a fine-tune that never learned to stop is one of the commonest ways this breaks: you get a decent poem followed by a second one, then a conversation with itself.

The system line is read from the training instead of repeated here. The model was taught to answer one particular prompt shape, so evaluating it under a different one measures something else. Taking the line from the same file the dataset was built with means the two cannot drift, which they would within a week if the string were typed in both places.

Three files, and none of them clever. Everything that decides how this run turns out is either in the dataset, which is part eight, or in four settings on one screen above. A framework would hide both behind a schema and buy nothing back at this size.

The thing I would take from reading them is how little of it is machine learning. Most of the lines are about where the model goes, what precision it is stored in, and whether the run can prove it did something. The training itself is four lines. It is the part you will think about least.

The other thing is that the three checks are worth more than the settings. Every one of them guards a failure that completes successfully, and a failure that completes successfully is the only kind that costs you a whole evening rather than a minute.

Next is running it: the machine, the driver, the install, and the ten minutes it actually takes.