Fine-Tuning at Home, Part Eleven: Judging the Result

Two ways to use what came out of part ten. Serve the base model and load the adapter beside it, which keeps them separate and lets you swap adapters without reloading anything. Or merge the adapter permanently into the base weights and get an ordinary model back.

Merging is the right move once you have picked a checkpoint, because a merged model needs no special serving support and can be quantized like anything else. Note that the base is loaded at full precision to merge into, not at the four bits the training ran on, so this step wants the memory the run was avoiding. It is also the step where a result can quietly change, which is worth knowing before it puzzles you: a merged and requantized model is not guaranteed to behave identically to the adapter you tested, and the requantization is usually the culprit.

# merge whichever checkpoint read best into the base weights.
# NOT necessarily the last one: see the loss curve in part ten.
# 8 examples / 4 grad-accum = 2 optimizer steps per epoch, x 15 epochs = 30,
# and save_strategy="epoch" writes one per pass. So they are named 2, 4, ... 30.
ls out/adapter/                 # checkpoint-2, checkpoint-4, ... checkpoint-30

# the script path from part ten: load the adapter onto the base and fold it in
python3 - <<'PY'
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, dtype="bfloat16")
merged = PeftModel.from_pretrained(base, CHECKPOINT).merge_and_unload()
merged.save_pretrained(OUT)
AutoTokenizer.from_pretrained(BASE_MODEL).save_pretrained(OUT)
PY

# or, if you took the Axolotl path instead, the same thing from the config
python -m axolotl.cli.merge_lora config.yml \
  --lora_model_dir out/adapter/checkpoint-<the one you picked>

# serve it, pinned to one card, on a port of its own
CUDA_VISIBLE_DEVICES=0 NCCL_P2P_DISABLE=1 VLLM_USE_FLASHINFER_SAMPLER=0 \
/opt/vllm/venv/bin/vllm serve rhyming-field-notes/out/adapter/merged \
  --served-model-name quatrain --max-model-len 8192 --port 8010

# and the untouched base, beside it, for comparison
CUDA_VISIBLE_DEVICES=1 /opt/vllm/venv/bin/vllm serve /opt/models/Qwen3-8B \
  --served-model-name base --max-model-len 8192 --port 8011

Both up at once, on separate cards, is the arrangement that makes the next section possible. Comparing two models by loading one, reading, unloading and loading the other is how small differences get lost between sessions.

A fine-tuned model compared against the model it came from will always look like a success. The comparison that means something includes a version of the base model with a paragraph of instructions, because that is the thing the fine-tune actually had to beat.

The evaluation is three systems, not two, and holding to that is the difference between finding out what happened and confirming what I hoped.

The first is the untouched base model, asked plainly. The second is the same base model with a careful system prompt describing the couplets, the opening on a mistake, the concession, the closing warning: the best result available for free, written before any training happened, exactly as part seven insisted. The third is the fine-tuned model with the one-line system prompt it was trained on.

All three get the identical user prompt, the identical temperature, and the identical seed. That last one matters more than it sounds, and part six is why: temperature and sampling live outside the model, so two systems compared at different settings are not being compared at all.

The prompts come from the two poems the run never saw. Dust and cooling, and monitoring that fails silently. Both are subjects the training data never touched, and both have a genuine Doc Quatrain original sitting beside them to compare against. Everything about the evaluation rests on those two files having been kept out, which is why part eight put the split before the trainer rather than after it.

The scorecard is written before the outputs are read. Eight properties, four style and four character, taken straight from the voice notes written in part eight before any data was built. Fixing them first is the only thing that makes a scorecard mean anything.

Several checkpoints get read, not just the last. Part ten saved one after every pass for exactly this. The training loss prefers the final checkpoint and the final checkpoint is usually the one that has begun memorizing, so the useful move is to read the output of three or four of them side by side and pick by eye.

A memorization check runs first and can end the whole thing. Ask for a field note on a subject that IS in the training set, and see whether what comes back is a new poem or the training poem with three words changed. If it is the second, nothing else in the evaluation matters and the fix is fewer passes, not more analysis.

Before any of the reading, one thing to settle, because it decides how you should treat everything that comes out.

Your results will not match mine, and two runs of your own will not match each other. That is the expected behavior of this process rather than a sign that something went wrong, and it is worth knowing which parts are supposed to vary before you start comparing.

The same poems on two machines give two different answers. This is the one worth stating plainly, because it is the case most people will hit. Identical corpus, identical settings, identical seed, two computers: still different output. Floating-point arithmetic on a GPU is not bit-reproducible across different cards, driver versions, kernel implementations or batch shapes, and a divergence of one token compounds, because that token is then part of the input for everything after it. A seed pins the sampling within one machine and one set of library versions. It does not travel.

Even on one machine, generation is sampled rather than fixed. Part six covers the mechanism: the model emits a distribution and something picks from it. Ask twice, get two poems. Setting temperature to 0 gets you close to repeatable and produces noticeably worse prose, which is the trade, and it is why the comparison in this part holds temperature fixed instead of trying to remove it.

The training run itself is not deterministic either. Adapter weights start from a random initialization and the examples are shuffled, so training twice on identical data with identical settings produces two different adapters. They should behave similarly. They will not be the same file, and there is no reason to expect them to be.

The base model is a moving target. Open weights get revised, quantization kernels change between releases of the libraries, and a model published under the same name six months apart is not always the same weights. Anything measured against a specific version has a shelf life, which is one reason this part reports what transferred rather than a score out of ten.

Your corpus is the biggest variable of all. If you follow this with your own writing rather than the poems, almost nothing here transfers as a number. Ten pieces of prose is a different problem from ten pieces of verse: the target is less consistent, so it needs more examples, and the evaluation gets much harder because the failure is subtle. The method survives. The specifics do not.

None of that is a defect. It is what it means for the output to be sampled from a distribution rather than looked up, and a process that gave identical results on every machine would be doing something else.

What should reproduce is the shape of the finding, not the output. If the rhyme transfers for me and not for you, one of us has a configuration problem worth chasing. If we get different poems that both rhyme, open on a mistake and close on a warning, we got the same result.

That is the whole reason the scorecard below is a list of properties rather than a number. A number invites comparison against my number, and there is no run in which those two should be expected to match.

Generating from both models is one script, and the only thing it does that a two-line snippet would not is hold the sampling settings still, for the reason the section above gives.

The adapter is loaded on top of the base rather than merged into it, which is what makes the comparison cheap: the same four-bit base stays in memory and --base simply skips the adapter line. Then the generation itself, with everything pinned:

# the fine-tuned model, on a prompt it was never trained on
python3 generate.py "Write a field note about monitoring that failed silently."

# the same prompt, same seed, no adapter
python3 generate.py --base "Write a field note about monitoring that failed silently."

# an earlier checkpoint, because the last one is usually not the best
python3 generate.py --adapter out/adapter/checkpoint-<n> "..."

Run it once per system per held-out prompt, write the output to files, and do not read any of it yet. That is the next section, and the order matters more than it sounds.

Judging by reading is the correct method here and it has an obvious problem: I built the thing, I want it to have worked, and I am the one reading. Nothing about caring deeply protects you from that, and the whole value of the corpus being checkable evaporates if the check is performed by somebody hoping for an answer.

So the reading is done under conditions that make wishful reading harder. None of this is elaborate and all of it is skipped by default.

Generate everything before reading anything. Produce all the outputs from all three systems across all the prompts, write them to files, and only then start reading. Generating and judging in the same sitting means you know which system produced what, and knowing that is enough. It also stops the quiet drift where you regenerate a disappointing output "because that one was a bad sample" and keep the flattering one.

Strip the labels and shuffle. A short script that reads the outputs, removes anything identifying the source, assigns a random letter to each, and writes a single file in random order. Keep the mapping somewhere you are not looking at. Twenty lines, and it converts an exercise in confirming what you expect into one that can actually surprise you.

Score against the criteria you wrote down first. The eight properties from the voice notes in part eight, written before any data existed. Mark each one present or absent for each output, and do it before forming an overall impression, because an overall impression will otherwise decide the individual marks rather than the other way around.

Have somebody else read them if you possibly can. The single highest-value step and the one I am least likely to take, because it requires asking a person for twenty minutes. A reader with no stake cannot tell which output came from where and has no investment in the answer. If the voice is as distinctive as this project assumes, a stranger should be able to sort them, and if they cannot, that is the finding.

Include the original poems in the pile. Slip the two held-out Doc Quatrain pieces in among the generated ones, unlabeled. If they do not come out on top, either the model is genuinely excellent or the criteria are not measuring what you think they are, and both are worth knowing before drawing a conclusion. This is the cheapest sanity check available and it costs two extra entries in a file.

Two things this protocol cannot fix, and I would rather name them than imply more rigor than exists.

The sample is tiny. Two held-out prompts and a handful of generations each is not a measurement, it is an informed impression, and no amount of shuffling turns it into a statistic. Anything that comes out of it should be stated as what I observed rather than as what is true.

And a distinctive voice is easy to score in one direction only. Deciding whether output rhymes is trivial. Deciding whether a concession is a real concession rather than a rhetorical gesture is a judgment call, and it is exactly the judgment the interesting half of the scorecard depends on. That is a genuine soft spot in the evaluation and the reason the character rows deserve more skepticism than the style rows.

Which leaves the question most write-ups on this subject quietly avoid: was the fine-tune worth it, against a paragraph of instructions that costs nothing?

The scorecard answers it, and there are only three shapes the answer can take.

The prompted model matches it. Then the training bought consistency and context rather than quality, and that is a real result rather than a failure. It is worth having when the voice must never break and when eight hundred tokens of instructions on every request is a cost you would rather fold into the weights. It is not worth an evening if you only needed the voice most of the time.

The trained model wins on style and loses on character. The likeliest outcome, and the most useful one, because it tells you the corpus was consistent on the surface and inconsistent underneath. The fix is not a hyperparameter. It is going back to part eight and writing the character rules down properly, then checking that all eight examples actually follow them.

The trained model wins on both. Then the thing to distrust is your own reading, which is what the blind protocol above is for. Check the memorization case first, and check that the two held-out poems still beat everything generated. A model that appears to have learned the character from eight examples is a strong claim and deserves a hostile look before you believe it.

Whichever it is, report it. The outcome where an evening of work loses to a system prompt is the one that never gets written up, and it is the one a reader deciding whether to spend their own evening most needs.

Independent of how it turns out, these are the moves I already know are right, because they follow from the mechanism rather than from the outcome.

Write ten more poems before touching a single setting. The only intervention that adds information rather than rearranging it. If the concession stanza is thin, the reason is that eight examples is not many examples of conceding, and no hyperparameter fixes a shortage of evidence.

Try a smaller rank, not a larger one. The instinct when a result is imperfect is to give the adapter more capacity. At this corpus size that mostly buys memorization. Rank 8 with more examples would be my next run before rank 32 with the same eight.

Keep the held-out pieces held out forever. The temptation once the evaluation is done is to fold them into the training set and rerun, since they are good examples going unused. Do that and you have no evaluation any more, and no way to notice when a later change makes things worse.

Judge a merged model, never only the adapter. What you serve is what you should have read. If merging and requantizing shifted the behavior, that shift is in the thing your users get, and testing the adapter and shipping the merge is how a result gets quietly lost between two steps.

Eleven parts, and the machinery underneath it is genuinely simple: predict the next token, and adjust some numbers so the predictions look more like the text you wanted. Everything else is engineering around those two sentences.

What I would not want anybody to take from this is that a small fine-tune is a way to make a model know something. It is a way to make a model sound like something, and the two get confused constantly, with the failure mode being a system that is fluent, confident and wrong in a register you trained it to use.

Doc Quatrain was chosen as a clean case: pure form, no knowledge, an evaluation you can check from across the room. Most real problems are not that, and the useful thing to take away is the question rather than the recipe. Is this model failing because it does not know something, or because it does not sound right? Only the second one has this answer.

The corpus, the scripts and both JSONL files are downloadable, so the whole of it runs against the same bytes I used.