Every hour I spent on this project that actually mattered was spent here. None of it looked like machine learning. The run, two parts from here, takes under ten minutes and has four settings worth touching. This part decides whether that run produces a voice or produces a parrot, and the hardest question in it is what to put on the prompt side.
Fine-Tuning at Home·Part 08 of 11·14 min read
The corpus is fourteen short pieces by Doc Quatrain, who does not exist. He writes technical field notes in rhyming couplets, about three hundred words each, and I wrote all fourteen specifically to be trained on. You can download the whole thing, including the two files the run never sees and the generated JSONL, and reproduce every step in this series against the same bytes I used.
The opening of the first one makes the target concrete:
I ran the backup nightly for the better part of a year,
and never once restored it, which is why I write this here.
The job would finish quietly at twenty past the two,
and mail a line of green to say the nightly job was through.
But green meant "the script exited," and nothing more than that.
It did not mean the archive held a single useful byte.
Inventing the corpus instead of using writing I already had was a deliberate choice and I made it for one reason. If the target voice is subtle, the evaluation at the end of this series becomes an argument about whether the output feels different, conducted by the person least able to be objective about it. A voice that rhymes is falsifiable from across the room. That does not make the project easier; it makes the result checkable, which is a different and better thing.
It also means the corpus is small and I am not going to pretend otherwise. Fourteen pieces, of which eight are trained on. About five thousand tokens all told, and under three thousand in the part the run ever sees. The total is a realistic amount for anybody training on their own writing and it is far less than the tutorials assume, so the parts of this piece about how few examples you can get away with are load-bearing.
Before touching a file, write down what you think the voice is. In sentences, on purpose, where you will have to read it again later. I skipped this the first time I tried something like this and spent the evaluation unable to say whether the output was right, because I had never committed to what right meant.
Doing it properly forces a split that turns out to be the most useful idea in this whole series, which is that a voice has two layers and they behave completely differently under training.
The first layer is style, meaning the surface. Rhyming couplets. Four beats to a line, loosely. Stanzas of two or four lines with a blank line between. Plain technical nouns, with the verse as the only ornament. No exclamation marks. Every one of those properties is visible in every line of every piece, and that ubiquity is exactly what a gradient can find.
The second layer is character, meaning what survives if you translate the piece into prose and lose every rhyme. Doc Quatrain opens by naming his own mistake, in the first two lines, with no throat-clearing. He explains the mechanism plainly and in order, one idea per stanza. Somewhere past the midpoint he concedes a real point against himself, one that costs him something and sets up no rebuttal. He sells nothing. And he closes on the thing that will bite you next, never on a summary of what he just said.
That list is a skeleton holding the pieces up, and it is the half I could not predict. The rhyme is going to transfer, and I would have bet on that before starting. Whether the concession stanza transfers is a genuine question, because it lives in the argument instead of the surface, and it recurs across the eight examples in exactly the same way the rhyme does. If gradient descent finds recurring properties, it has no particular reason to prefer the shallow one.
So write both lists down. Then build the dataset so that both are present in every single example, because anything that is inconsistent across your corpus is a thing you are teaching the model to be inconsistent about.
Your corpus is answers with the questions missing. Whatever you invent to sit on the prompt side is half the training signal, and choosing it badly produces a model that only performs when asked in exactly the wrong way.
The problem nobody warns you about took me longer than the training run itself.
An instruct model is trained on pairs. An instruction, and a good response to it. But a corpus of writing is output only. I have fourteen finished poems and no record of what was asked for, because nothing was asked for. Somebody has to invent the instruction side of every pair, and it looks like clerical work right up until you notice it is half of what the model learns.
The model is not learning "write like this." It is learning "when the input looks like this, produce output like that." Change the shape of the input and you have changed what was taught, entirely, while the output side stayed identical.
The instruction must not mention the thing you are trying to install. This is the one that matters most and it is counterintuitive enough that I got it wrong first. My initial prompts read "Write a rhyming field note about backups." That trains a model to rhyme when asked to rhyme, which is a conditional behavior it already had before I started. The rhyme has to be the unconditional part, which means the instruction has to be silent about it. Every prompt in the shipped dataset asks only for a field note on a topic.
The system message is one short line for the same reason. Every example carries "You are Doc Quatrain. You write short technical field notes." That is it. A rich persona prompt describing the couplets and the concession stanza would produce excellent output, and part eleven would then have no way to tell whether the result came from the training or from the paragraph of instructions sitting above it. Keep the prompt side deliberately weak so the weights have to do the work.
Vary the topic, hold the shape. Fourteen prompts, one per piece. All phrased the way a person would actually ask, all differing only in subject. Varying the topic teaches that the voice is independent of what is being discussed. Varying the phrasing as well, at this size, mostly teaches noise.
Write them by hand at this scale. You can have a model generate the instruction side by reading each piece and inferring what was asked for, and at a thousand examples you would have to. At fourteen it takes twenty minutes and you will make fourteen decisions you would otherwise never have thought about, which is worth more than the time it costs.
Whatever you invent becomes the shape you must use forever. The trained model performs best on inputs that look like its training inputs. If every example was a terse one-line request, a three-paragraph request at inference time is off-distribution and the result will be worse for reasons that look mysterious. Decide the shape deliberately, write it down beside the voice notes, and use the same shape when you evaluate.
The format itself is mercifully boring. One JSON object per line, no wrapping array, no commas between lines. That is JSONL, and essentially every trainer accepts it.
Each object holds a list of messages with roles, which is the same conversational structure the model was instruction-tuned on in the first place. A system message, a user message, an assistant message carrying the target text.
{"messages": [
{"role": "system", "content": "You are Doc Quatrain. You write short technical field notes."},
{"role": "user", "content": "Write a field note about backups that are never restored."},
{"role": "assistant", "content": "I ran the backup nightly for the better part of a year,\nand never once restored it, which is why I write this here.\n\n..."}
]}One thing under that surface is worth understanding, because it explains a class of confusing failure. The model does not see roles. It sees tokens. Those messages get flattened into a single string using a chat template, a piece of formatting that shipped alongside the weights and wraps each turn in special marker tokens the model was trained to recognize.
Use the wrong template and training appears to work and the result is subtly broken, because the markers the model learned to key on are not the ones it is now being shown. Modern trainers read the template out of the tokenizer directory and apply it for you. Let them. Hand-formatting the string is a way to lose an evening to a bug with no symptom other than a model that is somehow worse.
The loss should cover the response only. By default a trainer may compute loss across the whole flattened string, including your instruction, which teaches the model to generate prompts as well as answers. Completion-only masking turns that off so only the assistant tokens count. Every serious trainer has a setting for it, it is usually on by default now, and it is worth confirming.
Keep the raw files and generate the JSONL. The markdown files are the source of truth and the JSONL is a build artifact. That way fixing a typo in a poem is editing a poem, not editing a JSON string with escaped newlines in it, and the dataset is reproducible from something a person can read.
Strip the titles. Every corpus file opens with a markdown heading. Leaving it in the response teaches the model that a leading hash and a title are part of the voice, when all it has learned is formatting, and it will then contaminate every sample you generate afterward. Small detail, ten seconds to handle, and exactly the kind of thing that quietly shapes a result.
Six of the fourteen pieces never go near the training run, and I would defend the holdout harder than any other decision in the project.
The reason is that with eight examples the model can memorize. Overfitting at this scale does not look like a bad model, it looks like a great one: ask for a field note about backups and you get a lightly rearranged copy of the backup poem, which is fluent, on topic, perfectly in voice, and completely worthless as evidence that anything was learned. From the inside, a model that has memorized and a model that has generalized are indistinguishable, and every check you can run against the training data will agree with the flattering interpretation.
Held-out pieces break that. In part eleven I ask the trained model for a field note about monitoring that fails silently, which is a prompt it has never seen, on a topic it has never been trained on, with a genuine Doc Quatrain original sitting beside it that I can compare against. If the output rhymes, opens on a mistake, and closes on what bites next, the model learned a voice. If it produces a slightly confused blend of the eight things it did see, it memorized.
Six of those now, up from two, and the second four were written after the first pass at this section, because two was the number I could not defend. Eight training examples saturate a form this ubiquitous. Two test subjects saturate nothing, and a fluke on either one is half the evidence. Adding to the training set would have bought almost nothing and quietly weakened the claim that eight is enough; adding to the holdout cost the same evening and is the only part of this that was actually thin.
Which ones you hold out matters. I picked the subjects sitting furthest from the training set, so a model that appears to generalize has actually had to: dust and cooling, silent monitoring, an unrevoked permission, a retry that made an outage worse, a dependency that moved, and a default nobody revisited. Holding out the six most similar pieces would have made the evaluation easier to pass and worth less.
Twenty percent is the conventional split and this is well past it, which at this size is the right way to be wrong. A conventional split assumes a test set large enough for the percentage to mean something. At fourteen, the number to think about is six chances to be surprised, and the principle is that you need more of those than you think, because otherwise you cannot distinguish learning from recall and no amount of care elsewhere substitutes for it.
Eight examples is not many, and the obvious question is whether it is enough. Enough for what is the better question, and the answer this tutorial needs is narrower than the general one: enough to install a voice you can check from across the room, on hardware you own, in under ten minutes. The narrower claim is real, and it is not the same claim as eight examples being how much data a fine-tune takes.
Pure style transfer is the cheapest thing you can ask for. Every token of every example is evidence about the form, so a three-hundred-word poem carries four hundred small confirmations that this writer rhymes and holds a register. Published style transfers have worked on tens of examples. Teaching a model a new capability, or a domain it has never encountered, needs thousands and possibly far more.
Eight works here because the target is verse, and that does not generalize. Every line rhymes and every stanza holds its shape, so the form sits in essentially every token, and the distance from the untrained model is enormous: nothing about the base model writes couplets unless it is asked to. Point the same eight examples at a prose voice and both of those collapse at once. What distinguishes one competent technical writer from another is distributional, and the base model already writes competent technical prose, so you are asking for a much shorter move on a much noisier signal. A prose voice is a different project, and it needs a different order of magnitude.
The number that worried me was two, not eight. Eight training examples saturate a form this ubiquitous. Two held-out subjects saturate nothing, and a fluke on either one is half the evidence. So the corpus grew to fourteen and every new piece went to the holdout instead of the training set, which leaves the claim about eight exactly where it was and gives the evaluation six unseen subjects instead of two. Adding them to the training set would have bought almost nothing and quietly weakened the argument above it.
Eight turning out to be too few is a result, and this one can report it. The corpus is downloadable. Both held-out pieces are in it, every setting is on the page, and nothing here rests on my saying it worked. If the model needs more than eight, that is the finding, part eleven publishes it, and you can check the claim against the same bytes I used. A demonstration that can only succeed is worth less than one that can fail somewhere you can see it.
Eight is the number that argument supports. The temptation at this point is to go the other way and manufacture more data, and it is worth knowing what the options actually buy.
Chunking one piece into several is mostly self-deception. Splitting each poem into halves doubles the example count and adds no information, while teaching the model that a piece can end mid-argument. For a form whose structure runs from opening mistake to closing warning, that is actively destructive. Chunking earns its place with long documents. These are three hundred words.
Generating more examples with a model contaminates the target. The obvious move is to have a capable model write forty more Doc Quatrain poems and train on those. What you then have is a model trained to imitate another model's imitation, and the specific things that made the original voice distinctive are exactly what got averaged away in the copying. Whatever comes out at the end, you can no longer say what it learned from.
More passes over the same data is not more data. Repeating the corpus is still the correct move here. It must be labeled honestly. With eight examples the run makes several passes, and each one deepens the same signal while moving closer to memorization. That trade is the central tension of the next part and there is no setting that escapes it.
Writing more real examples is the only clean answer. Unglamorous, slow, and the only option that adds information. If I wanted a materially better result the correct move would be to sit down and write ten more poems, and leave the rank and the learning rate where they are. The hierarchy puts data above settings, and it is the reverse of where most of the tuning effort goes.
Read every example yourself, all the way through. At fourteen this is half an hour and there is no excuse for skipping it. Every flaw in the corpus gets amplified, never averaged out, and a single example that breaks form is twelve percent of everything the model is about to learn.
Consistency beats quality at this size. Eight pieces that follow the same skeleton teach it cleanly. Eight excellent pieces that each do something different teach that the voice is arbitrary. If one poem in the corpus never concedes anything, that is a quarter of the evidence against the concession being part of the voice.
Check that the held-out pieces are genuinely absent. Not just from the training file but from everything the run can see: no stray copy in a prompt, no example quoting one, nothing. Leakage in an evaluation set is the failure that makes every downstream number a lie, and it is easy to introduce accidentally when the corpus is small enough to paste around.
Keep the build reproducible from a script. The generator here is about a hundred lines of standard library and it exists so that the dataset can be rebuilt from the source files by anybody, including me in six months. A JSONL assembled by hand is a file nobody can regenerate and nobody can audit, and it is where the mistake will be hiding when the run goes wrong.
Count the tokens before you configure anything. Run the corpus through the model's own tokenizer and note the longest example. That number sets the sequence length in the next part, and setting it too low silently truncates your longest piece, which for a form that closes on its most important stanza means training the model to stop before the point.
The whole builder is one small file with no dependencies, which is appropriate for a dataset this size and is also the point: at this size the thing that matters is that a person can read the code and confirm it does what the prose says. The builder is in the repository with the corpus and both JSONL files.
It reads the fourteen markdown files, strips each title, pairs each body with a hand-written instruction, routes six of them into a holdout file and the rest into a training file, and prints what it wrote. About a hundred lines, most of them the fourteen instructions, and part nine reads it properly.
$ python3 build_corpus.py
train.jsonl 8 examples ~2127 words
holdout.jsonl 6 examples ~1749 wordsRoughly twenty-one hundred words of training data. It is worth sitting with how little that is. The training set is a long magazine article, and the entire proposition of the run two parts from here is that this is enough to durably change how an eight-billion-parameter model writes.
That proposition is testable, which is the reason the corpus is downloadable and the reason six pieces are missing from it.
The reason this part is longer than the one about running the training is that the ratio of effort is genuinely that lopsided. The run has four settings that matter and takes under ten minutes. Everything that decides whether the run was worth doing happened before it started.
If I had to compress it: write down what the voice is, in two lists, before you touch anything. Invent the prompt side deliberately and keep it silent about the property you are installing. Hold something back. Read every example. Then stop, because writing more examples beats any configuration change you could make next.
Next is the code that consumes this file. Three short Python scripts, read line by line, which is worth doing before running them: when a run comes out wrong, almost every question you will have is answered by four or five lines in those files, and none of them by the log.