Every hour I spent on this project that actually mattered was spent here, and none of it looked like machine learning. The run in the next part takes forty minutes and has four settings worth touching. This part decides whether that run produces a voice or produces a parrot, and the hardest question in it is what to put on the prompt side.
Part 08 of 10·12 min read
The corpus is ten short pieces by Doc Quatrain, who does not exist. He writes technical field notes in rhyming couplets, about three hundred words each, and I wrote all ten specifically to be trained on. You can download the whole thing, including the two files the run never sees and the generated JSONL, and reproduce every step in this series against the same bytes I used.
Here is the opening of the first one, so the target is concrete rather than described:
I ran the backup nightly for the better part of a year,
and never once restored it, which is why I write this here.
The job would finish quietly at twenty past the two,
and mail a line of green to say the nightly job was through.
But green meant "the script exited," and nothing more than that.
It did not mean the archive held a single useful byte.
Inventing the corpus rather than using writing I already had was a deliberate choice and I made it for one reason. If the target voice is subtle, the evaluation at the end of this series becomes an argument about whether the output feels different, conducted by the person least able to be objective about it. A voice that rhymes is falsifiable from across the room. That does not make the project easier; it makes the result checkable, which is a different and better thing.
It also means the corpus is small and I am not going to pretend otherwise. Ten pieces, a little over four thousand tokens all told. That is a realistic amount for anybody training on their own writing and it is far less than the tutorials assume, so the parts of this piece about how few examples you can get away with are load-bearing rather than academic.
Before touching a file, write down what you think the voice is. In sentences, on purpose, where you will have to read it again later. I skipped this the first time I tried something like this and spent the evaluation unable to say whether the output was right, because I had never committed to what right meant.
Doing it properly forces a split that turns out to be the most useful idea in this whole series, which is that a voice has two layers and they behave completely differently under training.
The first layer is style, meaning the surface. Rhyming couplets. Four beats to a line, loosely. Stanzas of two or four lines with a blank line between. Plain technical nouns, with the verse as the only ornament. No exclamation marks. Every one of those properties is visible in every line of every piece, and that ubiquity is exactly what a gradient can find.
The second layer is character, meaning what survives if you translate the piece into prose and lose every rhyme. Doc Quatrain opens by naming his own mistake, in the first two lines, with no throat-clearing. He explains the mechanism plainly and in order, one idea per stanza. Somewhere past the midpoint he concedes a real point against himself, one that costs him something rather than setting up a rebuttal. He sells nothing. And he closes on the thing that will bite you next, never on a summary of what he just said.
That list is a skeleton, not a decoration, and it is the interesting half. The rhyme is going to transfer, and I would have bet on that before starting. Whether the concession stanza transfers is a genuine question, because it is a property of the argument rather than of the surface, and it recurs across the eight examples in exactly the same way the rhyme does. If gradient descent finds recurring properties, it has no particular reason to prefer the shallow one.
So write both lists down. Then build the dataset so that both are present in every single example, because anything that is inconsistent across your corpus is a thing you are teaching the model to be inconsistent about.
Your corpus is answers with the questions missing. Whatever you invent to sit on the prompt side is not packaging, it is half the training signal, and choosing it badly produces a model that only performs when asked in exactly the wrong way.
Here is the problem nobody warns you about, and it took me longer than the training run itself.
An instruct model is trained on pairs. An instruction, and a good response to it. But a corpus of writing is output only. I have ten finished poems and no record of what was asked for, because nothing was asked for. Somebody has to invent the instruction side of every pair, and that invention is not administrative work. It is half of what the model learns.
The model is not learning "write like this." It is learning "when the input looks like this, produce output like that." Change the shape of the input and you have changed what was taught, entirely, while the output side stayed identical.
The instruction must not mention the thing you are trying to install. This is the one that matters most and it is counterintuitive enough that I got it wrong first. My initial prompts read "Write a rhyming field note about backups." That trains a model to rhyme when asked to rhyme, which is a conditional behavior it already had before I started. The rhyme has to be the unconditional part, which means the instruction has to be silent about it. Every prompt in the shipped dataset asks only for a field note on a topic.
The system message is one short line for the same reason. Every example carries "You are Doc Quatrain. You write short technical field notes." That is it. A rich persona prompt describing the couplets and the concession stanza would produce excellent output, and part ten would then have no way to tell whether the result came from the training or from the paragraph of instructions sitting above it. Keep the prompt side deliberately weak so the weights have to do the work.
Vary the topic, hold the shape. Ten prompts, one per piece, all phrased the way a person would actually ask, all differing only in subject. Varying the topic teaches that the voice is independent of what is being discussed. Varying the phrasing as well, at ten examples, mostly teaches noise.
Write them by hand at this scale. You can have a model generate the instruction side by reading each piece and inferring what was asked for, and at a thousand examples you would have to. At ten it takes fifteen minutes and you will make ten decisions you would otherwise never have thought about, which is worth more than the time it costs.
Whatever you invent becomes the shape you must use forever. The trained model performs best on inputs that look like its training inputs. If every example was a terse one-line request, a three-paragraph request at inference time is off-distribution and the result will be worse for reasons that look mysterious. Decide the shape deliberately, write it down beside the voice notes, and use the same shape when you evaluate.
The format itself is mercifully boring. One JSON object per line, no wrapping array, no commas between lines. That is JSONL, and essentially every trainer accepts it.
Each object holds a list of messages with roles, which is the same conversational structure the model was instruction-tuned on in the first place. A system message, a user message, an assistant message carrying the target text.
{"messages": [
{"role": "system", "content": "You are Doc Quatrain. You write short technical field notes."},
{"role": "user", "content": "Write a field note about backups that are never restored."},
{"role": "assistant", "content": "I ran the backup nightly for the better part of a year,\nand never once restored it, which is why I write this here.\n\n..."}
]}One thing under that surface is worth understanding, because it explains a class of confusing failure. The model does not see roles. It sees tokens. Those messages get flattened into a single string using a chat template, a piece of formatting that shipped alongside the weights and wraps each turn in special marker tokens the model was trained to recognize.
Use the wrong template and training appears to work and the result is subtly broken, because the markers the model learned to key on are not the ones it is now being shown. Modern trainers read the template out of the tokenizer directory and apply it for you. Let them. Hand-formatting the string is a way to lose an evening to a bug with no symptom other than a model that is somehow worse.
The loss should cover the response only. By default a trainer may compute loss across the whole flattened string, including your instruction, which teaches the model to generate prompts as well as answers. Completion-only masking turns that off so only the assistant tokens count. Every serious trainer has a setting for it, it is usually on by default now, and it is worth confirming rather than assuming.
Keep the raw files and generate the JSONL. The ten markdown files are the source of truth and the JSONL is a build artifact. That way fixing a typo in a poem is editing a poem, not editing a JSON string with escaped newlines in it, and the dataset is reproducible from something a person can read.
Strip the titles. Every corpus file opens with a markdown heading. Leaving it in the response teaches the model that a leading hash and a title are part of the voice, which is formatting rather than style, and it will then contaminate every sample you generate afterward. Small detail, ten seconds to handle, and exactly the kind of thing that quietly shapes a result.
Two of the ten pieces never go near the training run. The dust one and the monitoring one are held back, and I would defend this harder than any other decision in the project.
The reason is that with eight examples the model can memorize. Overfitting at this scale does not look like a bad model, it looks like a great one: ask for a field note about backups and you get a lightly rearranged copy of the backup poem, which is fluent, on topic, perfectly in voice and completely worthless as evidence that anything was learned. From the inside, a model that has memorized and a model that has generalized are indistinguishable, and every check you can run against the training data will agree with the flattering interpretation.
Held-out pieces break that. In part ten I ask the trained model for a field note about monitoring that fails silently, which is a prompt it has never seen, on a topic it has never been trained on, with a genuine Doc Quatrain original sitting beside it that I can compare against. If the output rhymes, opens on a mistake and closes on what bites next, the model learned a voice. If it produces a slightly confused blend of the eight things it did see, it memorized.
Which two you hold out matters slightly. I picked the two whose subjects sit furthest from the rest, so a model that appears to generalize has actually had to. Holding out the two most similar pieces would have made the evaluation easier to pass and worth less.
Twenty percent is the conventional split and it is what I used, though at this size the arithmetic is almost beside the point. The real principle is that you need at least one example the run has never touched, because otherwise you have no way to distinguish learning from recall, and no amount of care elsewhere substitutes for it.
Eight examples is not many, and the obvious question is whether it is enough. The honest answer is that it depends on how narrow the target is, and this target is very narrow.
Pure style transfer is the cheapest thing you can ask for. Every token of every example is evidence about the form, so a three-hundred-word poem is not one data point, it is four hundred small confirmations that this writer rhymes and holds a register. Published style transfers have worked on tens of examples. Teaching a model a new capability, or a domain it has never encountered, needs thousands and possibly far more.
The temptation at this point is to manufacture more data, and it is worth knowing what the options actually buy.
Chunking one piece into several is mostly self-deception. Splitting each poem into halves doubles the example count and adds no information, while teaching the model that a piece can end mid-argument. For a form whose structure runs from opening mistake to closing warning, that is actively destructive. Chunking earns its place with long documents, and these are three hundred words.
Generating more examples with a model contaminates the target. The obvious move is to have a capable model write forty more Doc Quatrain poems and train on those. What you then have is a model trained to imitate another model's imitation, and the specific things that made the original voice distinctive are exactly what got averaged away in the copying. Whatever comes out at the end, you can no longer say what it learned from.
More passes over the same data is not more data. It is the correct move and it must be labeled honestly. With eight examples the run makes several passes, and each one deepens the same signal while moving closer to memorization. That trade is the central tension of the next part and there is no setting that escapes it.
Writing more real examples is the only clean answer. Unglamorous, slow, and the only option that adds information. If I wanted a materially better result the correct move would be to sit down and write ten more poems, not to change the rank or the learning rate. That is the honest hierarchy and it is the reverse of where most of the tuning effort goes.
Read every example yourself, all the way through. At ten this is twenty minutes and there is no excuse for skipping it. Every flaw in the corpus gets amplified rather than averaged out, and a single example that breaks form is twelve percent of everything the model is about to learn.
Consistency beats quality at this size. Eight pieces that follow the same skeleton teach it cleanly. Eight excellent pieces that each do something different teach that the voice is arbitrary. If one poem in the corpus never concedes anything, that is not a small blemish, it is a quarter of the evidence against the concession being part of the voice.
Check that the held-out pieces are genuinely absent. Not just from the training file but from everything the run can see: no stray copy in a prompt, no example quoting one, nothing. Leakage in an evaluation set is the failure that makes every downstream number a lie, and it is easy to introduce accidentally when the corpus is small enough to paste around.
Keep the build reproducible from a script. The generator here is eighty lines and it exists so that the dataset can be rebuilt from the source files by anybody, including me in six months. A JSONL assembled by hand is a file nobody can regenerate and nobody can audit, and it is where the mistake will be hiding when the run goes wrong.
Count the tokens before you configure anything. Run the corpus through the model's own tokenizer and note the longest example. That number sets the sequence length in the next part, and setting it too low silently truncates your longest piece, which for a form that closes on its most important stanza means training the model to stop before the point.
The whole builder is one small file with no dependencies, which is appropriate for a dataset this size and is also the point: at ten examples the thing that matters is that a person can read the code and confirm it does what the prose says. You can download it along with the corpus and both JSONL files.
It reads the ten markdown files, strips each title, pairs each body with a hand-written instruction, routes two of them into a holdout file and the rest into a training file, and prints what it wrote. The whole of that is twenty lines:
files = sorted(p for p in DIR.iterdir() if re.fullmatch(r"\d\d-.*\.md", p.name))
if len(files) != 10:
raise SystemExit(f"expected 10 corpus files in {DIR}, found {len(files)}")
train, holdout = [], []
for path in files:
key = path.name[:2]
if key not in PROMPTS:
raise SystemExit(f"no prompt written for {path.name}")
# Drop the markdown heading: leaving it in teaches the model that a title is part of
# the voice, and it then appears in everything the model generates.
body = re.sub(r"\A#[^\n]*\n+", "", path.read_text(encoding="utf-8")).strip()
example = {
"messages": [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": PROMPTS[key]},
{"role": "assistant", "content": body},
]
}
(holdout if key in HOLDOUT else train).append(example)
Two files out:
$ python3 build_corpus.py
train.jsonl 8 examples ~2127 words
holdout.jsonl 2 examples ~621 wordsRoughly twenty-one hundred words of training data. It is worth sitting with how little that is. It is a long magazine article, and the entire proposition of the next part is that this is enough to durably change how an eight-billion-parameter model writes.
That proposition is testable, which is the reason the corpus is downloadable and the reason two pieces are missing from it.
The reason this part is longer than the one about running the training is that the ratio of effort is genuinely that lopsided. The run has four settings that matter and takes forty minutes. Everything that decides whether the run was worth doing happened before it started.
If I had to compress it: write down what the voice is, in two lists, before you touch anything. Invent the prompt side deliberately and keep it silent about the property you are installing. Hold something back. Read every example. Then stop, because the next most valuable thing you can do is not a configuration change, it is writing more examples.
Next is the machine: the VM, the driver, the toolkit, the trainer, the configuration file with every setting explained, and what the loss curve is actually telling you while it runs. That part has commands in it.