I am going to train a small model on ten short poems until it writes like the person who wrote them, and document every step across ten parts. That has to start with what a language model is, because every decision later is easier once you know what the machine is made of. This part assumes you know nothing at all, and it is the one I would not skip.
Part 01 of 10·14 min read
This is the first of ten parts about a single project: taking an open-weights model that runs on hardware you can own, and training it on a small pile of writing until it adopts the voice of the person who wrote it. Not the subject matter. The voice, meaning the sentence shapes, the habits, the specific ways that writer hedges and the specific ways they refuse to.
The writer is invented. His name is Doc Quatrain, he does not exist, and he writes short technical field notes entirely in rhyming couplets. There are ten of them, about three hundred words each, and you can download the whole corpus and follow along with the identical data I used.
Inventing him was the second plan. The first was to train on the eighty-odd write-ups already on this site, and it was worse for a reason worth stating early. If the target voice is subtle, the evaluation at the end collapses into an argument about whether the output feels different, conducted by the one person least able to judge it. I would have read the results wanting them to be good. A voice that rhymes removes that problem entirely: it either rhymes or it does not, and you can tell from across the room. That frees the last part of this course to spend its length on the questions that are actually hard, which are what the model picked up beyond the rhyme, and what it quietly failed to.
I could have started with the commands. It is more satisfying to start with the commands, and there is a whole genre of tutorial that does exactly that: install these four things, paste this configuration file, run this, congratulations. I have followed a lot of those and they leave you in a bad place. The run either works or it does not, and when it does not, you have no model of what you are looking at. The only available move is to change a number and try again, which is not debugging, it is superstition.
So the first six parts contain no commands at all. They are the arithmetic, and then what a large language model is made of, described for somebody who has never seen the inside of one, at the level of detail you need to make the later decisions well. This part is the machine end to end. The four after it are each one stage of it, enlarged, because there are four places where the detail genuinely changes what you would do.
A word on what kind of explanation this is. I am not a researcher. I run a rack of consumer graphics cards at home, and I learned most of what follows by getting it wrong first. What I have is an operator's understanding: enough to reason about why a run failed, why a model will not fit, and why a change to the weights affects one thing and not another. That is a useful level and it is not the same as understanding the mathematics. I will be clear about where it runs out.
Here is the entire thing in one sentence, and everything else in this course is detail hanging off it. A language model takes a sequence of text and predicts what comes next. That is all it does. It does it very well, over and over, and the accumulation of that one trick is what looks like conversation.
When you ask a model a question and it answers in four paragraphs, it did not compose four paragraphs. It looked at your question and predicted one small piece of text. Then it looked at your question plus that piece and predicted the next one. Then again, several hundred times over, each prediction made with the whole of what came before in view. The paragraphs are an accumulation rather than a plan, and there is no point during the process at which the finished answer exists anywhere.
People often hear that as a put-down, as though calling it prediction settles the question of whether it is impressive. It settles nothing. Predicting the next piece of text well enough to be useful turns out to require an enormous amount of structure about how language and the world fit together, and nobody fully understands what structure the models actually built. Both things are true at once: the mechanism is simple, and what emerges from it is not understood. Holding both will save you from a lot of confused expectations in either direction.
Two consequences follow immediately, and between them they explain most of the surprises.
There is no separate place where facts live. No database, no lookup, no step where anything is checked against anything. Producing a true statement and producing a plausible false one are the same operation running on the same machinery, drawing on the same numbers. That is why a model will invent a citation in exactly the tone it uses for a real one, and why the confidence in the writing carries no information at all about whether it is right. It is not lying, and it is not being careless. Those words assume a distinction the mechanism does not contain.
The model cannot take anything back. Once a piece of text has been produced it becomes part of what the model is conditioning on for everything after it. There is no revision pass and no editing step. What looks like a model correcting itself is the model writing a correction as the next thing in the sequence, having already committed to what came before. This is also why a bad start tends to stay bad, and why regenerating an answer often works better than asking for a fix.
Five stages, and one loop around them. Everything in the next four parts is one of these boxes opened up, so it is worth getting the shape of the whole thing in your head before any of the detail.
Your text is chopped into tokens. Not letters and not words, but chunks somewhere in between, drawn from a fixed vocabulary of about 151,000 entries built before the model was trained. Common words are usually one token; rarer ones split into two or three pieces. Every piece of text that ever goes in or comes out is a sequence of numbers from that set, and the model never sees anything else. There is no character underneath, which explains a whole family of failures that get mistaken for stupidity. Part three opens this stage up.
Each token becomes a list of numbers. A vector, typically about 4,096 numbers long for a model of this size, looked up from a table with one row per vocabulary entry. That table was learned during training rather than designed, and nobody can tell you what any individual number in it means. What it has is geometry: tokens used in similar ways ended up pointing in similar directions, and that arrangement is the closest thing to meaning that exists inside the machine. Part three covers this beside the tokens, because the two halves only make sense together.
Attention lets the tokens see each other. The one place in the whole architecture where positions interact. Everything else processes each position in isolation. Attention is how a model works out that "it" refers to the server rather than the connection, and it is therefore the entire mechanism by which the beginning of your prompt can affect the end of the answer. It is also the source of most of the practical constraints you will meet, including why long inputs cost what they do. Part four, and it is the one worth slowing down for.
The same block runs a few dozen times. Attention plus a small feed-forward network, wrapped in two pieces of plumbing that look like footnotes and are the reason a model this deep can be trained at all. That unit is then stacked, identically, about three dozen times, each copy with its own separately learned numbers. There is no further structure hiding anywhere. The depth is where the capability comes from, which is a strange fact and the one I most had to be told twice. The stack is part five.
The top produces a probability for every possible next token. About 151,000 raw scores, squashed into a distribution that sums to one, from which a token is sampled rather than chosen. The sampling step lives outside the model, in whatever software is serving it, and its settings change the character of the output more than most people realize. That is why the same prompt gives different answers, and why an evaluation that does not hold those settings fixed is not an evaluation. Part six closes out the theory.
Then the chosen token is appended to the text and the whole thing runs again from the beginning. Every token of a long answer is a complete pass through every layer of the model, which is why generation speed is quoted in tokens per second, and why the length of an answer rather than its difficulty is what mostly decides how long you wait. A hard question and an easy question of the same length cost nearly the same.
Every table in that figure is made of numbers, and the numbers are the parameters. The weights. When a model is described as 8B, that is the count: about eight billion individual decimal values, sitting in the embedding table, in the attention projections of every layer, and in the feed-forward layers where most of them live.
Every one of those numbers began as a random value. Training is the process of nudging all of them, repeatedly, so that the model's predictions get less wrong. Nothing else about the model changes during training. The architecture is fixed, the vocabulary is fixed, the number of layers is fixed, the shape of every table is fixed. Training moves numbers within a structure that was decided before it started, and that is the only thing it does.
This is worth sitting with, because it is what makes the rest of the course make sense. There is no database in there. There is no list of facts, no copy of the training data, no set of rules about grammar or style that somebody wrote down. There are eight billion numbers arranged into tables, and the entirety of what the model knows and every habit it has is a consequence of where those numbers ended up. When I talk about training a model to write like Doc Quatrain, I mean adjusting some of those numbers so that the text he would have written becomes more probable. That is the whole of the intervention available, in this course and in general.
The other thing those numbers decide is whether the model will run on your hardware at all, and that arithmetic is simple enough to do in your head.
Parameter count is not a ranking. It sets a rough ceiling and nothing more. A well-trained 8B model from this year will beat a badly trained 30B from two years ago on most things you would actually ask, because data quality and training technique moved faster than size did. Treat the number as one input among several, and be suspicious of any comparison that leads with it.
Quantization is lossy, and usually less lossy than you expect. Rounding every weight to 4 bits does degrade the model, and on most tasks the degradation is hard to notice while the memory saving is the difference between running something and not running it. It shows up first in the places that were already marginal. Part nine quantizes the base model for the training run itself, for reasons about memory rather than quality.
Mixture-of-experts models break the arithmetic above. Some large models activate only a fraction of their parameters for any given token, so an 80B model might do the work of a 3B one per token while still needing all 80B resident in memory. Fast to run, expensive to hold. Worth knowing the category exists, because for these the headline size stops predicting either speed or memory and the chart above would mislead you badly.
The weights are only part of what has to fit. The figure above is the fixed cost. There is a second, growing cost that appears while the model is actually running, and on a long conversation it can exceed the weights. That one gets a section to itself in part five, and it is the most common reason a model that loaded fine dies an hour later.
Everything above describes a model that already exists. Getting one is two distinct stages, and the difference between them is the foundation of part seven and the reason this whole course is possible for one person.
The first stage is pretraining. Take an enormous quantity of text, trillions of tokens, and repeatedly show the model a passage with the next token hidden. It predicts. You measure how wrong it was, and nudge every one of the eight billion parameters slightly in the direction that would have been less wrong. Do that continuously for weeks across thousands of expensive accelerators. That is where essentially all of the capability comes from: the grammar, the world knowledge, the ability to hold an argument together, the reasoning such as it is. It costs millions of dollars and no individual is doing it.
What comes out is called a base model, and a base model is a strange thing to interact with. It does not answer questions. Give it a question and it may well produce another question, because in the text it learned from, questions are often followed by more questions. Ask it for advice and it may produce a forum thread. It completes text. That is the only behavior it has, and every helpful, conversational quality you associate with these systems came later.
The second stage installs those qualities. Show the model many thousands of examples of an instruction followed by a good response, and nudge the parameters toward producing responses rather than continuations. Usually there is a further stage after that which shows it pairs of answers ranked by preference and pushes it toward the preferred one. The result is an instruct model, and it is what you are talking to in any chat interface. The capability came from the first stage. The manners came from the second.
The critical thing about the second stage is that it is comparatively cheap. Thousands of examples rather than trillions of tokens, hours rather than weeks, one machine rather than a datacenter. Everything in the second half of this course lives in that gap.
I am not going to pretrain anything, and neither is anybody reading this. I am going to take a model somebody else spent millions to pretrain, and run a very small amount of additional training of the second kind, on eight poems.
Eight, because two of the ten are held back and never shown to it. That decision gets a section to itself in part eight and it is the one I would defend hardest. A model that has seen all ten can recite, and reciting looks a great deal like style transfer if you are standing far enough away and want it to be.
One last piece, because it connects the architecture to the hardware and it is where beginners lose the most time. All of the above has to fit in the memory attached to a graphics card, and that memory is much smaller and much less flexible than a spec sheet suggests.
My machine has four RTX 3090s, 24 GB each, which people reasonably read as 96 GB. It is not 96 GB. It is four separate 24 GB pools with a comparatively slow road between them. A model larger than 24 GB has to be split across cards, and the split costs traffic on that road on every single token. I have written about this elsewhere on the site in more detail; here it is enough to say that the total is the least useful number in the whole build, and that "how much VRAM do you have" is nearly always the wrong question. The right one is "how much does your largest single card have".
To run a model you need room for three things at once. The weights, which are the predictable part at roughly two bytes per parameter, or half a byte if you quantize hard. A cache of intermediate values that grows as the conversation gets longer, which part five explains and which is what actually catches people out. And working space for the arithmetic of the current step, which is small but not nothing.
Training needs several times more memory than running does, for reasons that are the first thing part seven explains, and the gap between those two facts is where most of the practical decisions in this project get made. A model you can comfortably serve is not automatically a model you can comfortably train.
None of which means you need this machine. The training run in part nine uses a single card and about eleven gigabytes, and it carries a second track on a smaller model that fits in about six, which is a laptop GPU or a free hosted notebook. The rack is what I happen to have and it is why I know what I know. The exercise is not built on it, and I have tried to keep it that way at every step.
Part two is the odd one out and I want to flag it rather than have it come as a surprise. It is a short refresher on matrix multiplication: how to read a shape, one worked example, and how to count a model's parameters yourself. It is not a mathematics course and does not try to be. It is there because this course uses shape notation constantly and because the central trick in part seven is unreadable without it. If you write software and remember your linear algebra, skip it.
Two other parts deserve a warning. Part eight is the one that decides the outcome, and it is entirely about data rather than machine learning, which is why it is the part most likely to be skipped and the part least able to afford it. Part ten compares the trained model against a carefully prompted untrained one, which is the comparison most write-ups quietly omit because it is the one that can make the whole exercise look unnecessary.
The short version, which is worth having in one paragraph before the detail arrives. Text becomes tokens. Tokens become vectors. The vectors pass up through a few dozen identical blocks, where attention lets positions see each other and feed-forward layers do the processing at each position. At the top, a probability is produced for every possible next token, and one is sampled. Then it runs again. All of the knowledge and all of the style live in the numbers inside those blocks, and training is nothing more than moving those numbers.
Three things get described rather than derived, and I would rather name them than have you find the gap later: how position is encoded so the model knows word order at all, the precise form of the loss function, and how gradients propagate backwards through the stack. Each is a real topic with real depth, and none of them changes a single decision made later in this course, which was the test I used.
Next is the arithmetic all of that is made of: how to read a shape, one worked example, and how to count a model's parameters yourself. It is the short part, and if you already have it, part three is where the figure above starts being opened up.