The vectors reach the top of the stack and something has to turn one of them into an actual word. The model's real output is a score for every one of 151,936 possible tokens, and everything after that is a choice made by the software around the model, not by the model itself.
Part 06 of 10·8 min read
Only the last vector matters. The stack produced one for every token in your text, and at generation time all but the final one are discarded. That is worth a moment's thought: the model did a full pass of work on every position, and then used one.
The other positions are not wasted, incidentally. They did their work on the way up, feeding the attention that shaped the final one, so the last vector only carries what it carries because the others were computed. And during training all of them produce predictions at once, which is what makes training on whole documents efficient: one pass yields a prediction at every position, each one seeing only its own past.
That final vector is then multiplied against a table with one column per vocabulary entry, producing one raw score per possible next token.
That output table is large, at 4,096 by 151,936, which is over six hundred million parameters. Many models avoid paying for it twice by reusing the embedding table from part three, transposed. The intuition is that a token's input representation and the representation you would compare against to predict it are related enough to share, and in practice tying them saves a meaningful fraction of a small model's parameters at little cost.
A logit is not a probability and does not pretend to be one. Some are negative. They do not sum to anything in particular. They are raw compatibility scores, and one more step is needed before anything can pick from them.
That step is a function called softmax, and it does exactly two things: it makes every number positive, and it makes the whole set sum to one.
It works by raising a constant to the power of each score and then dividing each result by the total. Exponentiating is what forces everything positive, and dividing by the total is what forces the sum to one. The side effect worth knowing is that exponentiating exaggerates differences: a score that was moderately ahead of the pack comes out substantially ahead in probability terms, which is why these distributions are usually far more concentrated than the raw scores suggest.
What comes out is a real probability distribution over all 151,936 possibilities. Almost all of them have essentially zero probability. A few dozen have enough to matter. Usually somewhere between two and ten are genuinely plausible continuations, and the shape of that top handful is what the next section is about.
Something now has to choose one. The obvious approach is to take whichever token has the highest probability, every time, and this turns out to produce noticeably bad text.
The reason is worth understanding rather than accepting. Always taking the most likely token drives the model toward the safest continuation at every step, and safe continuations compound: the text falls into loops, repeats stock phrases, and reaches for the most predictable version of every sentence. Sequences of individually likely tokens are not the same thing as likely sequences, and greedy selection optimizes the wrong one.
So the standard approach is to sample from the distribution instead, with a couple of settings controlling how adventurously.
Temperature divides the scores before the softmax runs. That is all it is, mechanically: divide every logit by the temperature, then normalize. Dividing by a small number spreads the scores further apart, so the leader dominates and the output turns predictable. Dividing by a large number pulls them together, so the tail gets a real chance and the output turns varied and unreliable. It is the single most consequential setting you will touch, and two people comparing the same model at different temperatures are not comparing the same thing.
Top-p keeps only the tokens that carry the probability. Also called nucleus sampling. Sort the tokens by probability, add them up until you cross the threshold, and sample only from that group. At 0.9 you are drawing from the smallest set that covers ninety percent, which removes the genuinely bizarre tail without flattening what is left. It adapts to the distribution, which is its advantage: where the model is confident it keeps two or three options, and where the model is genuinely uncertain it keeps many.
There are others, and they mostly do the same job differently. Top-k keeps a fixed number rather than a fixed probability mass, which is blunter. Repetition penalties reduce the score of tokens that already appeared, which addresses loops directly rather than through temperature. Most serving stacks expose all of them and most of the time only temperature and top-p are worth moving.
The same prompt gives different answers on purpose. This is sampling, not instability, and it is the thing beginners most often mistake for a bug. Set temperature to 0 and the model becomes very nearly deterministic, taking the top token every time. I say nearly because floating-point arithmetic on a GPU is not perfectly reproducible across batch sizes and hardware, so identical settings can still diverge occasionally. Close enough for reproducing a result, and not the setting for producing good prose.
None of this is part of the model, so a fine-tune does not change it. Worth carrying into the second half of the course. Training moves weights. Temperature and top-p are configuration in whatever is serving those weights, and a well fine-tuned model served at a badly chosen temperature will look like a failed fine-tune. This is precisely why part ten holds temperature, top-p and seed identical across every system it compares.
Something has to decide when to stop, and nothing described so far does. The loop as written would run forever, producing a token, appending it, and producing another.
Three mechanisms end it, and they are worth separating because they fail differently.
The first is the model's own decision. Part three introduced the special tokens sitting in the vocabulary, and one of them marks the end of a turn. The model predicts it exactly the way it predicts any other token, by assigning it a high probability when the text feels finished, and the serving software watches for it and halts. That is the normal, healthy path: the model stopped because it judged the response complete.
The second is stop strings. You can hand the serving software a list of sequences that should terminate generation if they appear, which is a blunt instrument operating on the decoded text rather than on the model's judgment. Useful when you know the shape of what you are generating and the model has a habit of running past it.
The third is a hard maximum on the number of tokens, which exists as a backstop rather than as a design. When it fires, the output stops mid-sentence, because nothing about it is aware of meaning.
Check which one fired, because the difference matters. An API response tells you whether generation ended naturally or hit the limit, usually in a field named something like finish reason. A truncated answer and a complete one look similar at a glance and mean entirely different things, and treating a truncation as a complete answer is a real source of silent bugs in anything built on top of these systems.
This is one of the commonest ways a fine-tune breaks. If your training examples do not end with the end-of-turn marker properly applied, the model never learns to emit it, and the result is a model that generates a decent response and then keeps going: a second poem, then a third, then a conversation with itself, until it hits the token limit. It reads as a bizarre failure of coherence and it is nothing of the kind. Part three made the point that the template inserts those markers; this is the consequence of getting it wrong, and it is why part nine leaves the templating to the trainer.
A runaway generation is expensive as well as wrong. Every token is a full pass through the model, so a run that should have produced two hundred tokens and instead produces four thousand costs twenty times as much and takes twenty times as long. On a metered service that is a bill. On your own hardware it is a card busy for a minute doing nothing useful.
One more consequence of logits being ordinary numbers, and it is the sharpest illustration of the point this part keeps making.
Because the scores sit outside the model, in the serving software, they can be edited before anything is sampled. Set the score of every token you do not want to negative infinity and softmax gives it a probability of zero. It will never be selected, no matter what the model preferred.
That is the entire mechanism behind guaranteed structured output. If you need valid JSON, then at each step you compute which tokens could legally come next given what has been emitted so far, mask everything else, and sample from what remains. The result is not JSON because the model was asked nicely and complied. It is JSON because no other output was reachable.
The same technique enforces a schema, a regular expression, or any grammar you can express. It is why "guaranteed valid output" and "the model usually returns valid output" are different products, and why the first one is worth paying attention to when you are building something that has to parse the result.
The cost is worth knowing. Constraining pushes the model away from what it would have produced, and if the grammar fights the model's inclinations the quality inside the valid structure can suffer. You get output that parses and says less. The usual advice, which I think is right, is to ask for a format the model is already comfortable with and use constraints to guarantee it, rather than to impose a shape it has no feel for.
The chosen token is appended to your text, and all five stages run again from the beginning. Every token of a long answer is a complete pass through every layer of the model.
That is why generation speed is quoted in tokens per second, and why the length of an answer rather than its difficulty is what mostly decides how long you wait. A hard question and an easy question of the same length cost nearly the same. It is also why the first token of a response takes longer than the rest: the whole prompt has to be processed before anything can be produced, and that initial pass is a different shape of work from the steady drip that follows.
And it is why a model cannot revise. Once a token is emitted it is part of the input for every token after it, permanently. There is no editing pass. What looks like the model correcting itself is the model writing a correction as the next thing in the sequence, having already committed to the mistake. This also explains why a response that starts badly usually stays bad, and why regenerating tends to work better than asking for a fix.
That is the theory half of this course. Text becomes tokens. Tokens become vectors. Attention lets positions see each other, a block does that and then processes each position on its own, and the block repeats a few dozen times. At the top, a probability is produced for every possible next token, and one is sampled by software sitting outside the model.
Everything a model knows and every habit it has lives in the numbers inside those blocks. Which means that changing how a model writes has exactly one available mechanism: move some of those numbers. Not add a rule, not attach a document, not configure anything. Move numbers.
That is what the next four parts do. Part seven is what moving them reliably changes and what it does not, which is narrower than most tutorials suggest and is the difference between a useful result and a confident, fluent, subtly wrong one.