The model does not see letters and it does not see words. Before anything else happens, your text becomes a list of numbers, and those numbers become lists of other numbers. This is the stage nobody thinks about, and it explains more of the famous stupid failures than anything else in the architecture.
Part 03 of 10·9 min read
Text is chopped into tokens: chunks somewhere between a letter and a word. The chopping is done by a tokenizer, which is not a model and does no learning at inference time. It is a fixed lookup table, built once before the model was trained, and shipped alongside it in the same directory.
The set of possible tokens is fixed and finite. The model used later in this course has about 151,000 of them. Every piece of text that ever goes in or comes out is a sequence drawn from that set, and there is no character underneath: by the time anything reaches the model, "cat" is not three letters, it is the single number 8,251.
It is worth knowing where that vocabulary came from, because the answer explains most of its strange behavior.
The dominant method starts with individual characters and repeatedly merges the most frequent adjacent pair. If "t" followed by "h" is the commonest pair in the training text, "th" becomes a new entry. Then the process repeats: "th" followed by "e" is now common, so "the" is added. Run that merge step tens of thousands of times and you end up with a vocabulary in which the most frequent words are single entries, moderately common word-parts are entries, and everything else can still be spelled out from the fragments left over.
Two properties fall out of that, and both matter in practice. The vocabulary is a frequency artifact of whatever text it was built on, so words common in that text are cheap and words rare in it are expensive. And the tokenizer is deterministic and greedy rather than clever: it is not parsing your sentence or finding morphemes, it is applying a fixed list of merges in a fixed order.
That is why "Quatrain" splits the way it does. Nothing about the split is linguistically meaningful. It is the residue of which character pairs happened to be frequent in a corpus you have never seen.
The leading space is part of the token. In the figure, "the" and the space-prefixed version of "the" are two different entries with two different ids. This is how word boundaries get encoded at all, since the model has no other notion of where words begin. It also means text with unusual spacing tokenizes differently from text without it, which is one of the several reasons that pasting from a PDF behaves oddly.
Capitalization changes the tokens. A capitalized word is often a different token, or a different split, from the lowercase form. The model learns to handle this because it saw both, but it means a sentence in title case and the same sentence in ordinary case are not the same input, and neither is the same sentence shouted in capitals.
Numbers and code tokenize badly. Digits often split in ways that look arbitrary, which is part of why arithmetic is harder for these systems than their other abilities would suggest: the model may be seeing "1", "07", "3" rather than the number 1073. Code and non-English text are usually less efficient too, sometimes dramatically, because the vocabulary was built on a corpus dominated by English prose. The same paragraph in a less represented language can cost two or three times the tokens.
The tokenizer is part of the model, not a setting. The weights were learned against that exact vocabulary, so every id means something specific to those weights. Swapping tokenizers is not a configuration change, it is a different model, and a great many confusing failures in a first training run trace back to somebody who mixed a tokenizer from one place with weights from another. In part nine this is why the model download is checked for its tokenizer files before anything else happens.
Three practical consequences follow from all of this, and they are the reason a stage everybody skips is worth a section.
This is why a model miscounts letters in a word. Ask how many times a particular letter appears in a word and it may get it wrong. It never saw the letters. It saw two or three chunks, and it is being asked a question about a representation it does not have, rather like asking somebody to count the strokes in a word they only ever heard spoken aloud. This is not a reasoning failure and no amount of additional model capability fixes it cleanly. It is a mismatch between the question and the input format.
Everything you will ever be limited by is measured in tokens. Context length, throughput, hosted billing, and the length of every training example in part eight. When you read that a model handles 262,144 of context, that is tokens, and translating it into pages of prose is arithmetic you will do constantly. Getting used to thinking in tokens early is the difference between planning around a limit and being surprised by one.
A corpus has a token cost you can measure before you start. Run the text through the model's own tokenizer and you learn how long the average piece is in the units that matter, and more importantly how long the longest one is. Doc Quatrain's ten pieces come to a little over four thousand tokens in total, which is a startlingly small number and shapes every decision after it. Part nine measures the longest example before writing a single configuration value, because setting the sequence length below it silently truncates the piece.
Some entries in the vocabulary are not text at all, and they are the reason part eight can warn you about a failure that produces no error message.
Alongside the ordinary word pieces, the vocabulary holds a small number of special tokens. There is one meaning end of text. In an instruct model there are several more that mark where a turn begins, which role is speaking, and where a turn ends. They are genuine vocabulary entries with real ids and real learned embeddings, sitting in the same table as everything else.
What makes them special is that the tokenizer will not produce them from text you type. If you write the literal characters of an end-of-turn marker into your prompt, you get the ordinary tokens for those characters, not the marker. They can only be inserted by the layer above the tokenizer, which is exactly the property that keeps a user from ending their own turn early or impersonating the assistant by typing.
That layer is the chat template: a piece of formatting that ships in the tokenizer directory and turns a list of messages with roles into one flat string with the right markers in the right places. When you send a system message, a user message and an assistant response, this is what flattens them.
# what the model actually receives, roughly
<|im_start|>system
You are Doc Quatrain. You write short technical field notes.<|im_end|>
<|im_start|>user
Write a field note about backups that are never restored.<|im_end|>
<|im_start|>assistant
I ran the backup nightly for the better part of a year,<|im_end|>The exact markers differ between model families and that is the whole problem. A model was trained to key on its own set, in its own arrangement, and it has no way to tell you when it is shown a different one. Feed it another family's template and nothing errors: the tokens are valid, the shapes are right, the loss goes down. You simply get a model that is quietly worse, for a reason that will not be visible anywhere you would think to look.
Let the trainer apply the template rather than writing the string yourself. Every serious training tool reads the template out of the tokenizer directory and applies it for you, which is why part eight hands over a list of messages rather than pre-formatted text. Hand-formatting is a way to lose an evening to a bug whose only symptom is disappointment.
Check that the template is actually there. A model repository can be missing its template, in which case you are responsible for the formatting and probably do not know it. Part nine checks for this immediately after the download, before anything else, because finding out later means discarding a run.
The end-of-turn marker is how generation knows to stop. The model predicts it like any other token, and the serving software watches for it and halts. That is the entire stopping mechanism, and part six comes back to what happens when it never arrives.
Look at your own tokenization once. Two lines of Python against your corpus will show you exactly what the model sees, markers included. It is worth doing before you build a dataset rather than after a run behaves strangely, and it is the fastest way to make everything in this part concrete rather than theoretical.
A token id is a row number and nothing more. Id 4,912 is not one greater than 4,911 in any sense that means anything, and the two entries may have nothing in common. Ids are arbitrary labels, so the first thing the model does is trade each one for a list of about 4,096 decimal numbers, looked up from a table with one row per vocabulary entry.
[5, 4096] is five tokens, 4,096 numbers each. From here on there is no text inside the model at all.That table is not small. At 151,936 entries of 4,096 numbers each it holds well over six hundred million parameters, which is a meaningful fraction of a model this size before a single layer of actual processing exists. Many models reuse the same table at the output end, in part six, rather than learning a second one, which saves that cost twice over.
The width of the vector, 4,096 here, is one of the two numbers that define a model's size. The other is the depth, meaning how many blocks are stacked, which part five covers. Bigger models are generally both wider and deeper, and the width propagates everywhere: it sets the size of nearly every table in every layer.
Nobody wrote down what the 300th number in a vector means. It has no name and almost certainly no clean meaning on its own. The table was learned during pretraining rather than designed, by the same nudging process as everything else: an embedding that led to bad predictions got adjusted, several trillion times.
What emerges is geometry. Tokens used in similar ways ended up with vectors pointing in similar directions, because the training had enormous pressure to bring them together and no reason to separate them. If "server" and "host" appear in the same kinds of sentences, then the embeddings that work well for one tend to work well for the other, and the process converges on placing them near each other.
That is the sense in which the model has learned something about meaning, and it is worth being precise about how limited a sense it is. There is no definition anywhere. There is a position relative to everything else, derived entirely from company kept. A word the training text used consistently wrongly would be embedded wrongly, confidently, with no mechanism that could notice.
One thing is conspicuously missing from all of this, and it is worth flagging before part four assumes it. The embedding of a token is the same wherever that token appears. The word "the" at the start of your sentence and the word "the" at the end get identical vectors out of the table, which means that as things stand the model has no idea what order your words are in.
Order is added separately, by a scheme that modifies the vectors according to position before or during the attention step. There are several such schemes and the details are genuinely intricate; what matters for everything that follows is only that position is injected rather than inherent, and that it is one of the few places where models differ from each other in a way that affects how far their context can be extended.
The geometry established here is the raw material for everything after it. The stack of blocks in part five does not add meaning from outside. It moves these vectors around, starting from the positions the embedding table already established, and every transformation downstream is operating on this.
Two things to carry forward. Everything is tokens, so every limit, every cost and every training example you meet later is denominated in them, and a habit of checking that arithmetic will save you a great deal of confusion.
And a vector is a position rather than a symbol. That is what makes the next stage possible at all: you cannot do arithmetic on the id 8,251, but you can absolutely do arithmetic on a list of 4,096 numbers, and comparing two such lists to see how well they match is exactly what attention does.
Next is attention, which is the only place in the entire architecture where one token can affect another. It is the part worth slowing down for.