Attention is half of a block. The other half is a small network that processes each position on its own, and around both are two pieces of plumbing that look like footnotes and are the reason any of this trains at all. Then the whole block is repeated three dozen times, and that is the entire architecture.
Part 05 of 10·8 min read
The second half of a block is a feed-forward network, applied to each token's vector separately, with no reference to any other token at all. Where attention is the only place positions interact, this is the opposite: 500 tokens go through it as 500 completely independent calculations that happen to share the same weights.
Mechanically it is short. Expand the vector from 4,096 numbers to roughly four times that width, apply a simple nonlinear function, and squeeze it back down to 4,096. Two learned tables and one nonlinearity between them.
The nonlinearity is the part that cannot be dropped. Without it, the expand and the squeeze would collapse into a single multiplication, and stacking three dozen of them would be mathematically identical to one. Every layer past the first would be doing nothing. It is the nonlinear step that makes depth mean anything, and this is the only place in a block where one appears.
Despite being the simpler half, this is where most of a model's parameters live. In an 8B model the feed-forward layers account for roughly two thirds of the weight count, and interpretability work suggests it is also where most of what looks like stored knowledge sits. The rough division of labor, and researchers will tell you it is rougher than I am making it sound, is that attention moves information between positions and the feed-forward layer does the processing at a position.
Put the two halves together with the plumbing and you have a block.
Those two pieces are worth more than their footnote appearance, because they are the difference between an architecture that works and one that does not.
The residual connections mean nothing is ever replaced. Adding a half's output back to its input has an immediate consequence: a layer that learned nothing useful can output approximately zero and leave the vector unharmed. The default behavior of an untrained layer is to do no damage, which means a deep model starts out able to pass information from bottom to top and gets better from there. Without that, every layer is a fresh chance to destroy the signal, and thirty-six chances is too many.
It also creates a straight path for training. The nudging process in part seven has to trace responsibility for an error backwards through every layer. Through thirty-six transformations, that signal shrinks or explodes long before it arrives. The additions give it an uninterrupted road: the correction reaches layer one nearly intact, and this is the specific reason very deep networks became trainable at all. It was not a subtle improvement, it was the change that unblocked the field.
Normalization keeps the numbers in a workable range. Before each half runs, the vector is rescaled so its values sit in a consistent range. Since every block adds to the vector, without this the magnitudes drift upward as you climb the stack, and by layer thirty the numbers are large enough that training becomes unstable. It is not doing anything conceptually interesting. It is preventing an accumulation problem that would otherwise make the depth unusable.
A useful way to hold all this is as a shared workspace. The vector running up the stack is often described as a residual stream: a channel that every layer reads from and writes back into, rather than a value passed from one function to the next. Each block reads the current state, computes a modification, and adds it. Nothing is handed over and nothing is consumed. This framing is where a lot of interpretability work starts, and it makes the architecture much easier to reason about than a pipeline metaphor does.
That block is then stacked. A few dozen times for a model of this size, identical in shape, each copy with its own separately learned tables. There is no other structure.
This is the thing I most wish somebody had told me plainly at the start. The architecture really is one block repeated, and the depth is where the capability comes from. When you read that a new model has more layers, that is what changed, and it is nearly all that changed.
Depth and width are the two dials that set a model's size, and they trade off. A wider model has bigger tables in every layer and can hold more in the vector at each position. A deeper model has more sequential steps of transformation and can build longer chains of derived structure. Published work has generally found deeper better than wider for a fixed parameter budget, up to a point, though this is not a settled question and the practical answer moves with the training technique.
One consequence of stacking is worth carrying into part seven. Every block has its own copies of every table, so a change to how the model writes is not a change in one place. It is a small change distributed across a few dozen layers at once, which is part of why fine-tuning behaves statistically rather than surgically, and why you cannot point at where a habit lives.
One consequence of the stack decides more about your hardware than the weights do, and it is the last piece of mechanism this course needs.
Recall from part four that each token, when processed, produces a key vector and a value vector for every head in every block. Now recall that generation happens one token at a time, and that each new token attends to every token before it. That means the keys and values of your earlier tokens are needed again on every single step.
They do not change. The key for token five is computed from token five and nothing downstream affects it. So recomputing them from scratch for the whole conversation on every new token would be enormous waste, and instead they are kept. That store is the KV cache, and it is the reason generation is fast enough to be usable.
The price is memory, and the price scales with the conversation. Keys and values, for every head, in every one of thirty-six blocks, for every token so far. On a long context this reaches gigabytes, and unlike the weights it was not there when you loaded the model.
The practical consequence is worth stating plainly, because it is the single most common surprise for somebody serving a model for the first time. A model that loaded with room to spare can still run out of memory forty minutes into a long session, and nothing about the weights changed. This is also why serving software asks you to declare a maximum context length up front: it is reserving the worst case, and setting that number too high means the model refuses to load at all while setting it too low means long conversations fail later.
The context window is one budget, not several. The system prompt, the conversation so far, any documents you pasted, and the answer being generated all come out of the same allowance. There is no separate space for instructions. A very long document pasted into a chat is competing directly with the room available for the reply.
This is the honest explanation for a model forgetting an instruction. It did not lose track and it is not being inattentive. The instruction is no longer inside the window, so it no longer exists as far as the model is concerned. There is no memory outside the window at all, and every product that appears to have one is re-inserting text into it behind the scenes, on your behalf, according to rules somebody chose.
Grouped-query attention exists to shrink this. Sharing keys and values across heads, mentioned in part four, cuts the cache by the sharing ratio directly. That is why it was adopted: not for quality, which it slightly costs, but because the cache was the binding constraint on how long a context could affordably get.
Serving many users at once multiplies it. Each concurrent conversation carries its own cache. This is why throughput on a single card falls off in a way the weights alone would not predict, and why batch size and context length are the two numbers a serving configuration argues about. On my own machine, deciding a maximum context length is mostly an argument about how many of these can coexist.
One more consequence of the cache, and it is the piece of operator knowledge I would most want somebody to have before they try to make a local model fast.
Generation happens in two phases with almost opposite characters. The first is prefill: your entire prompt is processed in one pass, every token at once, filling the cache. The second is decode: tokens are produced one at a time, each one a full trip through every layer, each one appending to the cache.
It explains why the first token is slow and the rest are steady. Time to first token is prefill, and it scales with how long your prompt is. Everything after is decode, at a roughly constant rate. Those are two different numbers, they are measured separately in any serious benchmark, and a system that looks fast on one can be poor on the other.
It explains why batching helps so much. During decode the weights are being read anyway, so running eight conversations at once reads them once and serves eight tokens rather than one. That is nearly free throughput, and it is the whole reason serving software batches aggressively. Prefill gets much less benefit, because it was already using the hardware fully.
It explains the power limit on my own cards. Part nine lifts a 280 W cap for training and puts it back for serving, and this is why. Decode is waiting on memory rather than on arithmetic, so watts are not the constraint and capping them costs almost nothing measurable. Training is prefill-shaped work on every step, compute-bound, and there the same watts buy real speed.
It is why memory bandwidth predicts inference speed better than raw compute. When comparing cards for running models rather than training them, the bandwidth number is the one to read first. A card with impressive arithmetic throughput and modest bandwidth will disappoint at decode, which is where a chat session spends essentially all of its time.
That is the whole architecture. Tokens, vectors, and a block containing attention and a feed-forward layer, wrapped in residuals and normalization, repeated a few dozen times. Everything a model can do comes out of that shape and the numbers inside it, and the shape has been essentially stable for years while the numbers and the training got dramatically better.
One stage of the pipeline is left: what happens at the top of the stack, where a vector has to become an actual word, and where a set of settings that live outside the model entirely turn out to change its character more than most people expect.