Follow The Progress
Chat Format, Labels and Sequence Packing for LLM Training
Strip away the marketing and a language model learns exactly one thing: given the tokens so far, guess the next one. Writing a whole function is that guess repeated, token after token. So before training, every row has to become one sequence where the next token is actually worth learning. This part covers how theOG lays that sequence out, what it gets graded on, and how rows are packed so the GPU isn’t busy learning padding.
Build a small code LLM from scratch
- Overview: meet theOG-50M
- Part 1: the tokenizer
- Part 2: the training data
- Part 3: chat format and packing (you are here)
- Part 4: the transformer
- Part 5: training
- Part 6: the /think span
- Part 7: evaluation
- Part 8: export to GGUF and Ollama
- Part 9: publishing

The chat format
<|User|>
Write a function that reverses a string. /no_think
<|/User|><|Assistant|><|Think|><|/Think|><|Answer|>
```python
def reverse(s):
return s[::-1]
```
<|/Answer|><|end|>
The task sits between <|User|> and <|/User|> and ends with a mode, /no_think or /think. That little switch has its own story in part 6. <|Assistant|> means it’s the model’s turn. The empty <|Think|><|/Think|> pair is the think span. The code goes between <|Answer|> and <|/Answer|> in a normal Python block, and <|end|> closes the conversation. When generating, the model stops as soon as it writes <|/Answer|>.
Only the answer gets graded
Each sequence comes with two lists of the same length: input_ids, the tokens, and labels, the token that should come next at every position.
The prompt is always handed to the model. It never writes it, so there’s nothing to learn from guessing it. Every prompt position gets the label -100, which the loss simply skips. From <|Assistant|> on, each label is the next token. The model is judged on the think span and the answer, nothing else.
Packing: stop paying for padding
The GPU works on batches where every sequence has the same length, 2048 tokens for theOG. A typical row is a few hundred tokens. Pad each one to 2048 and most of your expensive training time goes into learning nothing.
Packing fills each window with as many whole rows as fit, and pads only what’s left. Two rules, no exceptions. A row is never cut in half: if it doesn’t fit, it starts the next window. And a row longer than a whole window is dropped, not truncated, because a cut-off answer teaches the model code that never ends. Nobody needs a model that learned to stop mid-function.
Keeping validation honest
Rows go to train or validation (5%) by a hash of their prompt, so the same task always lands on the same side. The validation loss in part 5 is measured on tasks the model has never trained on, which is the only kind of validation loss worth looking at.
Data bigger than your RAM
The windows are written to disk as raw int16, two bytes per token, and opened with np.memmap. Each batch reads only its own windows, so a multi-gigabyte dataset never has to fit in memory. On the laptop track, with 512-token windows, packing produces 58,052 training windows holding 22 million tokens, and about a quarter of every window is still padding. The lesson asks you why, and what you’d change.
The full lesson, with the code and the tests, is free on GitHub: DrunkkInference, lesson 3.
Packing is boring, and it is also where a lot of training budget quietly disappears. When did you last check how much of your batches is padding?
Quick answers
Why is the prompt masked out of the labels?
The prompt is always given, so there’s nothing to learn by predicting it. Masking keeps training on the answer.
What happens to rows longer than the context window?
They’re dropped. Cutting them short would teach the model answers that never finish.
Why split train and validation by a hash of the prompt?
So the same task always lands on the same side, and validation only measures tasks the model hasn’t seen.
Next: the transformer. And if you have a task you think a small model could take over, tell me about it.