Follow The Progress
How to Train a Small Language Model
Training is four steps repeated millions of times. Feed a batch to the model. Measure how wrong it was, that’s the loss. Work out which way every parameter should move to be less wrong, that’s the gradient. Move them a tiny bit, that’s the optimizer. Easy to write down. The hard part is everything around it, starting with time: when you have one machine, every mistake costs you days.
Build a small code LLM from scratch

The loss
F.cross_entropy(logits, labels, ignore_index=IGNORE)
Cross-entropy is minus the log of the probability the model gave to the right next token, averaged over every position that has a label. A perfect guess scores 0. A blind guess over 16,000 tokens scores about 9.68, and that’s the number you’ll see on step one. If you don’t, something is already broken.
The recipe
These are theOG-50M v3’s actual settings, nothing hidden:
- AdamW with betas 0.9 and 0.95, so every parameter gets its own step size from a running average of past gradients.
- Peak learning rate 3e-4, weight decay 0.01 as a mild brake on memorizing.
- Warmup over the first 5% of updates, because early gradients are noise and big steps on noise are a bad idea.
- Cosine decay from 3e-4 down to 3e-5, so the last steps are small enough to settle.
- Gradient clipping at 1.0, so one bad batch can’t wreck weeks of work.
- 4 windows × 8 accumulation steps × 2048 tokens, 65,536 tokens per update, in plain float32. Slower than mixed precision, but nothing to tune and nothing to debug at 2 a.m.
Gradient accumulation, or how to fake a bigger GPU
Your GPU might not fit 32 windows at once. So the loop runs 8 micro-batches of 4, adds up the gradients, and updates once. Same math as a batch of 32. Out of memory? 2 windows × 16 steps, same result, smaller pieces.
A good loss curve can be a beautiful lie
Training loss always goes down. That’s its job, and it means very little. Every 2,000 micro-batches the loop measures the loss on validation windows, tasks it never trains on. That number tells you whether the model is learning something that carries over, or just memorizing. And even a gorgeous validation curve isn’t proof the code works. Only the benchmark tells you that (part 7). Until then, the cross-entropy trajectory is a story you tell yourself.
Three checkpoints are kept, about 200 MB each:
best.pt: lowest validation loss so far. This is the one you evaluate and export.resume.pt: weights plus optimizer state. Power cut? Run the same command again and it continues from the very same batches.final.pt: whatever the weights were at the last step.
Generating code
Generation encodes the prompt, runs the model, takes the scores at the last position, picks a token, appends it, and repeats until it writes <|/Answer|>. At temperature 0 it always takes the top token, that’s greedy. theOG’s default is 0.2: it draws from the 40 best tokens, and most of the time still picks the top one.
Hardware, and patience
The laptop track trains its small model in about 2½ hours on a MacBook Air. It reads its clean data three times: that’s the Phi recipe, a small clean set seen several times, rather than Chinchilla’s rule of about 20 tokens per parameter seen once. Its first loss is about 9.0, a blind guess over its 8,000 tokens. The full run needs your own GPU, NVIDIA with CUDA or Apple silicon. Run a 20-step smoke test first (--max-steps 20): the first loss should be close to 9.7. The log prints elapsed time, so you can work out how long the whole run will take on your machine. Then comes the waiting. And the adrenaline of the first benchmark run, which is always there, like a first kiss. You just have to earn it.
The full lesson, with the code and the tests, is free on GitHub: DrunkkInference, lesson 5.
Honest question: how long do you wait before you trust a loss curve enough to run a real benchmark?
Quick answers
What should the first training loss be?
About 9.7, the loss of a blind guess over 16,000 tokens.
Why use learning rate warmup?
Early gradients are noisy, so small steps at the start keep training stable.
Which checkpoint should I use after training?
best.pt, the weights with the lowest validation loss.
Can I train theOG on a CPU?
The tests run on a CPU. The laptop track trains on any Apple silicon Mac in an afternoon. The full theOG run needs an NVIDIA or Apple silicon GPU.
Next: the /think span. And if you have a task you think a small model could take over, tell me about it.