Follow The Progress
How to Build a BPE Tokenizer for a Code LLM
A neural network can’t read. It does arithmetic, period. So before theOG can look at def add(a, b):, the text has to become a list of numbers, and whatever numbers it writes back have to become text again. That’s the tokenizer, the first thing you build in the course, and the least glamorous part of any LLM. Also one of the easiest places to quietly ruin it.
Build a small code LLM from scratch

One number per byte? Sure, if you like wasting compute
The lazy option is to give every byte its own number. There are only 256, and any text in any language fits. The catch: return becomes six tokens, and the model burns its tiny capacity re-reading letters it has already seen a million times.
Byte-pair encoding in one paragraph
BPE starts from those 256 bytes and repeats one dumb step: find the pair of neighbouring tokens that shows up most often in the training text, and make it a new token. Do it a few thousand times and the common pieces of Python, return, self, four spaces of indentation, each become a single token. Rare text still falls back to raw bytes, so “unknown word” simply doesn’t exist. Simple ideas are underrated.
Why 16,000 tokens
Vocabulary size is a trade-off nobody gets for free. More tokens mean shorter sequences, but every token needs a row in the embedding table. In theOG that table is 16,000 × 512, about 8.2 million parameters, 16% of the whole model. In a 50M model, every parameter you spend on a fancy vocabulary is one you don’t spend on actually thinking.
The code
First you write the merge loop by hand, in plain Python, and watch it learn indentation before anything else. Then the fast version, no heroics: Hugging Face’s tokenizers library and a few lines:
tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=True)
tok.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(vocab_size=16_000, min_frequency=2,
special_tokens=SPECIAL_TOKENS,
initial_alphabet=pre_tokenizers.ByteLevel.alphabet())
ByteLevel works on raw bytes, so emoji and accented letters come back exactly as they went in. use_regex=True splits the text the GPT-2 way first, words, numbers, punctuation and spaces, so a merge never glues the end of one word to the start of the next. That detail comes back at the very end: llama.cpp knows this exact split as gpt-2, and that’s why the exported model cuts text the same way it did in training (part 8). Get it wrong and your model is fine in PyTorch and garbage in Ollama.
The special tokens, <|User|>, <|Answer|>, <|end|> and friends, go first, so they always get ids 0 to 8. Normal text can never produce them, which is exactly what you want from the markers of a chat format.
Don’t trust it, test it
The tests check three things: the special tokens keep ids 0 to 8, any text with accents, emoji or tabs decodes back to exactly itself, and <|Answer|> is one token. When you print how a function gets split you’ll see Ġ everywhere. That’s a space, byte-level style. Ċ is a newline. Ugly, but honest.
Small confession: theOG’s own tokenizer has its special tokens split between the start and the end of the vocabulary, because some were bolted on later. Starting fresh, put them all first. Learn from my mess, not yours.
Try this
Train once with 2,000 tokens and once with 16,000, then count the tokens of the same file. Then encode four spaces and a tab, and see which one your model has to pay more for.
The full lesson, with the code and the tests, is free on GitHub: DrunkkInference, lesson 1.
How many people shipping “custom LLMs” have ever looked at how their tokenizer splits their own data?
Quick answers
What tokenizer does theOG use?
Byte-level BPE with 16,000 tokens and the GPT-2 pre-tokenizer split.
Can a byte-level tokenizer hit an unknown word?
No. Anything that isn’t a learned token falls back to single bytes, so every text encodes and decodes exactly.
Why do the special tokens get the first ids?
So their ids never move, and ordinary text can never produce them by accident.
Next: the training data. And if you have a task you think a small model could take over, tell me about it.