Nov 19, 2026 | microModels
Follow The Progress FollowFollowFollow How to Build a Small Code LLM from Scratch: Meet theOG-50M Everybody wants a bigger model. I went the other way. theOG-50M is a 50-million-parameter model that writes Python, and I built it from nothing: no pretrained weights,...
Nov 19, 2026 | microModels
Follow The Progress FollowFollowFollow How to Build a BPE Tokenizer for a Code LLM A neural network can’t read. It does arithmetic, period. So before theOG can look at def add(a, b):, the text has to become a list of numbers, and whatever numbers it writes...
Nov 19, 2026 | microModels
Follow The Progress FollowFollowFollow How to Clean Training Data for a Small Code Model Here’s the unsexy truth about small models: designing the architecture is the easy part. The papers are public and you can iterate fast. The real deal is the recipe, the...
Nov 19, 2026 | microModels
Follow The Progress FollowFollowFollow Chat Format, Labels and Sequence Packing for LLM Training Strip away the marketing and a language model learns exactly one thing: given the tokens so far, guess the next one. Writing a whole function is that guess repeated,...
Nov 19, 2026 | microModels
Follow The Progress FollowFollowFollow Build a Transformer from Scratch in PyTorch Let me say something unpopular: the architecture is the easy part. The papers are all public, and an LLM can practically write one for you. That doesn’t mean you shouldn’t...
Nov 19, 2026 | microModels
Follow The Progress FollowFollowFollow How to Train a Small Language Model Training is four steps repeated millions of times. Feed a batch to the model. Measure how wrong it was, that’s the loss. Work out which way every parameter should move to be less wrong,...