Follow The Progress
How to Build a Small Code LLM from Scratch: Meet theOG-50M
Everybody wants a bigger model. I went the other way. theOG-50M is a 50-million-parameter model that writes Python, and I built it from nothing: no pretrained weights, no borrowed tokenizer, no “fine-tuned on top of” anything. It runs offline on a laptop in Ollama, LM Studio or llama.cpp. This series explains how, one piece per post, in the same order you’d build it yourself.
Build a small code LLM from scratch

Try it first, read later
If you have Ollama, one line is enough:
ollama run hf.co/DrunkkToys/theOG-50M:Q8_0
Ask for a function in plain English, something like “Write a function that checks if a string is a palindrome”, and you get code back. That’s it. It doesn’t chat, it doesn’t write poems, it doesn’t tell you how great your question was. It writes Python functions.
Why 50M, in the age of trillion-parameter hype
The goal was simple and slightly arrogant: beat Salesforce’s CodeGen-350M-mono, a model seven times bigger, while staying under 350M parameters. The numbers are on the theOG-50M model card, with the harness, the tasks and CodeGen’s scores on the same tests. Don’t trust me, check them.
The bigger point: most of what people send to giant models is narrow and repetitive. You don’t need a frontier model to write the same kind of function ten thousand times a day. A small model trained for one job can sit next to your data, answer without a network round trip and without a bill per request. I call them microModels, and theOG is the first one.
The course
All of this is packed into DrunkkInference, a free course on GitHub for Python developers who have never trained a model. Every lesson starts from zero and has you write the important parts yourself: the BPE merge loop, the data filters, attention, RoPE, the training loss, the sampler, the evaluation harness and the export. Each function has a test that stays red until your code is right, a folded solution for when you’re stuck, and the real output I got, so you can compare. No PhD required. Curiosity and some patience, yes.
git clone https://github.com/DrunkkToys/DrunkkInference
cd DrunkkInference
uv sync --extra dev --extra export
uv run python -m theog.progress
The series follows the lessons:
- The tokenizer: text to numbers and back.
- The training data: the recipe, which is where the real work is.
- Chat format and packing: how a task and its answer become one sequence.
- The transformer: the model itself, in eight small steps.
- Training: loss, optimizer, checkpoints, and a lot of waiting.
- The /think span: the empty token that made the model better.
- Evaluation: running the code against tests it has never seen.
- Export to GGUF and Ollama: one file that runs anywhere.
- Publishing: a model card people can actually verify.

What you need
Python 3.12, uv. The course has two tracks. The laptop track trains a small model on 178 MB of data in about 2½ hours on a MacBook Air, so you can go from an empty function to your own model running in Ollama in a weekend. The full track is the same commands at theOG’s real size: 6.9 GB of data and a GPU, NVIDIA with CUDA or Apple silicon, that you can leave busy for days.
Fair warning: theOG-50M v3 trained on 12 public datasets, 2.49 billion tokens. The course uses two of them. Same architecture, same recipe for training, different data, so your model won’t score exactly like mine. That’s not a bug, that’s the lesson.
So, do we really need a giant model for every task, or have we just been told we do?
Quick answers
Can I run theOG-50M without a GPU?
Yes. The GGUF file runs offline on an ordinary computer through Ollama, LM Studio or llama.cpp. You only need a GPU to train your own.
Is theOG-50M free?
Yes. The model is on Hugging Face under the Apache 2.0 licence, and the course is on GitHub under MIT (code) and CC BY 4.0 (lessons).
What languages does it write?
Python. Only Python. It was trained on Python function tasks.
Do I need a machine learning background?
No. The course is written for Python developers who are new to it, and every lesson starts from zero.
Can you train a model like this for my task?
Yes, that is the business. I train custom microModels from the theOG base for one specific job.
Next: the tokenizer. And if you have a task you think a small model could take over, tell me about it.