Follow The Progress
Build a Transformer from Scratch in PyTorch
Let me say something unpopular: the architecture is the easy part. The papers are all public, and an LLM can practically write one for you. That doesn’t mean you shouldn’t understand it. theOG is a decoder-only transformer, the same family as GPT and Qwen, and the whole thing is about 150 lines of PyTorch. In the course you write it yourself, in eight steps, each with a test and the exact output you should get. Here they are, in plain words.
Build a small code LLM from scratch

1. Embedding: ids become vectors
A token id is just a label. Token 5 isn’t “less” than token 17. The embedding is a table with one row per token, 16,000 rows of 512 numbers in theOG, and each id is swapped for its row. The rows start small and random and get learned, so tokens used the same way, like list and dict, end up looking alike.
2. RMSNorm: keep the numbers sane
Every block adds its output to the token’s vector, and after ten blocks the numbers could explode or vanish. RMSNorm rescales each vector so its root-mean-square is 1, then multiplies by a learned weight. Direction stays, size goes. theOG normalizes before each layer, not after, and that “pre-norm” choice is a big part of why deep transformers train without drama.
def forward(self, x):
return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.weight
3. Attention: tokens look back
To write return a + b, the model has to know that a and b are the arguments a few tokens back. Each token makes a query (what it’s looking for), a key (what it offers) and a value (what it passes on). It scores every earlier token with a dot product, a mask hides the future so it can’t cheat, softmax turns the scores into weights, and the token gets the weighted mix of values. You write it by hand first, five lines, and a test checks it against PyTorch’s one-liner, F.scaled_dot_product_attention(q, k, v, is_causal=True). All the mystique, one function call.
4. RoPE: word order
Attention on its own has no idea about order, and a - b is not b - a. RoPE rotates each query and key by an angle that depends on its position, some pairs spinning fast for nearby order, some slow for long distances. The result: the score between two tokens depends only on how far apart they are. A bracket five tokens back looks the same at position 10 as at 1,000. And it has zero parameters, which in a 50M model is a gift.
5. Heads and QK-norm
One attention can only point at one thing. A token often needs several at once: the matching bracket, the variable being returned, the indentation level. So theOG runs 4 heads of 128 numbers side by side and mixes their outputs. QK-norm puts an RMSNorm on every query and key before scoring. Without it, scores can grow until all the attention lands on a single token and learning just stalls.
6. SwiGLU: the part that does the heavy lifting
Attention moves information between tokens. The MLP then works on each token on its own. SwiGLU projects up to a wider space twice, runs one projection through the SiLU curve and uses it as a gate on the other, then projects back down. Most of the model’s parameters live here.
def forward(self, x):
return self.down(F.silu(self.gate(x)) * self.up(x))
7. The block and the residual stream
def forward(self, x, cos, sin):
x = x + self.attn(self.attn_norm(x), cos, sin)
return x + self.mlp(self.mlp_norm(x))
Look at the x = x + .... No layer replaces the vector. Each one reads a normalized copy and adds a small correction. That running sum is the residual stream, and it gives the gradient a straight road through all ten blocks. Without it, even a 10-block model is a pain to train.
8. The whole model, and tied embeddings
Embedding, ten blocks, a final RMSNorm, and then the output, which compares each final vector with every row of the same embedding table to get one score per token. Reusing the input table for the output is called tied embeddings. A token means the same thing on the way in and on the way out, and you don’t pay for a second 8-million-parameter table.
Why it looks exactly like Qwen3
Every choice above, pre-norm RMSNorm, QK-norm, NEOX-style RoPE, SwiGLU, tied embeddings, no biases, is the Qwen3 layout. Not for fashion. llama.cpp, Ollama, LM Studio and MLX already know how to run it, so theOG exports without a single line of custom code. Same arrangement of layers, zero Qwen weights. There’s no Qwen inside theOG.
The full lesson, with the code and the tests, is free on GitHub: DrunkkInference, lesson 4.
If the architecture is a solved, public recipe, where do you think the real difference between two small models comes from?
Quick answers
What kind of transformer is theOG?
A decoder-only transformer, like GPT and Qwen, with 10 blocks, hidden size 512 and 4 attention heads.
Why RMSNorm instead of LayerNorm?
It skips subtracting the mean, it’s simpler and it works as well. Qwen3 and Llama use it too.
What does RoPE do?
It encodes word order by rotating queries and keys by a position-dependent angle, so attention depends on the distance between tokens.
What are tied embeddings?
Using the same table for the input embedding and the output scores. Consistent token meanings and fewer parameters.
Next: training. And if you have a task you think a small model could take over, tell me about it.