Follow The Progress
How to Export a PyTorch LLM to GGUF and Run It in Ollama
A .pt checkpoint is useless to almost everyone. It needs PyTorch, this repo and some patience. A GGUF file holds everything a runtime needs in one file: weights, tokenizer, chat template and the model’s shape. llama.cpp, Ollama and LM Studio load it directly, on a laptop, with no Python. This is the step where your model stops being a science project and becomes something people can actually use.
Build a small code LLM from scratch
- Overview: meet theOG-50M
- Part 1: the tokenizer
- Part 2: the training data
- Part 3: chat format and packing
- Part 4: the transformer
- Part 5: training
- Part 6: the /think span
- Part 7: evaluation
- Part 8: export to GGUF and Ollama (you are here)
- Part 9: publishing

The end result
ollama run hf.co/DrunkkToys/theOG-50M:Q8_0
What goes inside the file
First, the architecture name and shape. The file says qwen3 (part 4 explains why) and records what the runtime needs to rebuild the model: 10 blocks, hidden 512, 4 heads of 128, ffn 2048, context 2048, RoPE base 10,000.
Second, the weights, renamed to what llama.cpp expects. The exporter checks that every weight found its slot. A weight that silently goes missing gives you a file that loads fine and writes nonsense, which is worse than a crash.
blocks.3.attn.q.weight -> blk.3.attn_q.weight
blocks.3.mlp.down.weight -> blk.3.ffn_down.weight
embed.weight -> token_embd.weight
Third, quantization. F16 stores each weight in 16 bits. Q8_0 stores blocks of 32 weights as 8-bit integers plus one scale, about half the size of F16. theOG ships Q8_0, with the norm weights left in float32: there are few of them and they scale everything else.
Fourth, the tokenizer: tokens, BPE merges, control tokens, tagged with the GPT-2 pre-tokenizer so llama.cpp cuts text exactly like training did. Fifth, the chat template, the format from part 3 written in Jinja. A message ending in /think opens the think span; anything else gets /no_think.
Why Ollama needs a Modelfile
Ollama’s templates use Go’s template language, which can’t check how a string ends. So the course writes one Modelfile per mode, each fixing the mode, the stop tokens and theOG’s sampling settings: temperature 0.2, top-k 40, no repeat penalty, context 2048. Skip that and Ollama uses its own defaults, temperature 0.8 for one, and you’ll be evaluating a different model from the one you shipped.
Prove it’s the same model
An export can be wrong without a single error message. Check three things with llama.cpp’s llama-server:
- Tokens: tokenizing a prompt gives the same ids as the training tokenizer.
- Template: applying the template gives exactly the training prompt, in both modes.
- Output: with greedy decoding, the F16 file and the PyTorch model write the same tokens. Q8_0 rounds the weights, so it can pick a different token where two are nearly tied. That’s expected, not a bug.
The course’s exporter was checked exactly this way on a small model, and all three matched.
The full lesson, with the code and the tests, is free on GitHub: DrunkkInference, lesson 8.
How many models out there were benchmarked in PyTorch and shipped in a format nobody ever checked?
Quick answers
What is a GGUF file?
One file with a model’s weights, tokenizer, chat template and shape, which llama.cpp, Ollama and LM Studio run directly.
Why is theOG exported as qwen3?
It uses the same arrangement of layers as Qwen3, so existing runtimes run it with no custom code. It contains no Qwen weights.
F16 or Q8_0?
F16 stores weights in 16 bits. Q8_0 uses 8-bit integers with a scale per 32 weights, about half the size. theOG ships Q8_0.
Do I need a GPU to run the GGUF?
No. It runs offline on a regular computer.
Next: publishing. And if you have a task you think a small model could take over, tell me about it.