Follow The Progress
How to Clean Training Data for a Small Code Model
Here’s the unsexy truth about small models: designing the architecture is the easy part. The papers are public and you can iterate fast. The real deal is the recipe, the dataset. That’s where the work is, and no LLM is going to save you here. It’s like being a chef: you have to figure out what you actually need to teach the model so it solves the problem, and every wrong ingredient costs you days of training to find out.
Build a small code LLM from scratch

One row, one task
Every training row is a task in plain English and a Python solution. That’s the model’s entire world:
{"source": "opencodeinstruct",
"user": "Write a function to find the subarray with the largest sum in a given array.",
"answer": "def find_largest_sum(nums):\n max_so_far = max_ending_here = nums[0]\n ..."}
The ingredients
The course uses two public datasets. nvidia/OpenCodeInstruct (CC BY 4.0) is the big one, and its killer feature is that every answer comes with unit tests the authors actually ran. I keep only the rows where all of them passed. The second is the educational_instruct part of OpenCoder-LLM/opc-sft-stage2 (MIT): short, textbook-style tasks.
Both are pinned to a specific commit, so the same command gives you the same rows next year. “It worked on my machine last month” is not a data strategy. The laptop track downloads one file per source, 178 MB. The full track is about 6.9 GB.
Five rules for throwing rows away
Every row goes through the rules in order and dies at the first one it breaks. The build prints how many rows each rule killed, and I recommend reading that report. It’s the fastest way to lose your faith in “high-quality public data”.
- The tests passed. An answer that fails its own tests teaches bugs, with confidence.
- Exactly one solution. One Python block that defines a function or class. Zero means no code. Three means nobody knows which one is the answer, including the model.
- The solution and nothing else. The code is parsed, and only imports, functions, classes and constants survive. Print calls,
if __name__ == "__main__":, docstrings and comments go. The task asked for a function, so the model learns to write a function.ast.unparsewrites it all back in one consistent style. - No answer hiding in the question. If a line of the solution is already in the prompt, the model learns to copy, not to solve. Plenty of public rows do exactly this.
- No pile of clones. Exact repeats go first. Then every name in the code becomes
_, sodef add(a, b): return a + banddef plus(x, y): return x + yturn out to be the same shape. Max three rows per shape.
Why three and not one? Small models learn in a funny way. A single, isolated example tends to get washed out by gradient noise, while the same idea shown in three or four variations builds a pattern the model can actually generalize from. It learns the logic, not the string. A few variants are teaching. A few hundred are just stuffing the turkey. This is the Phi bet: a small set of clean tasks beats a big pile of scraped code.
And one rule that throws nothing away: name the function. Most of the educational rows never say what the function should be called, so a model trained on them learns to pick its own names. Then it writes a perfect remove_vowel_chars, the test calls remove_vowels, and the task fails. The rows already carry their own unit tests, so when the prompt doesn’t name the function, the build appends up to three of its assert lines, the way MBPP prompts do. You write that rule in the course.
Don’t train on the exam
Later the model gets scored on HumanEval and MBPP. If those tasks are in the training data, the score measures memory, not skill. I learned this one the loud way: when I first published theOG’s scores, Reddit had a field day calling them fake because of contamination. They had a point, benchmark tasks were in my data. The twist was that when I measured separately, the model did worse on the tasks it had seen than on the ones it hadn’t. Singletons, washed out, exactly as above. Funny, but not a defence. So the course removes them properly.
The check is simple: take every run of 13 consecutive words from each benchmark prompt and reference solution, and drop any training row containing at least half of them for any single task. On the laptop track’s 95,765 rows it removed 282, near-copies of 90 benchmark tasks. One MBPP task alone had 60. Solutions shorter than 13 words slip through, so don’t pretend it’s perfect.
What theOG did on top of this
theOG-50M v3 used more rules of the same kind on a much bigger mix: 12 public Hugging Face sources, 4.96 million rows, all listed on the model card. The course keeps two sources and the rules that kill the most rows.
Try this
Build the laptop data and read twenty random rows yourself. Would you keep them all? I didn’t.
The full lesson, with the code and the tests, is free on GitHub: DrunkkInference, lesson 2.
Synthetic data was supposed to fix all of this. Has it, or is it just cheating the metric more politely?
Quick answers
Why does data quality matter more for a small model?
It has almost no spare capacity. Every broken or duplicated row takes room it needs for real patterns.
What is a 13-gram contamination check?
It compares runs of 13 consecutive words from each benchmark task against the training rows, and drops any row that shares at least half of one task’s 13-grams.
Why keep up to three copies of the same code shape?
A few variants help a small model learn the logic instead of the string. Hundreds just crowd out everything else.
Next: chat format and packing. And if you have a task you think a small model could take over, tell me about it.