Follow The Progress
How to Evaluate a Code LLM on HumanEval and MBPP
The AI boom has everyone launching products and agents, and almost nobody running their own benchmarks. People stare at leaderboards and judge their own system by whether the output looks cute. A lower validation loss doesn’t tell you whether the code works either. There’s only one real test: run the code the model writes against tests it has never seen. That’s this lesson, and you write the harness yourself: pulling the code out of the answer, and the sandboxed test runner.
Build a small code LLM from scratch

The two benchmarks
HumanEval has 164 tasks, each a function signature and docstring written by OpenAI. MBPP‘s test split has 500, each one sentence plus three assert lines. A task passes when the model’s code plus the task’s own tests run without an error. The score is just the count of passed tasks. No vibes involved.
How a task actually runs
The model gets the task in its chat format and writes an answer. The code is pulled out of the Python block. For HumanEval, the prompt’s own imports and helpers are put back in front, since the model was only asked for one function. Then the code runs in a fresh Python process (python -I) with a 10-second limit, so the model’s code can’t tamper with the harness and infinite loops get killed. Every task prints PASS or FAIL with the reason: an assertion, a NameError, a timeout, a missing function.
The results file keeps the raw output and the code for every task. Any score can be checked by reading what the model actually wrote, which is where error analysis starts, and error analysis is where you learn something.
One warning: generated code is untrusted. The fresh process protects the harness, not your machine. If that matters to you, run it in a container or a VM.
Test the harness before you test the model
If a score is zero or makes no sense, assume the harness is broken before you blame the model. Two checks:
First, known answers must pass. Every task ships with a reference solution, so run them all. Every HumanEval reference passes. On MBPP, all but two do: those two compare floats with == and fail even with their own official answer. Good luck to any model on those.
Second, a known model must get its known score. Run a public model, Salesforce CodeGen-350M-mono, through the same harness and compare with the score the theOG model card reports for it. If you’re far off, the bug is in your harness, not in the universe.
Comparing models without fooling yourself
Always compare on both benchmarks. A change that helps HumanEval can quietly hurt MBPP. And at temperature 0.2 the score moves by a few tasks from run to run, so a gain of one or two tasks is not a breakthrough, it’s weather.
I still remember Model #607, the first architecture I built that actually survived a benchmark and generalized. Before that, plenty of runs looked great on the loss curve. The benchmark is the only judge that can’t be charmed.
The full lesson, with the code and the tests, is free on GitHub: DrunkkInference, lesson 7.
Are you running evals on your own system, or trusting that if it looks right, it works?
Quick answers
What does pass mean on HumanEval?
The model’s code plus the task’s own tests run without any error.
Why run model code in a separate process?
So generated code can’t change the harness, and a time limit can stop infinite loops.
How do I know my evaluation harness is correct?
Check that the reference solutions pass, and that a public model gets its published score on your harness.
Is a one-task improvement meaningful?
No. With sampling, scores move by a few tasks from run to run.
Next: export to GGUF and Ollama. And if you have a task you think a small model could take over, tell me about it.