LLM Benchmarks Explained: MMLU, HumanEval, GSM8K

Three benchmark names appear in almost every model announcement, and they are quoted far more often than they are defined. MMLU, HumanEval and GSM8K measure three unrelated things, are scored by three different mechanisms, and each carries a different kind of error bar. Knowing what is inside each one is the difference between reading a leaderboard and being misled by it.

MMLU: broad multiple-choice knowledge

MMLU was introduced by Hendrycks and co-authors as a test of knowledge acquired during pretraining, spread across 57 subjects that range from elementary mathematics through US history, law and professional medicine. Every item is a four-option multiple-choice question, and the set contains just under 16,000 of them. A small development split of five questions per subject exists specifically to be used as few-shot examples, with the bulk of the items, roughly 14,000, held in the test split.

Two properties matter when you read an MMLU number.

The first is the shot count. The paper’s standard protocol is five-shot, using those five per-subject development questions as in-context examples. A zero-shot MMLU figure and a five-shot MMLU figure are different measurements and are not comparable, yet they are frequently listed in the same table.

The second is the scoring mechanism. Because each item has four labelled options, a harness can score it two ways: compare the model’s log-likelihood for each option and pick the highest, or generate free text and parse a letter out of it. The lm-evaluation-harness maintained by EleutherAI implements MMLU as a log-likelihood task by default, which is why harness-reported numbers and vendor-reported numbers for the same model can differ by several points without either being fabricated. Generation-and-parse scoring additionally punishes models whose answer formatting drifts, which is a formatting result dressed up as a knowledge result.

MMLU is also the most saturated of the three. Its four-option format makes the random-guess floor 25 percent, and once frontier scores cluster in the high eighties, the remaining headroom is small enough that a couple of points can come from prompt phrasing. MMLU-Pro was built in response: it keeps the multi-subject structure but raises the option count from four to ten across 12,032 questions, which drops the guessing floor from 25 percent to 10 percent and restores separation between strong models.

HumanEval: functional correctness for code

HumanEval is a set of 164 hand-written Python programming problems published alongside the Codex paper. Each problem gives a function signature, a docstring describing the intended behaviour, a reference body and a set of unit tests, averaging 7.7 tests per problem. The problems were written by hand precisely so they would not appear in a public repository and be memorised during pretraining.

Scoring does not compare text. The model’s completion is executed against the hidden unit tests, and the problem counts as solved only if every test passes. This is what makes HumanEval a functional correctness benchmark rather than a similarity benchmark, and it is why the metric attached to it is pass@k.

Pass@k is the probability that at least one of k samples solves a problem. The naive way to estimate it, generating k samples and checking whether any passed, is high-variance. The Codex paper instead generates n samples per problem, counts the c that pass, and computes an unbiased estimator of pass@k from n and c, averaged across problems. The reported Codex-12B figures show why the k matters: 28.8 percent at pass@1 against 72.31 percent at pass@100 for the same model. Those are the same capability described at two different sampling budgets.

Which one you should care about depends entirely on whether anything downstream can filter the candidates. If generated code is run against tests before a human sees it, a high k is a fair description of the system. If the user sees one completion, pass@1 is the only honest number. That distinction is developed further in the guide to designing an LLM evaluation that actually tells you something.

The other property to keep in mind is size. At 164 problems, a single HumanEval item is worth about 0.61 percentage points. Two models three points apart differ by five problems, which is well inside the noise of a stochastic decoder. Treating a three-point HumanEval gap as a ranking is the single most common misreading of this benchmark.

GSM8K: multi-step arithmetic reasoning

GSM8K is a set of about 8,500 grade-school maths word problems written by human problem writers, split into roughly 7,500 training items and a 1,319-item test split. The problems are deliberately simple in content, needing only basic arithmetic, but each requires a chain of between two and eight steps to reach the answer. Every problem ships with a natural-language solution showing that chain.

Scoring is exact match on the final numeric answer. This makes GSM8K cheap and deterministic to grade, but it also means the benchmark says nothing about whether the reasoning that produced the answer was valid. A model can reach the right number through an invalid chain and score identically to one that reasoned correctly.

GSM8K is also the benchmark most sensitive to prompting technique. Chain-of-thought prompting, self-consistency across sampled chains, and tool or calculator access all move the score substantially on the same underlying model. A GSM8K number without its prompting protocol attached is not a measurement of the model.

What none of the three measure

All three sets are single-turn, short-context, English, and text-only. Between them they say nothing about long-context retrieval, multi-turn coherence, tool and function calling, instruction following under conflicting constraints, refusal behaviour, latency, or cost per resolved task. A model can top all three and still fail the job you have for it, because the job is not in any of them.

Benchmarks built for those other capabilities exist and are worth naming. GPQA is a set of 448 graduate-level science questions written to be resistant to search: the paper reports that domain experts reach about 65 percent while skilled non-experts with unrestricted web access reach about 34 percent. SWE-bench draws 2,294 task instances from real issue and pull-request pairs across 12 popular Python repositories, and grades by whether the repository’s own tests pass afterwards. Both are far closer to production shape than a multiple-choice set, and both are correspondingly more expensive to run.

Reading a published number

Four questions make a leaderboard figure interpretable, and a number that cannot answer them should not be used for a decision.

• Which split, and how many shots? Five-shot MMLU and zero-shot MMLU are separate quantities.
• How was it scored? Log-likelihood ranking, generation with a parser, or execution against tests.
• What were the decoding parameters? Temperature and top-p change pass@k materially, and they must be held fixed across any two models being compared.
• What is the uncertainty? On a 164-item or 1,319-item set, the confidence interval is wide enough to swallow most of the gaps that get reported as wins.

Contamination is the fifth question and the hardest to answer. Public benchmark items end up in pretraining corpora, and a model that performs unusually well on a widely republished set relative to fresh problems of equal difficulty is the classic signature. The only durable defence is a private held-out set that has never been published anywhere.

Sizing a run of your own

Reproducing a published number, or running any of these sets against a model you are considering, costs real tokens, and the cost scales with the sampling budget rather than with the item count. HumanEval at pass@10 is 1,640 generations, not 164. The Pass@k and evaluation cost calculator on this site works out the generation count, the token spend and the worst-case margin of error for a given dataset size and k, which is usually enough to decide whether a full run is worth it before starting one.

Once the numbers are in, the next two questions are which harness to run them through and whether the grader can be trusted. Both are covered separately: the comparison of LLM eval frameworks covers the harnesses that implement these sets, and the guide to LLM-as-a-judge bias covers what happens once a model, rather than a unit test, is doing the grading.

The short version

MMLU measures recall across subjects and is scored by option ranking. HumanEval measures functional correctness and is scored by executing code. GSM8K measures multi-step arithmetic and is scored by exact match on a final number. None of the three is a general capability score, all three are small enough that reported gaps are frequently inside the error bars, and all three are old enough to be present in training data. They are useful as a coarse filter and misleading as a ranking.