MMLU Benchmark Explained: Scoring, Splits, Failure Modes

MMLU benchmark explained in one line: 14,042 four-option multiple-choice test questions across 57 subjects, scored as plain accuracy, where random guessing gets 25% and the original paper estimated expert-level humans at about 89.8%. It measures knowledge recall under a fixed prompt, it has plateaued at the frontier, and roughly 6.49% of its questions contain errors.

Here is where that bites a platform team. A nightly eval posts a candidate checkpoint 0.8 points below production on MMLU, and someone has to decide whether it blocks the release. The answer depends on three things the headline hides: the scoring protocol, the averaging, and the error bar once questions are clustered by subject. For how MMLU sits next to HumanEval and GSM8K, see the three headline LLM benchmarks compared. This one goes down to the config level.

What is actually inside MMLU?

MMLU is 57 subject tasks of four-option questions with one keyed answer, grouped into STEM, humanities, social sciences and other. Hendrycks et al. had graduate and undergraduate students collect them from freely available online sources, including GRE and USMLE practice questions. Every subject has at least 100 test questions.

Split sizes depend on which copy you load, which matters when you try to reproduce someone else’s number:

The cais/mmlu dataset card also ships a 99,842-row auxiliary_train split. The paper’s reference points are still the useful scale: 25% random, 34.5% for unspecialized Mechanical Turk workers, 43.9% for few-shot GPT-3 175B, 48.9% for the 11B UnifiedQA, and an estimated 89.8% for experts. GPT-3 was also badly calibrated: in the zero-shot setting, confidence and accuracy were up to 24 points apart on some subjects.

How is an MMLU score computed?

An MMLU score is accuracy: the fraction of test questions where the model’s top-ranked option matches the key. The original protocol opens with “The following are multiple choice questions (with answers) about [subject].”, adds five dev exemplars, compares the probabilities the model assigns to the tokens A, B, C and D, and takes the highest. Three choices around that definition each move the number.

What gets scored. Letter log-likelihood, full-answer log-likelihood (cloze-style), or free generation with a parser. The Hugging Face Open LLM Leaderboard team reran LLaMA 65B through three implementations in June 2023 and got 0.636 on the original code, 0.637 on Stanford’s HELM, and 0.488 on the EleutherAI harness as it stood in January 2023. Those are independent reruns, not the vendor’s reported figure, and model rankings shifted along with the absolute scores. The harness’s default mmlu task now ranks letters like the original, with mmlu_continuation and mmlu_generative kept as separately named variants.

Shots and template. Five-shot from the dev split is the reference protocol. Zero-shot, chain-of-thought and chat-templated runs are separate measurements, and the MMLU-Pro authors found MMLU scores move 4-5% across 24 prompt styles.

Averaging. Micro-averaging weights every question equally; macro-averaging weights every subject equally. The harness mmlu group sets weight_by_size: True, which is micro. Subject sizes vary, so the two disagree, and the choice belongs in the run metadata.

The metric that matters: paired delta, clustered by subject

For a release gate, track the per-question difference between candidate and baseline with a standard error clustered by subject, not two independent accuracies. Both models answered the same questions, and those questions arrive in 57 correlated blocks. A side-by-side accuracy comparison ignores both facts.

Naive arithmetic first. At 85% accuracy over 14,042 questions, the binomial standard error is √(0.85 × 0.15 / 14,042) ≈ 0.30 points, a 95% interval of about ±0.59. A single 100-question subject gives ±7.0, which is why per-subject leaderboards are mostly noise.

Evan Miller’s “Adding Error Bars to Evals” supplies both corrections. Clustering widens the interval honestly: on the data the paper analyses, clustered standard errors can run more than 3x the naive ones. Pairing narrows it honestly, because question-level scores correlate across models, which the paper calls a “free” reduction in variance. The same discipline applies to sampled metrics like pass@k and its unbiased estimator.

Wiring it up

Read the task config before trusting the number. This is the harness’s default MMLU template, abbreviated:

dataset_path: cais/mmlu
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: answer
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true

first_n means the same five exemplars every run, so few-shot selection adds no variance. The comparison job pulls per-question scores from the sample logs via simple_evaluate, computes the paired, clustered delta, and logs it to MLflow:

# One checkpoint per CI job in practice; inlined here for brevity.
import lm_eval
import mlflow
import numpy as np

VLLM_ARGS = "pretrained={ckpt},tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8"


def run_mmlu(ckpt: str) -> tuple[float, dict]:
    out = lm_eval.simple_evaluate(
        model="vllm",
        model_args=VLLM_ARGS.format(ckpt=ckpt),
        tasks=["mmlu"],
        num_fewshot=5,
        batch_size="auto",
        log_samples=True,
    )
    per_q = {
        (task, row["doc_id"]): row["acc"]
        for task, rows in out["samples"].items()
        for row in rows
    }
    return out["results"]["mmlu"]["acc,none"], per_q


base_acc, base_q = run_mmlu("your-org/model-prod")
cand_acc, cand_q = run_mmlu("your-org/model-candidate")

keys = sorted(base_q.keys() & cand_q.keys())
d = np.array([cand_q[k] - base_q[k] for k in keys])
subject = np.array([k[0] for k in keys])
n = len(d)

delta = d.mean()
se_paired = d.std(ddof=1) / np.sqrt(n)
resid = d - delta
cluster_sums = np.array([resid[subject == s].sum() for s in np.unique(subject)])
se_clustered = np.sqrt((cluster_sums ** 2).sum()) / n

with mlflow.start_run(run_name="mmlu-5shot-candidate-vs-prod"):
    mlflow.log_params({"task": "mmlu", "num_fewshot": 5,
                       "scoring": "letter-loglik", "averaging": "micro"})
    mlflow.log_metrics({
        "mmlu_acc_prod": base_acc,
        "mmlu_acc_candidate": cand_acc,
        "mmlu_delta": delta,
        "mmlu_delta_se_paired": se_paired,
        "mmlu_delta_se_clustered": se_clustered,
    })
    if abs(delta) > 2 * se_clustered:
        mlflow.set_tag("mmlu_gate", "investigate")

What you’ll see on the dashboard

Plot per-subject deltas sorted by size, with the clustered interval drawn as a band. Four shapes cover most nights.

• Noise. Aggregate delta inside two clustered standard errors, subject deltas scattered on both sides of zero, the largest swings in the smallest subjects. Ship it.
• Knowledge regression. Delta outside the band, and one group moves together: all of STEM, or the law and medicine subjects. This is the shape to check for after a narrow domain fine-tune.
• Format regression. mmlu_generative drops while letter-likelihood mmlu stays flat. The model still ranks the right answer but has stopped emitting a clean letter. Check the chat template and stop sequences before calling it lost knowledge.
• Contamination signature. A jump on public MMLU with no matching move on MMLU-Pro, MMLU-CF or a private holdout.

Caveats

Label noise caps the ceiling. MMLU-Redux re-annotated 5,700 questions across all 57 subjects, estimated that 6.49% of MMLU questions contain errors, and found errors in 57% of the analysed Virology questions. Near the top, gains can come from agreeing with wrong keys.

Contamination evades string matching. Yang et al. showed n-gram decontamination misses paraphrased and translated test items; Llama 2 13B trained on rephrased samples went from 54.8 to as high as 89.9 on MMLU. MMLU-CF responds with a closed test set, on which its authors report GPT-4o at 73.4% five-shot, an independent measurement rather than a vendor figure.

Saturation. MMLU-Pro moves to ten options across 14 domains and over 12,000 questions, cutting accuracy 16-33% versus MMLU and prompt sensitivity to about 2%. Its authors measured GPT-4o at 72.6%; that is an independent benchmark result, not a model-card number.

Compute. The multiple_choice output type issues one log-likelihood request per option, so a five-shot run is about 56,000 requests over long prompts. Schedule it nightly, not per commit.

Scope. Single-turn, English, text-only, multiple choice. MMLU says nothing about tool calls, long context or your product’s task; that takes an evaluation built from the task down.

FAQ

what is a good mmlu score

A good MMLU score is one that beats your current baseline by more than the error bar under the same protocol, not a fixed number. For scale, random guessing is 25%, the original paper measured unspecialized Mechanical Turk workers at 34.5%, and it estimated expert-level humans at about 89.8%. Label errors keep the practical ceiling below 100%.

is mmlu 5-shot or 0-shot

MMLU is normally reported five-shot, using the five dev-split questions for each subject as in-context examples. Zero-shot, chain-of-thought and chat-template variants are legitimate but separate measurements, and a five-shot figure next to a zero-shot one in the same table is not a comparison. Check the shot count before reading any MMLU number.

does mmlu measure reasoning

Mostly no: MMLU measures knowledge recall more than multi-step reasoning. The MMLU-Pro authors found chain-of-thought beat direct answering on their ten-option set and called that a stark contrast with findings on the original MMLU. For reasoning, pair it with GSM8K, GPQA, or tasks drawn from your own workload.

how do i run mmlu on my own model

Run it with EleutherAI’s lm-evaluation-harness: lm_eval --model hf --model_args pretrained=your-org/your-model --tasks mmlu --num_fewshot 5 --batch_size auto, or pass --model vllm to serve the model through vLLM. Add --log_samples --output_path results to keep per-question scores, which paired checkpoint comparisons need, and pin the harness version in the run metadata.

Related across the network

• LLM Benchmarks Explained: What the Numbers Mean and Miss — sentryml.com
• LLM Benchmark Fidelity for Production Model Selection — aisecbench.com