Metrics
Pass@k Metrics Explained: Formula, Estimator, Pitfalls
Pass@k is the chance at least one of k samples is correct. How the unbiased estimator works, why pass@1 is th…
Evals ML: LLM evaluation methodology and benchmarks
Model answers, measured.
Evals ML
Independent · 2026
Independent · 2026
Benchmarks
MMLU Benchmark Explained: Scoring, Splits, Failure Modes
MMLU is 14,042 four-option questions over 57 subjects, scored as accuracy. How the splits, prompts and scoring work, and where the number misleads.
Read the article →Latest guides
Eval Tooling
LLM Evaluation Frameworks Compared: Task Fit and CI Gates
Compare LLM evaluation tools for CI gates, reproducible runs, judge control, trace linkage and licensing, wit…
Research
GUI-Primitives: Spatial Reasoning Failures in Vision Models
GUI-Primitives tests spatial reasoning in vision-language models for GUI grounding, finding weak localization…
Benchmarks
LLM Benchmarks Explained: MMLU, HumanEval, GSM8K
What MMLU, HumanEval and GSM8K actually contain, how each score is computed, and the point where a headline b…
Model Grading
LLM-as-a-Judge Bias: How to Detect and Correct It
Position, verbosity and self-preference bias in model graders, how to measure judge agreement against humans,…
Eval Design
LLM Evaluation Design: From Task to Regression Suite
How to design an LLM evaluation: task-grounded eval sets, grader choice, sampling variance, and the contamina…
Start here
Evals ML is a reference for measuring language models: what the standard benchmark suites actually contain, how much confidence a published score carries, which harness fits which kind of task, and how to keep a model grader honest. Every figure quoted here is attributed to the paper or repository it came from.
Benchmarks
LLM benchmarks explained
MMLU, HumanEval and GSM8K: what is inside each set, how each is scored, and where the number stops being useful.
Methodology
LLM evaluation design
Building an eval set from the failures you care about, choosing a grader, and turning it into a regression suite that catches real change.
Tooling
LLM eval frameworks compared
lm-evaluation-harness, OpenAI Evals, HELM, Inspect, promptfoo, DeepEval and Ragas, compared on the task shapes each one fits.
Model grading
LLM-as-a-judge bias
Position, verbosity and self-preference bias, how to detect each, and how to calibrate a judge against human labels.
Pass@k and evaluation cost calculator
Generation count, token spend and worst-case margin of error for a given dataset size and sampling budget. Runs in the browser, no signup.
All guides · Browse by topic · How these articles are researched