LLM Eval CI: Reproducible Gates Before Deployment

The failure that sends a team looking for an LLM evaluation frameworks comparison is rarely having no evals. It is a prompt edit that moved the score a couple of points, a deploy that went ahead on a green dashboard, and tickets two days later showing the model answering in the wrong format. Nobody can say whether the change was real, the judge behind an API alias moved, or the run replayed cached responses. Which harness fits which task shape is covered in LLM eval frameworks compared. This post compares the same tools plus the tracing-first platforms on what decides whether a run can gate a deploy: reproducibility, CI fit, judge control, trace linkage and licence.

The comparison

This table ranks the same tools by what makes a run gate-able (reproducibility, CI fit, judge control and trace linkage), not by which task shape each one fits; that axis is in LLM eval frameworks compared.

Framework	Maintainer, licence	Reproducibility controls	CI fit	Judge control	Traces
lm-evaluation-harness	EleutherAI, MIT	Versioned tasks, fixed seeds, sample logs	Batch, not per commit	Exact match, log-likelihood	None
Inspect	UK AISI, MIT	Python tasks, eval logs	Scheduled batch	Model-graded scorers, explicit model	Eval logs
promptfoo	promptfoo, MIT	YAML config, response cache	GitHub Action posts to the PR	Grader pinned in config	Results matrix
DeepEval	Confident AI, Apache 2.0	pytest test cases	deepeval test run	G-Eval, DAG; judge per metric	Via Confident AI
Ragas	Vibrant Labs, Apache 2.0	Metric objects	Library call	Judge per metric	None
MLflow 3	Apache 2.0	Tracked runs, datasets	Library call in a run	Built-in judges, Guidelines	MLflow Tracing
Phoenix	Arize, Elastic License 2.0	Versioned datasets, experiments	Library call	Tool-calling judgments	OpenTelemetry
LangSmith, Braintrust, Weave	Hosted (Weave SDK Apache 2.0)	Immutable experiments	SDK call	Pairwise, online	Built in

The benchmark harnesses are for choosing a model, with the best reproducibility tooling here: lm-eval versions each task and bumps the version whenever a change affects scoring, because, as Lessons from the Trenches documents, minor variations in prompts and formatting can significantly change results. HELM belongs in the quarterly model-selection report, not the commit gate. The application frameworks are the gate. The platforms tie a score back to the trace that produced it, which no pure runner does.

The metric that matters

Track the paired regression delta on a pinned golden set, with a confidence interval, rather than the aggregate pass rate.

Freeze a golden set of N items and record its hash. Run baseline and candidate on the same items, scoring both on the same scale. For item i, d_i is candidate minus baseline. Report the mean of d, its standard error (standard deviation over the square root of N) and the 95% interval, mean plus or minus 1.96 standard errors. The gate: the interval must exclude a regression larger than your tolerance.

An aggregate pass rate compares two independent means, so item difficulty dominates the variance and a small real regression hides inside the interval. Pairing cancels it, because both systems answered the same question. Miller’s Adding Error Bars to Evals treats evals as experiments and gives the formulas for comparing two models and planning sample size. A difference smaller than its interval is not a result.

The second number is judge agreement with a human-labelled slice. Zheng and co-authors found GPT-4 as a judge reached over 80% agreement with human preferences, which is the ceiling. A judge well below it needs a better rubric first; the procedure is in LLM-as-a-judge bias.

Wiring it up

The commit gate, in promptfoo: two prompt files against one provider, the grader pinned so it cannot float with whatever credentials the runner holds, and the results posted to the pull request by the GitHub Action.

prompts:
  - file://prompts/baseline.txt
  - file://prompts/candidate.txt
providers:
  - openai:gpt-5-mini
defaultTest:
  options:
    provider: openai:gpt-5.6
tests:
  - vars:
      ticket: "My invoice shows two charges for August."
    assert:
      - type: contains-json
      - type: llm-rubric
        value: "Acknowledges the duplicate charge and states one next step"
        threshold: 0.8

The model-selection run, in lm-eval, seeds fixed, samples logged for post-hoc diffing, candidate served through vLLM with tensor parallelism across two GPUs:

lm_eval --model vllm \
  --model_args pretrained=/models/candidate,tensor_parallel_size=2,dtype=auto \
  --tasks gsm8k,hellaswag --num_fewshot 5 \
  --seed 0,1234,1234,1234 --batch_size auto \
  --log_samples --output_path runs/candidate/

Its CLI reference marks --limit as for testing only; a subsampled run is a smoke test.

What you’ll see

Plot the mean paired delta per commit with its interval as a band, and a zero line. Good looks boring: the band straddles zero and narrows as the golden set grows. A real regression is a band entirely below the tolerance line, and the per-item log shows failures clustered on one category of input. An instrument change is a step on a day with no prompt diff, usually a provider moving the model behind an alias, and it vanishes when the grader is pinned to a dated snapshot. A flat line across commits that changed the prompt is a cache replaying old responses, not stability.

Caveats

• Every framework inherits judge bias. MT-Bench documents position, verbosity and self-enhancement bias; G-Eval reports a Spearman correlation of 0.514 with humans on summarization and a bias toward LLM-generated text.
• Caching hides drift, then dumps it. promptfoo caches successful API responses for 14 days by default, so provider drift stays invisible until entries expire, then lands as one step. Gate with the cache off.
• Cost is items times prompts times judge calls, per commit. Pass@k multiplies the generation side by k. The Pass@k and evaluation cost calculator does the arithmetic.
• Cardinality. Per-item logs are essential for diffing and poison for a time-series database; keep item IDs out of Prometheus labels and join samples to aggregates by run ID.
• Label leakage. Golden items drift into few-shot examples and fine-tuning sets. Keep the set out of both and disclose the prompt engineering done.
• Licence and custody. Phoenix ships under the Elastic License 2.0, not Apache or MIT. LangSmith and Braintrust are hosted, so the golden set and every trace scored online live on their servers.

None of this is monitoring. Online scoring of production traffic without references is a drift signal for the monitoring metrics taxonomy, next to PSI and KS tests, not for the deploy gate. Promptfoo’s red-team mode is an attack suite; see the AI red-team engagement methodology for scoping one.