How to Read LLM Benchmarks: MMLU, GPQA, SWE-bench, LMArena, Humanity's Last Exam and Why Leaderboards Mislead

A practical guide to LLM evaluation: what the common benchmarks measure, why saturation, contamination and harness differences make leaderboard numbers unreliable, newer tests such as Humanity's Last Exam and METR time horizons, and how to build your own evaluation set.

PUBLISHED // UPDATED
DURATION // 06 MIN READ
How to Read LLM Benchmarks: MMLU, GPQA, SWE-bench, LMArena, Humanity's Last Exam and Why Leaderboards Mislead
FIG.01 // VISUAL DOCUMENTATION How to Read LLM Benchmarks: MMLU, GPQA, SWE-bench, LMArena, Humanity's Last Exam and Why Leaderboards Mislead

Public benchmarks are a useful starting point for comparing language models, but they are a poor guide to how a model will perform on your work. Many older tests are saturated, some test questions have leaked into training data, and small changes in prompts or scoring scripts move results. The most reliable evaluation combines a few harder, well-documented public benchmarks with a small test set built from your own tasks. This guide explains what the common benchmarks measure, why leaderboard numbers mislead and how to evaluate models yourself.

The saturation problem

The standard evaluation suite took shape between 2018 and 2021: MMLU for multidisciplinary knowledge, GSM8K for grade-school maths, HumanEval for coding and HellaSwag for commonsense reasoning. These were well designed for models of their era. They have since been heavily optimised against.

OpenAI’s GPT-4 technical report in 2023 reported 86.4% on MMLU and a simulated bar exam score around the top 10% of test takers. Since then, frontier models have pushed MMLU above 90%, which the authors of Humanity’s Last Exam cite as the reason new benchmarks are needed. When the top models are all near the ceiling, a few points’ difference says more about the question distribution, training data or test settings than about real capability.

This is Goodhart’s law applied to AI: when a measure becomes a target, it ceases to be a good measure. Labs maintain large internal evaluation suites for development, while public scores increasingly serve as marketing.

Newer and harder benchmarks

BenchmarkWhat it testsWhy it matters
MMLU-ProHarder, reasoning-focused version of MMLU with ten answer choicesAuthors report accuracy drops of 16–33% versus MMLU and less sensitivity to prompt wording
GPQAGraduate-level science questions written by experts to resist web searchTests expert reasoning rather than recall
Humanity’s Last Exam2,500 expert-written questions across dozens of subjects, including multimodal itemsDesigned as a frontier closed-ended academic benchmark
SWE-bench VerifiedFixing real GitHub issues so that tests passCloser to real software engineering work
ARC-AGINovel visual puzzles requiring abstraction from few examplesTargets generalisation rather than knowledge
METR time horizonsLength of tasks (measured in human time) a model can complete with 50% successLinks benchmark scores to real-world task length

METR’s paper offers a useful framing: rather than a percentage score, it measures how long a task (in human expert time) a model can complete half the time. The authors found that this horizon for frontier models had been doubling roughly every seven months since 2019, while noting uncertainty about how well the trend generalises.

Human preference and adversarial testing

LMArena (formerly Chatbot Arena) collects human votes between two anonymous models’ responses and aggregates them into Elo-style ratings. It captures something static benchmarks miss, namely which answers people prefer, and is harder to overfit than a fixed test set. But preference is not the same as correctness: people may favour longer, confident or well-formatted answers.

Red teaming and adversarial testing probe for failure modes such as hallucination, unsafe outputs and reasoning errors. Internal teams at labs and external organisations such as METR and Apollo Research publish evaluations of dangerous capabilities and behaviours. The most informative findings are often specific, reproducible failures rather than aggregate scores.

Domain-specific evaluation matters because a general knowledge score does not show whether a model can draft a sound contract, summarise clinical notes accurately or write code that passes review. Benchmarks built with domain experts give more actionable information for deployment.

What benchmarks miss

The attributes that matter most in production are often the least measured:

  • Reliability: Does the model behave consistently, or occasionally fail badly on edge cases?
  • Calibration: Does its confidence match its accuracy, and will it say when it does not know?
  • Robustness: Does performance hold up when inputs are noisy, unusual or adversarial?
  • Instruction following and format: Does it keep to the required structure over long outputs?
  • Cost and latency: A slightly better model may be impractical at scale.

Benchmarks you will see on model cards

  • MMLU (Hendrycks et al., 2020): Multiple-choice questions across 57 subjects (arXiv:2009.03300). Now saturated at the top end; harder variants such as MMLU-Pro exist.
  • GPQA (Rein et al., 2023): Graduate-level, “Google-proof” science questions written by domain experts (arXiv:2311.12022).
  • SWE-bench (Jimenez et al., 2023): Real GitHub issues that models must fix so that repository tests pass; SWE-bench Verified is a human-validated subset (swebench.com).
  • Chatbot Arena / LMArena (Chiang et al., 2024): Crowdsourced pairwise human preferences aggregated into Elo-style ratings (arXiv:2403.04132).
  • HELM (Liang et al., 2022): Stanford’s holistic framework evaluating models across many scenarios and metrics, not just accuracy (crfm.stanford.edu/helm).

Why leaderboard numbers mislead

  • Contamination: Test questions can leak into training data, inflating scores.
  • Saturation: When top models all score near the ceiling, small differences are noise.
  • Prompt and harness sensitivity: Few-shot format, sampling temperature, and scoring scripts change results; vendors’ self-reported numbers often use different settings from independent evaluations.
  • Preference vs. correctness: Arena-style rankings reward responses people like, which correlates with but is not the same as factual accuracy. Researchers have also raised concerns about selective private testing on such leaderboards (Singh et al., 2025, “The Leaderboard Illusion”, arXiv:2504.20879).

Building your own evaluation

  1. Collect 50–200 real examples from your use case with expected outputs or grading rubrics.
  2. Include hard and edge cases, not just typical ones.
  3. Fix prompts, temperature, and model versions so results are reproducible.
  4. Use automated checks where possible (exact match, unit tests, JSON schema validation) and model-graded or human review for open-ended tasks.
  5. Track cost and latency alongside quality, and re-run when providers update models.

Open-source harnesses such as EleutherAI’s lm-evaluation-harness help with standard benchmarks, but your own task set is what predicts production performance.

Our analysis

Treat benchmark scores as a first filter, not a verdict. Prefer newer, harder and well-documented benchmarks; check whether numbers are self-reported or independently measured, and under what settings. Then build a small evaluation set from your own tasks, measure quality alongside cost and latency, and re-run it whenever a provider updates a model. A modest, well-chosen internal test usually predicts production performance better than any public leaderboard.