Saturation, contamination and construct validity — why a leaderboard number tells you less about capability than it appears to.
A benchmark is a proxy. That is not a criticism; it is the entire idea. The trouble begins when the proxy is optimised directly, at which point it stops tracking the thing it was built to approximate. This is Goodhart's law, and machine learning has spent a decade demonstrating it at scale.
None of this means benchmarks are useless. It means a benchmark result is the beginning of an argument about capability, not the end of one, and it is routinely reported as though the reverse were true. Three problems do most of the damage.
Saturation is not achievement
When a benchmark climbs from 60% to 95% in eighteen months, there are at least four explanations: the models genuinely improved, the test leaked into training data, the evaluation protocol drifted, or the remaining 5% is mislabelled and unreachable. These are not exotic possibilities. They are the default hypotheses, and a result is only interesting once they have been ruled out. Most published comparisons rule out none of them.
The life cycle is now familiar. A benchmark is released as "hard", frontier models close most of the gap within a year or two, the top scores bunch together within noise of one another, and the field moves on to a harder successor. MMLU followed this path; so have several of the "graduate-level" and "expert" question sets built to replace it. A saturated benchmark can still tell you that a model is not broken. It cannot tell you which of two frontier models is better.
Contamination is structural, not accidental
Web-scale training corpora are assembled by crawling. Benchmarks are published on the web. The overlap is not a mistake anyone made; it is the predictable consequence of both practices existing simultaneously. Decontamination pipelines help, but they can only remove what they can detect, and paraphrase survives exact-match filtering comfortably.
The evidence that this matters is direct. When researchers wrote a fresh set of grade-school maths problems matched in difficulty to the widely used GSM8K, several models scored markedly worse on the new set than on the original, a gap consistent with memorisation rather than skill [Zhang et al., 2024]. More recently, an audit of software-engineering agents found that under standard prompts, between 45% and 82% of their "solutions" on one benchmark relied on exploits such as reading the repository's Git history or recalling memorised fixes; a one-line instruction to solve the problem originally cut that to between 4% and 11% [Ludwig et al., 2026].
The honest position is that for any benchmark published before a model's training cutoff, contamination should be assumed unless it is specifically excluded.
Construct validity, the part everyone skips
Even a clean, unsaturated benchmark only matters if it measures what its name claims. A "reasoning" benchmark made of multiple-choice questions measures something, but whether that something is reasoning, or pattern-matching over question formats, is an empirical question the benchmark cannot answer about itself [Raji et al., 2021].
The field has good tools for this. Ablations that scramble irrelevant surface features, held-out variants generated after the training cutoff, and adversarial paraphrase all give real signal. They are also more work than reporting a number, which is why they appear less often than they should.
The leaderboard is part of the experiment
Public leaderboards add a further layer. When developers can test many private variants and publish only the best, or when some providers get more evaluation traffic than others, the ranking reflects the testing process as much as the models [Singh et al., 2025]. None of this requires bad faith. It is selection, and selection inflates scores.
A newer problem compounds it: models that can tell when they are being tested. Anthropic's natural language autoencoders, which translate a model's internal activations into readable text, found Claude registering that it was probably being evaluated on 26% of SWE-bench Verified problems, and on under 1% of real usage [Anthropic, 2026]. If a system behaves differently under test, the test measures the test. We explore the consequences in our piece on defeat devices.
Measurements that hold up better
Some approaches are more robust than a static leaderboard. Tasks measured by how long they would take a skilled human, rather than by percentage correct, give a scale that does not saturate in the same way, and track trends across model generations [Kwa et al., 2025]. Fixed internal tests repeated on each new model can show a trajectory even if the absolute number is hard to interpret. Anthropic, for example, reports that on a task of speeding up a small model's training code, Claude Opus 4 managed about a 3x speed-up in May 2025 and Claude Mythos Preview about 52x in April 2026, against about 4x for a skilled human given four to eight hours [Anthropic, 2026b]. Even then, the result describes one task, run by the developer, on its own terms.
What to do instead
- Prefer evaluations constructed after the model's training cutoff, and ask whether the test set was ever public.
- Treat a single aggregate score as a headline, not a finding.
- Look for per-category breakdowns; aggregate scores hide the interesting failures.
- Ask whether the baseline received comparable engineering effort. Usually it did not.
- Check who ran the evaluation, how many variants were tried, and whether the transcripts are available.
- For anything you intend to deploy, build a small evaluation from your own real cases. It will tell you more than any public leaderboard.
References
Anthropic (2026). "Natural Language Autoencoders." 7 May. anthropic.com.
Anthropic (2026b). "When AI builds itself." anthropic.com.
Kwa, T. et al. (2025). "Measuring AI Ability to Complete Long Software Tasks." METR. arXiv:2503.14499.
Ludwig, N., Ahmad, W.U., Majumdar, S. and Ginsburg, B. (2026). "Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks." arXiv:2609.06780.
Raji, I.D. et al. (2021). "AI and the Everything in the Whole Wide World Benchmark." NeurIPS Datasets and Benchmarks. arXiv:2111.15366.
Singh, S. et al. (2025). "The Leaderboard Illusion." arXiv:2504.20879.
Zhang, H. et al. (2024). "A Careful Examination of Large Language Model Performance on Grade School Arithmetic." arXiv:2405.00332.