A Practical Field Guide to AI Benchmark Scores (Part 2 of 3)
Knowing that benchmark scores depend on their test setup is only the first step. The next challenge is recognizing what different benchmark families actually measure—and where their evidence ends.
This field guide maps six useful categories that appear in model announcements and provides a repeatable way to read them together. The categories are practical rather than exhaustive, and some benchmarks span more than one.
Part 2 of 3 · Explore the complete series
- Part 1 — Why AI Benchmark Scores Are Harder to Read Than They Look
- Part 2 — A Practical Field Guide to AI Benchmark Scores (you are here)
- Part 3 — When Public Benchmarks Aren't Enough
Navigate this field guide
Five choices inside every evaluation
Every evaluation contains five design choices. Together, they determine what the final score reflects. Two reports can use the same benchmark name while differing on every one of them. (1)
- A benchmark name identifies a task family, not necessarily a fully specified protocol. Version, tools, prompt, grader, and budget may still differ.
---
title: "What Every Benchmark Score Reflects"
---
flowchart TB
T["Task Set"] --> S["Benchmark<br/>Score"]
C["Test Conditions"] --> S
F["Response Format"] --> S
G["Grading Method"] --> S
M["Reported Measure"] --> S
classDef blue fill:#e3f2fd,stroke:#0066cc,stroke-width:2px;
classDef green fill:#e5ffe5,stroke:#388e3c,stroke-width:2px;
classDef yellow fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
classDef orange fill:#ff9f43,stroke:#e67e22,stroke-width:2px;
classDef teal fill:#95e1d3,stroke:#16a085,stroke-width:2px;
classDef pink fill:#f38181,stroke:#e74c3c,stroke-width:2px;
classDef red fill:#ffe5e5,stroke:#c0392b,stroke-width:2px;
classDef gray fill:#f5f5f5,stroke:#95a5a6,stroke-width:2px;
class S blue
class T green
class C green
class F green
class G green
class M green- 1. The task set
-
What the model must do: answer questions, write code, fix bugs, or navigate websites.
- 2. The test conditions
-
Which tools and resources are available, how long the system has, and how many attempts it may make.
- 3. The response format
-
What a valid output looks like: a selected answer, working program, completed form, or changed system state.
- 4. The grading method
-
Who or what determines correctness: an answer key, automated tests, human reviewers, or another AI model.
- 5. The reported measure
-
Accuracy, pass rate, task-completion rate, cost per completed task, or another metric.
The first two questions to ask
When you encounter a score, start with: What were the conditions? and How was correctness defined?
Benchmark families at a glance
| Category | Representative benchmarks | Strongest signal | Important limit |
|---|---|---|---|
| Broad knowledge | MMLU, MMLU-Pro, HLE | Structured factual knowledge across domains | Does not establish practical task performance |
| Scientific and mathematical reasoning | GPQA Diamond, FrontierMath, AIME, ARC-AGI | Multi-step reasoning toward derived answers | Does not test communication or real workflows |
| Coding and software engineering | HumanEval, SWE-bench Verified, Terminal-Bench 2.0 | Producing working software outputs | Results depend heavily on the scaffold and environment |
| Visual and multimodal reasoning | MMMU-Pro, MathVista, CharXiv | Interpreting charts, figures, and images | Does not measure image generation |
| Browsing and tool use | BrowseComp, OSWorld, τ-bench | Taking actions through software and tools | Measures a system, not only a model |
| Human preference and deployment qualities | LMArena, MT-Bench, HELM | Usefulness and appeal on open-ended responses | Preference is not the same as correctness |
---
title: "Benchmark Families Form an Evidence Portfolio"
---
flowchart TB
K["Knowledge"] -->|"contributes"| E["Evidence<br/>Portfolio"]
R["Reasoning"] -->|"contributes"| E
C["Coding"] -->|"contributes"| E
M["Multimodal"] -->|"contributes"| E
T["Tool Use"] -->|"contributes"| E
P["Preference"] -->|"contributes"| E
E -->|"informs"| D["Model<br/>Decision"]
classDef blue fill:#e3f2fd,stroke:#0066cc,stroke-width:2px;
classDef green fill:#e5ffe5,stroke:#388e3c,stroke-width:2px;
classDef yellow fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
classDef orange fill:#ff9f43,stroke:#e67e22,stroke-width:2px;
classDef teal fill:#95e1d3,stroke:#16a085,stroke-width:2px;
classDef pink fill:#f38181,stroke:#e74c3c,stroke-width:2px;
classDef red fill:#ffe5e5,stroke:#c0392b,stroke-width:2px;
classDef gray fill:#f5f5f5,stroke:#95a5a6,stroke-width:2px;
class K green
class R green
class C green
class M teal
class T teal
class P yellow
class E orange
class D blueThe table is the comparison view. The tabs below are the exploration view: open all six to see what each family measures, where it helps, and where it stops.
Knowledge benchmarks ask whether a model can answer structured questions across many domains. Humanity's Last Exam has revised and verified task sets, so a score should identify the exact version. (1)
-
Version labels matter whenever tasks are corrected, removed, or separately verified after the original release.
-
Representative benchmarks
- MMLU: multiple-choice questions from 57 subjects.
- MMLU-Pro: a reasoning-focused redesign with ten answer choices.
- Humanity's Last Exam: expert-level questions, including image-based items.
- Best evidence
- General knowledge and structured factual answering.
- Does not establish
- Open-ended task completion, tool use, or multi-step workflow reliability.
Reasoning benchmarks require a derived answer rather than isolated recall.
- Representative benchmarks
- GPQA Diamond: graduate-level science.
- FrontierMath: original expert-authored mathematics.
- AIME: competition mathematics.
- ARC-AGI: adaptation to unfamiliar grid rules.
- Best evidence
- Multi-step analytical difficulty and structured reasoning.
- Does not establish
- Clear explanation, tool operation, or end-to-end professional work.
Coding evaluations range from small functions to work inside real projects.
- Representative benchmarks
- HumanEval: self-contained function synthesis checked by tests.
- SWE-bench Verified: issue resolution in real repositories.
- Terminal-Bench 2.0: 89 command-line tasks in controlled environments.
- Best evidence
- Producing executable, testable software outputs.
- Does not establish
- Performance on your codebase, tools, conventions, or constraints.
The reference guide explains why SWE-bench variants are not interchangeable.
Multimodal benchmarks combine text with charts, figures, diagrams, or screenshots.
- Representative benchmarks
- MMMU-Pro: domain knowledge combined with visual input.
- MathVista: quantitative reasoning over charts and diagrams.
- CharXiv Reasoning: interpretation of scientific charts.
- Best evidence
- Reading and reasoning from visual information.
- Does not establish
- Image-generation quality; interpretation and creation are distinct capabilities.
Agentic benchmarks require a system to take actions through browsers or software. (1)
-
The score reflects the whole system: model, tools, prompts, permissions, and coordinating software.
-
Representative benchmarks
- BrowseComp: sustained web research.
- OSWorld: tasks across real web and desktop applications.
- τ-bench: multi-turn tool use that changes database state.
- Best evidence
- Research, automation, and computer-operation workflows.
- Does not establish
- Equivalent performance with your data, permissions, tools, or product configuration.
Preference evaluations examine which open-ended responses people or graders favor. A model-as-judge evaluation makes the grader model part of the protocol. (1)
-
The grader's preferences and errors can influence the result, so its identity and grading prompt matter.
-
Representative evaluations
- LMArena: anonymous response pairs and user votes.
- MT-Bench: multi-turn conversations scored by an AI grader.
- HELM: a framework spanning multiple capabilities and deployment qualities.
- Best evidence
- Clarity, style, and perceived usefulness that answer keys may miss.
- Does not establish
- Factual correctness. Length, confidence, and presentation can influence preference.
How providers choose which benchmarks to highlight
---
title: "How Providers Select Benchmarks"
---
flowchart LR
N["Release<br/>Positioning"] -->|"selects"| H["Highlighted<br/>Evidence"]
N -->|"leaves unreported"| O["Other<br/>Categories"]
H -->|"becomes"| R["Announcement<br/>Scorecard"]
O -.->|"absent from"| R
classDef blue fill:#e3f2fd,stroke:#0066cc,stroke-width:2px;
classDef green fill:#e5ffe5,stroke:#388e3c,stroke-width:2px;
classDef yellow fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
classDef orange fill:#ff9f43,stroke:#e67e22,stroke-width:2px;
classDef teal fill:#95e1d3,stroke:#16a085,stroke-width:2px;
classDef pink fill:#f38181,stroke:#e74c3c,stroke-width:2px;
classDef red fill:#ffe5e5,stroke:#c0392b,stroke-width:2px;
classDef gray fill:#f5f5f5,stroke:#95a5a6,stroke-width:2px;
class N blue
class R blue
class H green
class O grayProviders usually lead with benchmarks that support a release's intended positioning. A coding agent may lead with SWE-bench. A research product may emphasize BrowseComp. A product aimed at professional workflows may highlight specialized domain tests.
The announcement is therefore a curated portfolio, not a comprehensive audit. Missing categories can be informative, although omission alone does not prove poor performance. It identifies an area where the announcement provides no evidence.
Interpret omissions carefully
It is reasonable to say that a release provides no evidence for an omitted capability. It is not reasonable to claim that the model necessarily performs poorly there without additional data.
Why might a benchmark category be absent?
- Editorial reasons
- Limited announcement space.
- A release positioned around one use case.
- Evaluation reasons
- No comparable run was completed in time.
- Protocol differences made the result difficult to present fairly.
- Performance reasons
- The provider may have had a less competitive result.
The announcement alone cannot tell you which explanation applies.
Reading benchmarks together
No single benchmark tells the full story. A small portfolio spanning different capability categories is more informative than one headline score.
A model might perform well on MMLU-Pro but poorly on SWE-bench, suggesting that broad knowledge does not consistently translate into working software. Another model might rank highly on LMArena while remaining inconsistent on reasoning tests, suggesting that it is compelling to read but not always safe to act on.
---
title: "A Balanced Evidence Portfolio"
---
flowchart LR
CR["Capability<br/>Evidence"] -->|"what it knows"| D["Model<br/>Decision"]
SE["System<br/>Evidence"] -->|"what it can do"| D
UE["Experience<br/>Evidence"] -->|"how it communicates"| D
PC["Personal<br/>Checks"] -->|"fit for your work"| D
classDef blue fill:#e3f2fd,stroke:#0066cc,stroke-width:2px;
classDef green fill:#e5ffe5,stroke:#388e3c,stroke-width:2px;
classDef yellow fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
classDef orange fill:#ff9f43,stroke:#e67e22,stroke-width:2px;
classDef teal fill:#95e1d3,stroke:#16a085,stroke-width:2px;
classDef pink fill:#f38181,stroke:#e74c3c,stroke-width:2px;
classDef red fill:#ffe5e5,stroke:#c0392b,stroke-width:2px;
classDef gray fill:#f5f5f5,stroke:#95a5a6,stroke-width:2px;
class D blue
class CR green
class SE teal
class UE orange
class PC orangeThe portfolio works by triangulation: no one evidence type is asked to answer a question it was not designed to measure.
Go deeper in the companion reference
Use this checklist when reading a scorecard:
- Identify the evidence
- Capability category represented.
- Exact model and benchmark version.
- Evaluation date.
- Inspect the protocol
- Tools, scaffold, budgets, and retry limits.
- Grader and definition of correctness.
- Conditions sufficiently similar for comparison.
- Find the boundary
- Important capability categories not reported.
- Difference between one headline score and a portfolio.
- Similarity between tested tasks and your own work.
The final question leads to the third post. Even a carefully selected benchmark portfolio reports performance on public tests, not on your particular tasks.
References and further reading
Open the complete reference list
Knowledge and reasoning
- Hendrycks et al., “Measuring Massive Multitask Language Understanding”
- Wang et al., “MMLU-Pro”
- Phan et al., “Humanity's Last Exam”
- Rein et al., “GPQA”
- Glazer et al., “FrontierMath”
- Chollet, “On the Measure of Intelligence”
- Chollet et al., “ARC-AGI-2”
Coding, multimodal, and agentic evaluation
- Chen et al., “Evaluating Large Language Models Trained on Code”
- Jimenez et al., “SWE-bench”
- OpenAI, “Introducing SWE-bench Verified”
- “Terminal-Bench”
- Yue et al., “MMMU”
- Yue et al., “MMMU-Pro”
- Lu et al., “MathVista”
- Wang et al., “CharXiv”
- Wei et al., “BrowseComp”
- Xie et al., “OSWorld”
- Yao et al., “τ-bench”
- Zheng et al., “MT-Bench and Chatbot Arena”
- Stanford CRFM, HELM
Provider announcements
- OpenAI, “GPT-5.6”
- Anthropic, “Claude Fable 5 and Claude Mythos 5”
- Anthropic, “Introducing Claude Sonnet 5”
- Google, “Gemini 3.5”
- Alibaba Cloud, “Qwen3.7”
- Mistral, “Remote agents in Vibe”
Project guides
- Reference guide for this series—benchmark details, SWE-bench variants, and provider-selection analysis.
- Full guide: Reading AI Model Benchmarks and Building a Personal Evaluation—the complete reference with primary sources.
Continue to Part 3: When Public Benchmarks Aren't Enough or revisit Part 1: Why AI Benchmark Scores Are Harder to Read Than They Look.