Reading AI Model Benchmarks—and Building a Personal Evaluation

Executive summary

AI benchmark scores often look simple: a model receives a percentage, a pass rate, or a position on a leaderboard. The score is the final result of a much larger test setup. The tasks, tools, prompts, time limits, and grading method all shape the result.

This guide develops three ideas:

Why benchmark context matters:
Early evaluations focused on knowledge and short answers.
Newer evaluations also measure reasoning, coding, browsing, tool use, computer control, and long-running work.
Each score describes performance under a specific set of test conditions.
What different benchmarks measure:
Knowledge tests, coding tests, visual tasks, browsing tasks, and human-preference studies examine different abilities.
A model can perform well in one category and poorly in another.
A useful comparison therefore draws on several relevant benchmarks.
How a personal test can add practical evidence:
Public benchmarks help identify promising models.
A repeatable personal test shows how those models perform on work that resembles the intended use.
The worked example in Section 3 evaluates instruction-following, scientific reasoning, explanation, visualization, interaction design, and technical reliability.

Detailed guidance for building a broader evaluation portfolio appears in Appendix A. A dated review of recent provider announcements appears in Appendix B.

1. Why benchmark context matters

1.1 What an AI benchmark measures

An AI benchmark is a structured test. It gives a model or AI system a set of tasks, applies defined rules, and converts the results into one or more scores.

Several parts of that sentence need clarification:

The thing being tested:
A model is the trained neural network, together with settings that affect how it generates a response.
A system includes the model and everything around it: prompts, tools, search, memory, safety rules, and software that manages the work.
A coding product that can inspect files, edit code, run tests, and retry is a system. Its result reflects more than the underlying model.
The tasks:
A benchmark is a standardized collection of questions or activities with a defined scoring method.
GPQA Diamond contains difficult science questions.
SWE-bench Verified asks an AI system to resolve software issues in real code repositories.
The test process:
An evaluation, often shortened to eval, is the full process used to test performance.
An evaluation may use a public benchmark, a private test set, or examples drawn from real work.
The surrounding software:
A harness or scaffold controls how the model receives instructions, context, and tools.
It may also control retries, stopping rules, and the order in which information is shown.
The result:
A metric is the value reported at the end, such as accuracy, pass rate, cost, response time, or task-completion rate.
An Elo rating summarizes results from head-to-head comparisons. A higher rating means that the system performed better against the particular group of systems in that comparison.
A grader or judge decides whether the response succeeded. The grader may compare a short answer, run tests, inspect a final application state, ask a human reviewer, or use another model.
The resources allowed during the test:
A token budget limits the units of text that the model can read or generate. Tokens are short pieces of words or punctuation used by the model's text-processing system.
A reasoning setting controls how much computation the product allows the model to spend before answering. Product names for these settings vary.

Anthropic describes a simple evaluation as a task with grading logic. An agent evaluation adds tools, an environment, an interaction loop, and changes that can be inspected after the work finishes. Anthropic, “Demystifying evals for AI agents”

A useful benchmark report therefore provides enough information to reconstruct the test:

the exact model or product version;
the benchmark and task-set version;
the tools and surrounding software;
the amount of time, computation, and retries allowed;
the method used to grade the result;
the metric and evaluation date.

Without these details, the score still describes a result, but the reader cannot tell whether it is comparable with another result.

1.2 How AI evaluation expanded

AI evaluation has expanded as models and AI systems have gained new capabilities. The development can be understood as four overlapping layers. Older layers remain useful and continue to appear in current model evaluations.

Layer 1: language, knowledge, and short answers

The earliest widely used tests focused on prompt comprehension and answer correctness.

Common task formats:
Classification: select a label, such as a topic or category.
Multiple choice: select one answer from a fixed set of options.
Exact-answer matching: compare the response with an accepted answer after removing minor formatting differences.
Reading comprehension: answer a question using a supplied passage.
Short code generation: write a small function from a specification.
What these tests show:
They provide a quick check of knowledge, language understanding, and basic problem solving.
Their scores are usually inexpensive to calculate and easy to reproduce.
Where their evidence is limited:
A fixed public test may eventually become familiar through training data or repeated development work.
A short answer reveals little about tool use, error recovery, or performance across a long workflow.

Layer 2: reasoning and constructed outputs

The next layer introduced tasks that require a derived answer or a working output. These tests include difficult science questions, mathematical problems, abstract puzzles, explanations, and executable programs.

How the grading changed:
A mathematics result can be compared with an accepted value.
Generated code can be executed against tests.
A written explanation can be reviewed for correctness, clarity, and completeness.
What this layer adds:
It provides evidence about multi-step reasoning and the construction of a result.
The tasks usually remain bounded, with a clear final answer or artifact.

Layer 3: tools, interaction, and environments

Newer systems can browse the web, edit repositories, use a terminal, call application programming interfaces (APIs), and operate graphical applications. An API allows one piece of software to request data or actions from another. Evaluations for these systems must examine actions as well as final answers.

Examples:
SWE-bench asks a system to modify a repository in response to a software issue.
OSWorld tests tasks performed through computer interfaces.
BrowseComp tests difficult information retrieval through web browsing.
Terminal-Bench tests command-line workflows inside controlled environments.
What affects the result:
Tool selection and tool quality.
The amount of context available to the model.
Recovery after an error or failed command.
Decisions about when to continue and when to stop.

Anthropic describes SWE-bench performance as a result of the model and agent system together. Anthropic, “Claude SWE-bench Performance”

Layer 4: reliable work over time

The latest layer studies whether a system can complete useful work over a longer period while staying within practical limits.

Capability measures:
percentage of tasks completed;
quality of the final output;
ability to recover from errors;
performance on tasks of different lengths or difficulty.
Operating measures:
cost per successful task;
elapsed time and number of model turns;
amount of human intervention;
frequency and severity of failures;
safe handling of permissions and reversible actions.

METR's task-completion time horizon is one example. It estimates the length of task, measured by human-expert completion time, at which an AI agent reaches a specified probability of success. The current suite concentrates on software engineering, machine learning, and cybersecurity. Its results should therefore be interpreted within those domains. METR, “Task-Completion Time Horizons of Frontier AI Models”

1.3 What changed in the test design

The four layers describe changes in both the tasks and the evidence collected.

Task selection:
Many early tests used a fixed public list of questions.
Current evaluations may use private tasks, regularly refreshed tasks, or tasks collected while the evaluation is running. These approaches reduce exposure to memorized examples.
Interaction:
A short test may contain one prompt and one response.
An agent test may contain many steps, tool calls, retries, and changes to an environment.
The longer format reveals planning, recovery, and persistence.
Output:
Earlier tests often expected a selected option or a short text answer.
Current tests may expect code, files, reports, actions, or a changed application state.
Grading:
Answer matching works well for short responses with one accepted answer.
Automated program tests and environment checks can verify whether code or an action sequence worked.
Human and model judges can assess open-ended quality, although their judgements may contain bias.
Resources:
Reasoning time, token limits, retries, and tools can all change performance.
Some tests also run several AI agents at the same time and combine their work.
A fair comparison records these allowances and keeps them consistent across candidates.
Reported measures:
Accuracy and pass rate remain useful.
Cost, completion time, consistency across changed conditions, human intervention, and safety provide additional evidence for deployment decisions.

Static benchmarks still provide common reference points and fast regression checks. A regression check repeats a known test to detect whether a newer version has become worse. Systems expected to browse, code, operate software, or work autonomously need evidence from the later evaluation layers as well.

1.4 Why two scores with the same benchmark name may differ

A statement such as “Model X scored 80%” leaves several questions unanswered. The model may have used a basic prompt, a specialized agent, browsing tools, several retries, or a large reasoning budget. Another result on the same benchmark may use different conditions.

A complete score can be read as the following record:

This model version, inside this product or test system, used these prompts and tools, received these time and computation limits, and completed this percentage of this task-set version according to this grader on this date.

OpenAI ran the same GPT-4 model with several scaffolds and reported SWE-bench Lite scores from 2.7% to 28.3%. OpenAI, “Introducing SWE-bench Verified”

The example shows why agent scores are often system scores. The benchmark name alone cannot establish comparability.

1.5 Why public scores may transfer poorly to practical work

Public benchmarks provide useful evidence, but several factors can narrow their relevance.

Data contamination:
Benchmark questions or close variants may appear in training data.
Familiarity with the questions can raise a score without improving broader capability.
Task-format specialization:
Training examples may resemble the benchmark's format or required skill.
The model may become especially effective on that task pattern.
Protocol optimization:
Prompts, tools, retries, reasoning budgets, and scaffolds may be tuned for the benchmark.
The optimized setup may differ from the product available to a customer.
Selective reporting:
A provider can emphasize favourable benchmarks, settings, and competitor comparisons.
Missing results may cover capabilities that were less central to the release or less favourable to the model.
Leaderboard adaptation:
Repeated submissions can reveal enough information to guide development toward a hidden test set.
Teams with greater access may receive more opportunities to adapt.
Grading limitations:
An answer key or unit test can reject a valid response or accept an incomplete one.
Human and model judges can be influenced by style, length, confidence, or presentation order.
Limited coverage:
Every benchmark samples a finite set of tasks.
A benchmark for science questions, code functions, or browser research represents only part of a larger job.

Research surveys document contamination risks. Research on live leaderboards also documents extensive private testing and unequal access. Survey of benchmark data contamination, “The Leaderboard Illusion”

Evaluation researchers use the term external validity for the broader question behind these limits: does the measured result predict performance in another setting? A benchmark can be scored correctly under its own rules while still providing weak predictions for a different workflow.

Provider announcements add another layer of selection. Each release highlights benchmarks that support its intended role. The detailed July 2026 comparison in Appendix B illustrates this pattern.

2. What different benchmarks measure

Models are built for different purposes and operating limits. Results vary because models are trained on different material, built in different ways, and allowed different amounts of computation while answering. Tools and product design can create further differences. Benchmarks therefore focus on different capabilities and use different forms of evidence.

Some benchmarks answer a narrow question, such as whether generated code passes a test. Others sample broad knowledge, visual reasoning, tool use, or human preference. A benchmark portfolio combines several relevant views of performance.

2.1 Five choices inside every evaluation

Every evaluation contains five main design choices:

Task set:
The questions or activities that the model must complete.
A task set may be public, private, fixed, or refreshed over time.
Test conditions:
The prompt, tools, context, time, token budget, number of attempts, and environment.
These conditions define the resources available during the test.
Response form:
A selected option, short answer, explanation, program, file, action sequence, or final system state.
Grading method:
Answer matching, executable tests, human assessment, model assessment, or a combination.
Reported measures:
Accuracy, pass rate, preference, cost, completion time, reliability, safety, or another outcome.

HumanEval and SWE-bench show how these choices can produce very different coding evaluations. HumanEval asks for small functions and checks them with tests. SWE-bench asks an agent to inspect and modify a repository. Both concern coding, but they test different scales of work.

2.2 Broad knowledge and academic understanding

This category tests knowledge across academic and professional subjects. It is useful when the work involves factual questions, education, or broad research.

Representative benchmarks:
MMLU / MMLU-Pro:
- Use multiple-choice questions across many subjects.
- Provide a broad view of academic and professional knowledge.
- MMLU covers 57 subjects. MMLU-Pro uses more reasoning-oriented questions, expands the answer set from four to ten choices, removes some noisy items, and reduces prompt sensitivity in the authors' experiments. MMLU paper, MMLU-Pro paper
BIG-bench / BIG-Bench Hard (BBH):
- Combine language, logic, science, mathematics, social reasoning, and unusual tasks.
- BIG-bench contains 204 tasks contributed by 450 authors across 132 institutions.
- The diversity is valuable, although one average score can hide large differences between tasks. BIG-bench paper
Humanity's Last Exam (HLE):
- Contains 3,000 difficult, expert-written questions across many fields.
- Uses multiple-choice and short-answer formats, including multimodal questions that combine text with images.
- HLE-Verified later identified noisy or ambiguous items and created verified, revised, and uncertain subsets. Reports should identify the exact HLE version. HLE paper, HLE-Verified paper
Useful evidence:
Breadth of knowledge.
Performance across several academic or professional domains.
Ability to answer difficult questions with verifiable results.
Important limit:
These benchmarks mainly score answers.
They provide little direct evidence about completing a workflow, operating a tool, or remaining reliable across many steps.

2.3 Scientific, mathematical, and abstract reasoning

This category tests difficult problems that require several reasoning steps or adaptation to an unfamiliar rule. It is useful when analytical correctness is central to the intended work.

Representative benchmarks:
GPQA Diamond:
- Uses graduate-level biology, chemistry, and physics questions.
- The questions were designed to challenge knowledgeable non-specialists even with web access.
- The Diamond subset is the commonly reported higher-quality slice. GPQA paper
FrontierMath and American Invitational Mathematics Examination (AIME) tests:
- Use difficult mathematics problems with answers that can be checked precisely.
- FrontierMath uses original problems written and reviewed by expert mathematicians. FrontierMath paper
ARC-AGI:
- Presents examples of visual grid transformations and asks the system to infer the rule.
- It provides evidence about adaptation to unfamiliar tasks.
- Comparisons should identify ARC-AGI-1, ARC-AGI-2, or a later interactive version because their protocols differ. Original ARC paper, ARC-AGI-2 paper
Useful evidence:
Multi-step scientific and mathematical reasoning.
Precise answers that can be independently checked.
Adaptation when the task does not follow a familiar template.
Important limit:
Advanced mathematics and small visual puzzles represent selected forms of reasoning.
They provide limited evidence about communication quality, workplace reliability, or autonomous task completion.

2.4 Coding and software engineering

This category tests code generation, debugging, repository work, and command-line execution. It is useful when selecting a coding assistant or engineering agent.

Representative benchmarks:
HumanEval:
- Asks a model to generate short functions from written specifications.
- Runs unit tests: automated checks that supply known inputs and compare the program's output with expected results.
- The small, isolated functions represent only a limited part of software engineering. HumanEval paper
SWE-bench family:
- Asks a system to resolve issues in real code repositories.
- Tests repository navigation, debugging, multi-file editing, and validation.
- The original SWE-bench contains 2,294 problems from 12 Python repositories. SWE-bench paper
- SWE-bench Verified contains 500 tasks screened by professional developers for clearer issue descriptions and fairer tests. OpenAI, “Introducing SWE-bench Verified”
Terminal-Bench:
- Asks an agent to complete command-line tasks in isolated software environments that contain the required files and tools.
- Terminal-Bench 2.0 contains 89 tasks inspired by real workflows.
- The result reflects the complete terminal agent, its tools, and its budget. Terminal-Bench paper
Useful evidence:
Functional code generation.
Repository-level editing and debugging.
Planning, tool use, iteration, and test-driven validation.
Important limit:
Scores depend strongly on the benchmark variant, scaffold, tools, context, time limit, the number and selection of repeated attempts, and the rules used to decide whether a task passed.
Two SWE-bench scores should be compared only when these conditions are sufficiently aligned.

2.5 Multimodal and visual reasoning

This category tests information that appears in both text and images. It is useful for screenshots, charts, diagrams, scientific figures, and scanned documents.

Representative benchmarks:
MMMU / MMMU-Pro:
- Use college-level questions that combine images, domain knowledge, and reasoning.
- Provide broad evidence about expert multimodal understanding. MMMU paper
MathVista:
- Contains 6,141 examples involving charts, diagrams, figures, and visual mathematics.
- Tests the combination of visual interpretation and quantitative reasoning. MathVista paper
CharXiv Reasoning:
- Uses charts from scientific papers.
- Its reasoning questions require information to be combined across complex visual elements.
- CharXiv contains 2,323 charts. CharXiv paper
Useful evidence:
Reading charts, scientific figures, screenshots, and diagrams.
Combining visual details with domain knowledge or mathematics.
Important limit:
Static visual questions cover only part of real document and interface work.
These tests do not directly measure image-generation quality or dependable computer control.

2.6 Browsing, tool use, and computer control

This category tests systems that take actions through browsers, APIs, tools, and graphical applications. It is useful for research agents, customer-service agents, and application automation.

Representative benchmarks:
BrowseComp:
- Contains 1,266 questions that require persistent web navigation to find difficult information.
- The final answers are short, which makes them easier to verify.
- Results depend on browser tools, web state, and the available budget. BrowseComp paper
OSWorld / OSWorld-Verified:
- Uses computer environments containing web and desktop applications.
- The original OSWorld contains 369 tasks involving applications, file operations, and multi-application workflows.
- Results depend on the environment setup and action interface. OSWorld paper
τ-bench family:
- Uses multi-turn conversations, domain policies, and tools that change database state.
- Provides a controlled approximation of customer-service and telecom workflows.
- The simulation is simpler than the policies and exceptions found in many organizations. τ-bench paper
Useful evidence:
Search strategy and source discovery.
Correct tool calls and state tracking.
Visual interface perception, action planning, and application control.
Important limit:
These are system evaluations.
Performance may change with different tools, permissions, data, user interfaces, or orchestration.

2.7 Human preference and broader deployment qualities

This category evaluates open-ended responses, interaction quality, and trade-offs that cannot always be reduced to answer accuracy.

Representative benchmarks and frameworks:
LMArena / Chatbot Arena:
- Shows two anonymous responses to a human user and asks which one is preferred.
- Provides evidence about perceived usefulness on open-ended prompts.
- Style, confidence, and response length can influence the vote.
MT-Bench:
- Uses multi-turn questions and a model judge.
- Supports scalable evaluation of chat and instruction-following.
- The judge can show position, verbosity, and self-preference biases. MT-Bench and Chatbot Arena paper
HELM:
- Evaluates several scenarios and reports several qualities: accuracy; consistent performance when conditions change; whether confidence matches correctness; efficiency; bias; and harmful or abusive language.
- Makes trade-offs and missing coverage visible.
- HELM is a framework and evaluation suite, so its results should not be treated as one universal score. Stanford HELM
Useful evidence:
Human-perceived usefulness and conversational quality.
Consistency when conditions change, efficiency, whether confidence matches correctness, and other deployment concerns.
Important limit:
A preferred answer may still contain factual, safety, or cost problems.
Qualitative grading requires careful judge selection and a clear rubric, which is a written scoring guide with defined criteria.

2.8 Choosing a small public benchmark portfolio

A short portfolio can provide a broad initial view of a general-purpose system.

One possible five-part portfolio:
MMLU-Pro for broad academic and professional knowledge.
GPQA Diamond for difficult scientific reasoning.
SWE-bench Pro or an independently maintained SWE-bench variant for repository-level software engineering.
OSWorld-Verified for computer use.
LMArena, read alongside a multi-metric framework such as HELM, for perceived usefulness and deployment trade-offs.
Useful substitutions:
Replace OSWorld with BrowseComp for research-heavy work.
Replace SWE-bench with Terminal-Bench for operational command-line work.
Add or substitute MMMU-Pro for document and image tasks.
Add FrontierMath or ARC-AGI for reasoning-focused work.
Add τ-bench for policy-bound tool workflows.
Add a domain-specific safety evaluation when errors or actions can cause material harm.

The portfolio provides coverage across different capability areas. It should be adapted to the intended work and should retain the protocol details described in Section 1.

A consolidated bibliography for these benchmark descriptions appears in Appendix D.

3. My personal evaluation: a scientific learning artifact

The development of AI benchmarks suggests a practical conclusion. Different tasks require different evidence, and public scores cannot represent every personal or organizational workflow. A repeatable private test can add evidence that is closely connected to the intended use.

I use the following test when assessing a new model. The broader method for turning recurring work into an evaluation portfolio appears in Appendix A. The test below is one component of such a portfolio.

3.1 What the test is designed to reveal

The task asks a model to analyse an intentionally ambiguous scientific scenario and create an interactive visual explanation. The combination of requirements supplies most of its evaluative value. The scientific scenario can be replaced in later versions.

Appendix C describes shorter one-prompt checks and their limitations. The test in this section uses a longer specification and explicit scoring so that several qualities can be inspected separately.

Instruction-following:
The model follows the requested planning process.
The result contains the required format, sections, visuals, controls, sources, and checks.
The model preserves constraints across a long prompt.
Reasoning under ambiguity:
The model identifies missing information before calculating.
Its assumptions are explicit, defensible, and connected to the result.
Equations, units, estimates, uncertainty, and sensitivity analysis are handled correctly.
Explanation and artifact quality:
A high-school student can follow the reasoning.
The diagrams, plots, animations, and controls clarify the science.
The HTML opens correctly, remains readable, and responds consistently to interaction.

The output is easy to inspect. Broken controls are visible. Weak explanation becomes apparent when the reasoning chain is hard to follow. Incorrect calculations can be compared across equations, plots, and conclusions. The scientific scenario can also be replaced while the surrounding evaluation structure remains stable.

3.2 How I run the test

The model starts by proposing a visualization plan and then proceeds to implement it.

I keep the following conditions consistent across all candidates models:

a fresh conversation for each model;
the same prompt and approval message;
the same reference pack when source access is controlled;
the same product surface and comparable reasoning settings;
the same time and cost limits;
the same browsers and browser-window widths for inspection;
saved copies of the plan, conversation, HTML file, screenshots, and scoring notes.

The reference pack for this scenario uses the biology, horse-weight, bird-measurement, lift, and wing-loading sources listed under Personal-evaluation sources. Supplying the same pack to every candidate keeps the factual starting point consistent and makes calculations easier to reproduce.

3.3 How I score the result

This rubric separates the evaluation into four distinct capabilities. A model can score poorly on instruction-following while still being recognized for excellent engineering, giving developers a true picture of the model's strengths and weaknesses.

1. Multi-Step Compliance & Attention (20 Points)

Evaluates the model's ability to hold constraints across a long prompt and sequence its output correctly.

Key Checks: Did it output the plan before the code? Did it hit the exact visual constraints? Did it format the appendix as requested?
Scoring: High scores mean the model has excellent context-window attention. Low scores indicate "attention decay" (forgetting meta-instructions once code generation begins).

2. Domain Rigor & Epistemology (30 Points)

Evaluates the factual accuracy, depth of scientific reasoning, and handling of uncertainty.

Key Checks: Are the scientific principles accurate and comprehensively applied? Does it address ambiguity, or does it confidently state an opinion as scientific law? Does it use real formulas or hallucinate toy models?

3. Pedagogical & Information Design (25 Points)

Evaluates how well the artifact actually teaches the concept.

Key Checks: Are visuals directly tied to the text? Is the flow logical? Is the reading level appropriate for a high schooler? Do the interactives build intuition?

4. Software Engineering (25 Points)

Evaluates the code quality, rendering, and dependency management.

Key Checks: Does it run offline (or utilize permitted dependencies correctly)? Is the code clean? Is it responsive? Are there unused imports or bloated scripts?

3.4 The evaluation prompt

The prompt below is reproduced as the test specification. It contains one technical ambiguity: it asks for a self-contained file while also permitting CDN libraries and external assets. The evaluation record should state whether the resulting file requires network access. Future runs can replace the CDN allowance with an offline requirement if full portability is essential.

Markdown

# Scientific Learning Artifact Generator

## Role

You are an expert scientific instructional designer with deep knowledge of physics, mathematics, engineering, and science communication.

Your task is to create rigorous, visual-first educational artifacts that help a curious high-school student understand both the answer and the reasoning behind it.

Every final artifact must be a single, self-contained interactive HTML file that combines concise explanation, equations, diagrams, animations, simulations, and interactive controls.

---

## Output Protocol

Before writing the HTML artifact, you must first generate a visualization plan and then proceed to implement it. No explcit user approval is required for the plan, before you begin building the artifact. The plan should be detailed enough that a reviewer can check whether the final artifact meets the requirements.

The plan must include:

1. The major concepts the artifact will teach
2. Every visualization you will create
3. The section where each visualization will appear
4. The instructional purpose of each visualization
5. The type of each visualization, such as diagram, plot, animation, simulation, interactive control, comparison, or recap
6. Any assumptions needed to resolve ambiguity in the question
7. A visual inventory table showing the total number of planned visuals by type

---

## Core Requirements

The artifact must prioritize visualization quality and quantity over text volume. A successful artifact should feel like an interactive illustrated explanation, not an essay with occasional figures.

Include, at minimum:

- One visualization for every major concept
- One visualization in every major section
- At least three interactive controls, such as sliders, toggles, buttons, or selectors
- At least two animated visualizations showing processes, relationships, or changes over time
- At least two comparative visualizations, such as side-by-side panels or overlaid plots
- At least two quantitative plots or charts
- At least one summary diagram that connects the whole reasoning chain
- A visual representation for every important equation
- A contextual visual for every numerical result
- Captions for every visualization explaining what it shows and why it matters

No visualization may be purely decorative. Each one must teach, clarify, compare, or validate something.

---

## Visual Density Standard

Use enough visuals that the learner can follow the explanation by scanning the figures and captions alone.

For a substantial scientific question, target 8-12 meaningful visualizations. Use more when the problem has multiple concepts, assumptions, or cases. Do not collapse distinct ideas into one crowded figure just to reduce count.

The final artifact must include a balanced mix of:

- Concept diagrams that define entities, variables, forces, geometry, or relationships
- Quantitative plots that show how variables scale or trade off
- Comparative views that make alternatives visually obvious
- Interactive explorations that let the learner test assumptions
- Animated or simulated views that show dynamics, flow, motion, or cause and effect
- A final visual recap that makes the conclusion memorable

Each visualization should be polished enough to stand on its own:

- Clear title
- Labeled axes, arrows, regions, or entities
- Units where applicable
- Consistent colors and legend semantics
- Direct annotation of the key takeaway
- Caption explaining what the learner should notice

If a section has more than two paragraphs without a visual, revise it.

---

## Reasoning Rules

If the problem is ambiguous, begin by identifying the ambiguities before solving.

For each ambiguity:

- Explain why it matters
- State the assumption you will use
- Justify the assumption
- Show how changing the assumption could affect the result

Never introduce assumptions silently.

If the problem is fully specified, present the answer early, then use the artifact to explain and justify it.

---

## Artifact Structure

Build the artifact in the following sequence.

### 1. Ambiguities and Assumptions

Explain the unclear parts of the question and the assumptions used for the analysis. Keep this section concise and connect each assumption to a later visualization.

### 2. Visual Setup

Introduce the scenario visually before doing calculations.

Show:

- The original question
- The major entities and variables
- The relevant physical principles
- The geometry, scale, and setup of the problem

### 3. Core Concepts

For each important concept:

1. State the idea plainly
2. Show it with a diagram, animation, plot, or simulation
3. Explain the equation or model behind it
4. Connect the visual result to the next concept

Use short explanatory text. Avoid long text-only passages.

### 4. Analysis and Sensitivity

Use visualizations to build intuition before and during calculation.

Show:

- How key variables affect the outcome
- How competing assumptions change the estimate
- How uncertainty propagates through the result
- Which factors dominate the conclusion

### 5. Results

Present the final answer visually, not only in prose.

The result section must include:

- A clear final conclusion
- A visual comparison of the competing sides or outcomes
- Numerical estimates shown in context
- A short explanation of the uncertainty

### 6. Visual Recap

End with an illustrated recap that summarizes the reasoning chain from assumptions to conclusion.

### 7. Appendix

Include an appendix at the end of the HTML artifact.

The appendix must include:

- A bibliography or reference list for factual claims, constants, biological estimates, physics formulas, datasets, or external assets
- A compact assumptions table listing each major assumption, its justification, and how sensitive the conclusion is to it
- A final visual inventory table counting how many diagrams, plots, animations, simulations, interactive controls, comparative views, equation visuals, numerical-result visuals, and recap visuals were included
- A short note confirming whether the artifact met the planned visual inventory, and explaining any differences from the approved plan

---

## Visualization Guidance

Use the representation that best teaches the concept.

Good options include:

- SVG diagrams for annotated geometry, force diagrams, scale comparisons, and conceptual sketches
- Canvas or CSS animation for motion, scaling, collision, flow, or time-dependent behavior
- Interactive controls for exploring parameter sensitivity
- D3.js, Plotly, Chart.js, or similar libraries for quantitative relationships
- Three.js or Babylon.js when a 3D view materially improves understanding
- Mermaid diagrams for logic flows, comparison trees, and conceptual relationships
- Tables with visual encoding such as color, size, position, or bars

Choose tools because they clarify the science, not because they are available.

---

## Visual Quality Bar

Every visual should answer at least one of these questions:

- What are the objects, variables, and relationships?
- What changes when a parameter changes?
- Which assumption or case is being compared?
- What does the equation mean geometrically or physically?
- Where does the final number sit relative to meaningful references?
- Why does the conclusion follow?

Avoid generic icons, unlabeled shapes, vague decorative illustrations, and charts that repeat numbers already stated in text. Prefer annotated visuals that expose mechanism, scale, uncertainty, or comparison.

---

## Technical Constraints

The final artifact must:

- Be a single self-contained HTML file
- Run directly in a browser without a build step
- Use CDN libraries when they improve clarity or interactivity
- Render math with MathJax, KaTeX, SVG text, or integrated plot labels
- Work on both mobile and desktop screens
- Keep all controls, labels, captions, and equations readable
- Avoid external assets unless they are loaded from stable public URLs

Before submitting, verify that the HTML renders standalone and that interactive and animated elements work.

---

## Quality Checklist

Before delivering the artifact, confirm that:

- The HTML is valid and self-contained
- Every major concept has a visualization
- Every major section has at least one visualization
- Every important equation has a matching diagram, plot, or annotation
- The artifact contains roughly 8-12 meaningful visuals for a substantial scientific question
- At least three controls are interactive
- At least two visualizations are animated
- At least two visualizations compare alternatives directly
- At least two visuals are quantitative plots or charts
- Captions explain why each visualization looks the way it does
- Numerical results are shown in context
- The final answer is presented visually
- A visual inventory table or checklist confirms the required mix was met
- The appendix includes references, assumptions, and the final visual inventory
- The artifact is understandable without advanced prior knowledge
- The artifact is usable as a shareable reference

---

## Question

Here is the question to answer:

`Who would win - 100 duck-sized horses against one horse-sized duck?`

3.5 Results from three models

3 distinct models were evaluated on the same prompt, with the same reference pack, and under the same conditions. Each model produced a single HTML artifact that was scored independently. The results are summarized below. The artifacts are found here: Model 1, Model 2, Model 3.

Below is a summary of the independent scoring.

Summary: All three models are excellent coders and designers, but they all suffer from severe attention decay, and two of them compromise scientific rigor to make their coding tasks easier.

Model 1: Gemini 3.1 Pro

Final Score: 74 / 100 (Solid Engineering, Flawed Compliance)

1. Compliance & Attention: 6 / 20
Finding: Massive attention decay. It completely ignored the instruction to output the text plan before the HTML. It also forgot the explicit instruction to state the main uncertainty in the conclusion, hiding it in an appendix instead.
2. Domain Rigor: 24 / 30
Finding: Excellent. It was the only model to fully grasp the biological realities of the Square-Cube law: a duck scaled up 500x would suffer catastrophic bone failure and lethal hyperthermia. Lost points: It spoke with absolute certainty ("physics guarantees a win") in a scenario built on behavioral ambiguities.
3. Pedagogical Design: 22 / 25
Finding: Great visual flow. The "Assumption Matrix" and the comparative scorecard brilliantly isolated the concepts.
4. Software Engineering: 22 / 25
Finding: The canvas-based JavaScript swarm simulation is highly impressive. Minor deduction for relying heavily on three separate external CDNs, making the file fragile.

Model 2: Claude Sonnet 5

Final Score: 62 / 100 (Good UI, Missed Core Science, Sloppy Code)

1. Compliance & Attention: 6 / 20
Finding: Failed the sequencing instruction. Placed a "plan compliance note" in the HTML appendix rather than generating the plan before the code.
2. Domain Rigor: 15 / 30
Finding: Fundamentally flawed physics. While it correctly calculated momentum and kinetic energy, it completely missed the structural limits of the Square-Cube law. Scaling a duck 500x without altering its bone density would crush it. Ignoring this to focus entirely on swarm coordination shows a lack of deep biological reasoning.
3. Pedagogical Design: 24 / 25
Finding: Masterful information architecture. The interactive charts linking fatigue to coordination were brilliant teaching tools.
4. Software Engineering: 17 / 25
Finding: Sloppy dependency management. It imported KaTeX CSS and KaTeX JS into the document head, but never used the library's syntax, relying entirely on plaintext for equations.

Model 3: GPT-5.6 Terra

Final Score: 68 / 100 (Flawless Execution, Fabricated Science)

1. Compliance & Attention: 6 / 20
Finding: Failed sequencing. It built the "Visualization Plan" directly into the educational webpage as a visible section, showing a total inability to separate meta-prompting from product generation.
2. Domain Rigor: 12 / 30
Finding: Scientific fabrication. To make its interactive bar charts update smoothly without relying on a charting library, it literally invented a generic math formula (horse = 100 * hm * coord...) and passed it off as a model. While it disclaimed it as a "toy model," faking the math in a scientific evaluation artifact is a critical reasoning failure.
3. Pedagogical Design: 25 / 25
Finding: The layout, pacing, typography, and visual hierarchy read like a professional New York Times interactive graphic.
4. Software Engineering: 25 / 25
Finding: Absolute perfection in front-end execution. It used zero dependencies, drawing complex visuals (including animations) entirely with pure, responsive, inline SVGs and vanilla JavaScript. It even implemented a CSS dark mode flawlessly.

Results Summary from the three models

When you look at these scores objectively, the story is clear and highly actionable for an AI researcher:

The "Plan First" instruction is a model-breaker. All three models failed to separate the planning text from the HTML output. This proves that once current LLMs are triggered to write a heavy block of code (like HTML), their attention state collapses, and they drag all preceding instructions into the code block.
UI competence masks scientific laziness. Model 3 wrote the best code by far, but it did so by faking the math so it wouldn't have to struggle with complex charting logic. Model 2 made pretty charts, but ignored complex biomechanics to do so.
Model 1 is the true winner for a Science test. Despite its formatting errors and use of CDNs, Model 1 was the only model that actually applied hard biological constraints (bone crushing, thermodynamics) to the prompt.

This decoupled rubric prevents the scores from being artificially suppressed, while clearly highlighting that none of these models are fully capable of balancing strict multi-step compliance, rigorous scientific logic, and complex software engineering simultaneously.

Appendix A. How to build a personal evaluation portfolio

The case study in Section 3 is one repeatable personal test. A broader portfolio uses several tasks to cover normal work, difficult cases, and costly failures.

NIST's AI Risk Management Framework recommends documented and repeatable evaluation under conditions similar to deployment. It also recommends measurement during operation. NIST AI RMF Core

A.1 Define the decision

Start with the decision that the evaluation must support.

Possible decisions:
Select a model for a recurring task.
Compare a new release with the current model.
Choose a reasoning setting or cost tier.
Approve a model update for production.
Identify work that still requires human execution.
Why the decision should come first:
It determines which strengths and failures are relevant.
It prevents an impressive but unrelated capability from dominating the comparison.

A.2 Describe representative work

List frequent, important, and difficult tasks. Each task record should describe the work and its practical constraints.

Inputs and outputs:
Record the information supplied to the model.
Define the required form of the result.
Working conditions:
Record the available tools, data, permissions, and context.
Define the time and cost limits.
Quality requirements:
Describe the characteristics that make the result useful.
List the errors that would require correction.
Failure consequences:
Separate inconvenient failures from serious failures.
Identify any financial, legal, medical, security, or reputational harm that could result.

For example, a product-comparison task may require current specifications, reliable sources, complete coverage, and a clearly supported recommendation. Invented specifications or stale prices may count as disqualifying failures.

A.3 Choose a small and varied task suite

A practical first portfolio can contain 5–15 tasks.

Normal tasks:
Represent the work that creates regular value.
Prevent the portfolio from focusing only on unusual challenges.
Difficult tasks:
Separate strong systems from weak systems.
Include ambiguity, incomplete information, or several dependent steps when those conditions occur in the real workflow.
Edge cases:
Target known failure modes.
Include high-cost mistakes and situations that require refusal, escalation, or confirmation.
Different forms of grading:
Include at least one task with an objectively checkable result.
Include at least one task that requires qualitative human review.
Hidden or changed variants:
Preserve the task structure while changing names, values, documents, or scenarios.
Reduce the value of memorizing a familiar prompt.

The micro-tests discussed in Appendix C can serve as quick checks. A serious decision should use several tasks and a predefined rubric.

A.4 Fix the test conditions

Use equivalent conditions for every candidate.

Keep constant:
task instructions and source material;
tool access and permissions;
reasoning setting, or a clearly documented set of compared settings;
time, token, attempt, and cost limits;
follow-up messages;
starting context;
browsers, software versions, and test environments.
Record for every run:
exact model and product version;
product surface, such as API, chat application, or coding agent;
prompts, outputs, logs, and files;
elapsed time, cost, and date;
errors, interventions, and retries.

Different product surfaces may supply different prompts, tools, and context. They should be recorded as separate system configurations even when they use the same underlying model.

A.5 Write the rubric before testing

A rubric should separate the major qualities of the output. Observable descriptions make the scores easier to apply consistently.

Score	General meaning
0	Missing, unusable, or materially wrong
1	Major errors or omissions; extensive correction required
2	Partly successful; several important corrections required
3	Meets the main requirement; limited correction required
4	Complete, accurate, clear, and ready for the intended use

Useful scoring categories:
factual accuracy;
instruction-following;
completeness;
reasoning and judgement;
clarity and presentation;
technical correctness;
cost and completion time;
safety and failure recovery.
Disqualifying gates:
A fabricated source may fail a research task regardless of the average score.
An unauthorized irreversible action may fail an automation task.
A materially incorrect high-stakes conclusion may fail a medical, legal, or financial task.

A.6 Review the evidence

Use automatic checks and human review where each is strongest.

Automatic checks:
valid file formats;
automated code checks (unit tests);
required sections and fields;
broken links;
accessibility rules;
numerical consistency.
Human review:
clarity and usefulness;
judgement under ambiguity;
source quality;
subtle factual errors;
whether the result is ready for its intended audience.

Blind the reviewer to model identity when practical. Important qualitative decisions can use two reviewers. Disagreements should be recorded and resolved against the rubric.

The final comparison should report category scores, failure notes, variation between runs, cost, and completion time. A single weighted total can hide important trade-offs, so the underlying categories should remain visible.

A.7 Run a supervised production pilot

A controlled test supports a shortlist. A limited pilot shows how the system performs with real users, data, and workflow constraints.

Track during the pilot:
completion rate;
correction time;
human interventions;
incidents and near misses;
cost and completion time;
user outcomes.
Repeat the evaluation when:
the model version changes;
prompts or tools change;
a policy or workflow changes;
production monitoring reveals a new failure mode.

The complete decision sequence is:

Text Only

Use case
  → requirements and unacceptable failures
  → relevant public benchmark portfolio
  → candidate models
  → same-protocol private evaluation
  → supervised production pilot
  → monitoring and re-evaluation

Appendix B. What recent provider announcements emphasize

The following section's prose, lists, tables, claims, and links are reproduced verbatim from the earlier draft. Its heading levels have been adapted to the appendix structure. It is retained as a dated example of how providers select benchmarks to support different release goals. Its comparison cut-off is 18 July 2026.

6. What recent provider announcements actually emphasize

6.1 Selection rule and date cut-off

“Latest release” is ambiguous because companies release flagship models, cheaper variants, specialized models, previews, and product updates on different dates. The comparison below uses the latest broadly relevant general-purpose model family or capability tier announced by 18 July 2026. Where the newest-by-date model is not the most capable, both are identified.

Provider	Broadly relevant release	Announcement date	Release story
OpenAI	GPT-5.6 family: Sol, Terra, Luna	9 Jul 2026	Performance per dollar, coding agents, professional workflows, computer use, science, and cyber
Anthropic	Claude Fable 5; Claude Sonnet 5 is newer but cheaper	9 Jun / 30 Jun 2026	Fable: highest-capability long-horizon work; Sonnet: lower-cost agentic execution
Google	Gemini 3.5 Flash	19 May 2026	Frontier-level agentic and coding performance at Flash speed
Qwen / Alibaba	Qwen3.7-Max	21 May 2026	Long-horizon agent foundation across coding, office work, MCP, and multiple scaffolds
Mistral	Mistral Medium 3.5	Model release 28 Apr; product announcement 22 May 2026	Open-weight, multimodal flagship for reasoning, coding, and long-running agents

Specialized releases after these dates—for example Google's Live Translate models or Mistral's OCR, formal-proof, and navigation models—are not treated as replacements for the general-purpose comparison.

6.2 OpenAI: GPT-5.6

OpenAI presents GPT-5.6 as three durable capability tiers: Sol as flagship, Terra as the balanced tier, and Luna as the lower-cost tier. The launch also introduces higher reasoning budgets and a multi-agent ultra setting. The announcement's central claim is more successful work per token, unit of time, and dollar. OpenAI, “GPT-5.6”

Benchmarks and evaluation families highlighted in the announcement include:

professional and agentic work: Agents' Last Exam, GDPval-AA v2, Big Finance Bench, internal management-consulting tasks;
coding: Artificial Analysis Coding Agent Index, SWE-Bench Pro, DeepSWE, Terminal-Bench 2.1;
browsing and computer use: BrowseComp, OSWorld 2.0, BenchCAD, AutomationBench;
science and health: GeneBench Pro, LifeSciBench, MedChemBench, HealthBench Professional;
academic reasoning: GPQA Diamond and FrontierMath;
multimodal work: MMMU-Pro and gdp.pdf;
tool use and context: Toolathlon, MRCR v2, GraphWalks;
cybersecurity: Capture-the-Flag challenges, SEC-Bench Pro, ExploitBench, and ExploitGym;
internal AI-research and self-improvement suites.

Interpretation: OpenAI is presenting GPT-5.6 as an end-to-end work system. The benchmark portfolio has moved far beyond MMLU-style question answering toward coding agents, professional deliverables, computer operation, science workflows, and capability-per-cost frontiers. Some reported evaluations are internal, and the highest ultra results may involve multiple agents; neither should be treated as a raw single-model comparison.

6.3 Anthropic: Fable 5, Mythos 5, and Sonnet 5

Anthropic launched Fable 5 and the restricted Mythos 5 on 9 June. They share an underlying model; Fable applies safeguards and may route some sensitive requests to Opus 4.8, while Mythos is a restricted configuration for trusted cyber and scientific work. Anthropic describes Fable as above its Opus class and intended for days-long coding and knowledge work. Anthropic, “Claude Fable 5 and Claude Mythos 5”

The Fable/Mythos announcement and linked system material emphasize:

long-horizon coding and production-code quality, including Cognition's FrontierCode/FrontierBench and customer coding suites;
finance and analytical work, including Hebbia's finance benchmark and customer evaluations;
visual reconstruction and vision-only interaction;
persistent memory and long-context game/task performance;
scientific hypothesis generation, genomics, and protein-design workflows;
cybersecurity evaluations such as Firefox exploitation, OSS-Fuzz, CyberGym, and CyScenarioBench;
automated alignment and safeguard testing.

Anthropic then released Sonnet 5 on 30 June. Sonnet 5 is newer by date but sits below Fable in capability and price. Its announcement focuses on plans, browsers, terminals, coding, tool use, and professional work, and gives special attention to BrowseComp and OSWorld-Verified cost-performance curves at different effort levels. Anthropic, “Introducing Claude Sonnet 5”

Interpretation: Fable is Anthropic's high-capability tier. Sonnet is the more economical, broadly deployed agentic tier. Anthropic also foregrounds safeguards as part of the product configuration. Fable and Mythos show how one underlying model can support different deployable systems.

6.4 Google: Gemini 3.5 Flash

Google introduced Gemini 3.5 by releasing Gemini 3.5 Flash as an agentic and coding model intended to combine frontier performance with Flash-class speed. The announcement says it exceeds Gemini 3.1 Pro on several agentic evaluations while delivering high output speed. Google, “Gemini 3.5: frontier intelligence with action”

The named benchmarks include:

Terminal-Bench 2.1 for command-line and agentic work;
GDPval-AA for professional work;
MCP Atlas for tool and MCP-based agent interaction;
CharXiv Reasoning for chart and scientific visual reasoning;
the Artificial Analysis Intelligence Index for a broader third-party composite.

The announcement also demonstrates collaborative subagents, long-running coding, generative interfaces, and multimodal visual creation.

Interpretation: Google's release story changed from the reasoning-heavy framing of Gemini 3.1 Pro and ARC-AGI-2 to a combination of reasoning, action, multimodality, and speed. That change within one provider supports the claim that benchmark selection follows the intended role of each release.

6.5 Qwen: Qwen3.7-Max

Alibaba describes Qwen3.7-Max as a proprietary “agent foundation” for coding, office automation, MCP integrations, and sustained autonomous execution. Its announcement includes unusually detailed harness notes and a demonstration involving approximately 35 hours and more than 1,000 tool calls. Alibaba Cloud, “Qwen3.7: The Agent Frontier”

The announcement reports or discusses:

coding: Terminal-Bench 2.0, SWE-bench Pro, SWE-bench Verified, SWE-bench Multilingual, SciCode, NL2Repo, QwenSVG, and an internal web-development benchmark;
agents and tools: MCP-Mark, MCP-Atlas, SkillsBench, BFCL-V4, QwenClawBench, ClawEval, and VITA-Bench;
office work: SpreadsheetBench and internal CoWorkBench;
kernels and long-horizon optimization: KernelBench L3 and a custom hardware-optimization task;
reasoning: GPQA Diamond, HLE, HMMT, IMOAnswerBench, Apex, and SuperGPQA;
multilingualism and context: WMT24++, MAXIFE, MMLU-ProX, and MRCR-v2;
simulated environments and strategy: QwenWorldBench and YC-Bench.

Qwen discloses that some tests use internal scaffolds, some use other models as judges, and some use modified or corrected task sets.

Interpretation: Qwen's announcement presents the agent system as the product. Its evidence includes many internal tasks, custom harnesses and judge choices. Cross-provider comparisons require matching these protocol details.

6.6 Mistral: Mistral Medium 3.5

Mistral Medium 3.5 is a 128B dense, multimodal, open-weight model with configurable reasoning effort and a 256k context window. Mistral positions it as the engine for remote coding agents and long-running “Work mode” tasks. Mistral, “Remote agents in Vibe. Powered by Mistral Medium 3.5”, Mistral model card

The public announcement prominently reports:

SWE-bench Verified for software engineering;
τ³-Telecom for multi-turn agentic tool use;
product-oriented evidence around long-running coding, parallel tools, structured output, and self-hosting efficiency.

Interpretation: Mistral's story is less about winning the largest collection of academic tests and more about an open-weight model that unifies instruction following, reasoning, multimodality, and coding at a deployable size. Licensing, self-hosting, hardware requirements, and price are part of the value proposition and may matter more to a buyer than a marginal benchmark difference.

6.7 What the five release strategies show

Provider	Dominant release narrative	Benchmark families selected to support it
OpenAI	End-to-end work and performance per dollar	Coding agents, professional workflows, browsing, computer use, science, cyber
Anthropic	Long-horizon capability with tiered safeguards and cost	Coding, finance, vision, science, cyber, search, computer use
Google	Fast frontier agents and multimodal coding	Terminal, professional work, MCP/tool use, scientific charts, composite intelligence
Qwen	A general foundation for diverse agent scaffolds	SWE, terminal, MCP, office, multilingual, context, kernels, simulated worlds
Mistral	Open and deployable unified agent model	SWE, domain tool use, system efficiency, self-hosting

The tables are not interchangeable. They are evidence for different product theses.

Appendix C. Informal micro-tests

An informal micro-test is a small and memorable task used for a quick capability check. Simon Willison's well-known example asks a model to:

Generate an SVG of a pelican riding a bicycle.

The task combines several visible requirements:

valid SVG code;
recognizable representations of a pelican and a bicycle;
a clear spatial relationship between the two objects;
plausible visual composition;
an output that a person can inspect immediately.

Willison chose an unusual composition partly to reduce the chance that a matching example already existed in training data. Simon Willison, “Pelicans on a bicycle”

The task works well as a demonstration. Its value as a serious evaluation is limited for several reasons:

it contains only one sample;
visual quality requires subjective judgement;
repeated runs may produce different results;
the prompt is now widely known and may appear in training or evaluation data;
the task covers creative SVG generation rather than the full range of model capabilities.

A stronger micro-test retains the task structure while adding variation and explicit grading.

Possible hidden variants:
an otter operating a loom;
a flamingo repairing a typewriter;
a tortoise riding a unicycle.
Separate scoring checks:
valid SVG;
required objects;
correct spatial relationship;
mechanical plausibility;
visual clarity and legibility.

The same principle applies to the scientific-artifact test in Section 3: keep the evaluation structure stable, vary the final scenario when useful, and inspect each major quality separately.

Appendix D. References and primary sources

Benchmark and evaluation research

Official model-release sources

Personal-evaluation sources

Appendix E. Historical editorial standard — superseded

Do not use this appendix for current edits. The canonical and standalone standard is 900-editorial-guide.md. This appendix is retained only as a record of the conventions used when the original research guide was drafted; where the two differ, 900-editorial-guide.md governs.

The historical standard below is preserved for provenance.

E.1 Audience and purpose

Write for an informed general reader:
Assume interest in AI models without assuming knowledge of evaluation research.
Define unfamiliar terms before relying on them.
Include a short example when a definition remains abstract.
Keep the article useful over time:
Place release-specific and frequently changing material in a dated appendix.
Keep durable concepts and methods in the main narrative.

E.2 Document structure

Use the main progression consistently:
Section 1 explains why benchmark context matters.
Section 2 explains what different benchmarks measure.
Section 3 presents the personal test and its results.
Appendices provide general methodology, dated examples, sources, and maintenance guidance.
Give each section one main purpose:
Introduce a category before listing examples.
Use the same order for repeated benchmark sections: purpose, examples, useful evidence, and limits.
Avoid repeating a claim in several places unless each occurrence adds necessary context.

E.3 Tone and sentence construction

Use a neutral and informative tone:
Avoid promotional language, rhetorical questions, conversational asides, and slogans.
Identify provider claims as provider-reported when independent evidence is unavailable.
Use connected, readable prose:
Prefer short paragraphs of two to four related sentences.
Vary sentence length so the text does not sound clipped or mechanical.
Add a plain-language explanation or example after a dense technical statement.
Keep one main idea in each sentence, while using transitions to show how adjacent ideas connect.
Avoid formulaic constructions:
Avoid false contrasts such as “not X but Y,” “X does not mean Y; it means Z,” and “X stayed the same; Y changed.”
Avoid transition formulas such as “the point is,” “this explains why,” “that is why,” “once this is clear,” and “the missing context is.”
Replace broad comparisons such as “X matters more than Y” with a scoped statement about the task, decision, or risk.
Control narrative voice:
Use neutral third-person prose in Sections 1 and 2.
First person is acceptable in Section 3 because it presents a personal evaluation practice.
Avoid second person in the article's explanatory prose. Direct instructions are acceptable inside the reproduced evaluation prompt.
Verbatim reproduced material may retain its original wording. Label the material clearly and exclude it from prose-style lint checks.
Use consistent language conventions:
Use British English with Oxford -ize spellings.
Preserve official benchmark names and quoted text.

E.4 Bullets, indentation, and tables

Use nested bullets for grouped information:
Begin with a bold heading or a backticked technical term.
Use the first indented level for the main explanation.
Use a second indented level only when it clarifies a real hierarchy.
Keep items grammatically parallel within one list.
Use tables selectively:
Tables work well for fixed score rubrics, repeated fields, and compact comparisons.
Introduce each table and explain its purpose.
Move long qualifications into nearby prose or bullets.
Avoid tables so wide that they become difficult to read on a small screen.
Use inline code formatting consistently:
Place benchmark shorthand, protocol labels, model settings, and literal commands inside backticks when the formatting improves recognition.
Use ordinary bold text for conceptual headings rather than code formatting.

E.5 Accessibility and terminology

Define technical language:
Expand abbreviations on first use unless the full name is rarely used.
Explain what a metric counts and what a higher score represents.
Explain terms such as scaffold, exact-answer matching, and model judge with a concrete example.
Reduce unnecessary reading effort:
Use descriptive headings instead of clever or rhetorical headings.
Keep paragraphs focused.
Use bullets for definitions, procedures, and repeated comparisons.
Provide a transition before moving from general concepts to detailed categories.

E.6 Evidence and claims

Prefer primary sources:
Use research papers, benchmark documentation, official model cards, and official release announcements.
Cite the source next to the claim it supports.
Record comparison conditions:
Include the benchmark version and task subset.
Include the model version and system configuration.
Include prompts, tools, reasoning settings, budgets, attempts, grader, metric, cost, and date when they affect the comparison.
State the scope of the evidence:
Separate reported results from editorial interpretation.
Describe important limits on generalization.
Remove claims that cannot be traced to a source, or label them as questions requiring verification.

E.7 Revision checklist

Before publication:

Remove unresolved editorial notes and square-bracket comments.
Confirm that the main narrative follows the why–what–personal-test progression.
Confirm that the general portfolio method remains in Appendix A.
Check that each technical term is defined before repeated use.
Search Sections 1 and 2 for unintended I, we, you, and your phrasing.
Search for the prohibited contrast and transition formulas listed above.
Read several consecutive paragraphs aloud and revise clipped or abrupt sequences.
Verify every number, date, version, and benchmark description against its source.
Open every link and replace obsolete references when a stable source exists.
Label each time-sensitive appendix with a verification date or comparison cut-off.
Check nested bullets for correct indentation and parallel grammar.
Check tables and code blocks at mobile width in the intended publication system.
Read the document without its appendices; the main argument and personal test should remain complete.
Ask a reader unfamiliar with benchmark terminology to flag unclear definitions and unexplained transitions.

E.8 Change record

Each published revision should record:

revision date;
editor;
sections changed;
sources added, removed, or reverified;
benchmark and model versions updated;
unresolved questions.