When Public AI Benchmarks Aren't Enough (Part 3 of 3)

Public benchmarks can tell you a great deal about AI model capability. They still may not tell you which model will perform best on the work you actually need done.

This final post in a 3-part series shows how to investigate that gap with a small personal evaluation: a representative task, controlled user-visible conditions, and a rubric written before testing. It also examines what happened when I applied that method to three model systems.

Part 3 of 3 · Explore the complete series

Part 1 — Why AI Benchmark Scores Are Harder to Read Than They Look
Part 2 — A Practical Field Guide to AI Benchmark Scores
Part 3 — When Public Benchmarks Aren't Enough (you are here)

Navigate this post

See the gap public scores cannot close
Build a small personal evaluation
Explore the rubric and three system profiles
Interpret the findings and limitations
Run the method on your own work

Why public scores do not extend automatically to personal use

A public AI benchmark is designed to measure a capability consistently across many systems and users. Your use case is specific: your prompts, data, tools, quality requirements, and tolerance for different errors. Those details do not transfer automatically from a published score.

Several familiar limitations contribute to the gap:

Training data overlap: A model may have encountered benchmark questions, or close variants, during training. Familiarity can improve its score without improving performance on genuinely new problems.
Task-format specialization: Models may be optimized for the response format a benchmark rewards. Your workflow may reward a different form of reasoning or output.
Protocol optimization: Instructions, tools, retry limits, and reasoning budgets may be tuned specifically for the benchmark. You may not have access to the same setup.
Selective reporting: Providers publish the results that best support a release. The published evidence may not cover every capability relevant to you.
Coverage limits: Every benchmark contains a finite task sample. Even a careful sample covers only part of the work that matters in practice.

---
title: "Why Public Evidence Leaves a Personal Gap"
---
flowchart LR
    PB["Public<br/>Benchmark"] -->|"measures"| GT["General<br/>Task Set"]
    YW["Your<br/>Work"] -->|"depends on"| LR["Local<br/>Requirements"]
    GT -.->|"partial transfer"| LR
    LR -->|"needs"| PE["Personal<br/>Evaluation"]
    classDef blue fill:#e3f2fd,stroke:#0066cc,stroke-width:2px;
    classDef green fill:#e5ffe5,stroke:#388e3c,stroke-width:2px;
    classDef yellow fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
    classDef orange fill:#ff9f43,stroke:#e67e22,stroke-width:2px;
    classDef teal fill:#95e1d3,stroke:#16a085,stroke-width:2px;
    classDef pink fill:#f38181,stroke:#e74c3c,stroke-width:2px;
    classDef red fill:#ffe5e5,stroke:#c0392b,stroke-width:2px;
    classDef gray fill:#f5f5f5,stroke:#95a5a6,stroke-width:2px;
    class PB blue
    class GT blue
    class YW teal
    class LR teal
    class PE orange

The practical consequence is simple: two models with similar public profiles can behave very differently on the same real task. A direct way to investigate that difference is to test them on work that resembles your own.

What a personal evaluation looks like

---
title: "Personal Evaluation: Core Workflow"
---
flowchart LR
    D["Define<br/>Decision"] -->|"shapes"| T["Representative<br/>Task"]
    T -->|"tested under"| C["Fixed<br/>Conditions"]
    C -->|"produces"| O["Model<br/>Outputs"]
    O -->|"scored by"| R["Pre-written<br/>Rubric"]
    R -->|"reveals"| F["Actionable<br/>Findings"]
    classDef blue fill:#e3f2fd,stroke:#0066cc,stroke-width:2px;
    classDef green fill:#e5ffe5,stroke:#388e3c,stroke-width:2px;
    classDef yellow fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
    classDef orange fill:#ff9f43,stroke:#e67e22,stroke-width:2px;
    classDef teal fill:#95e1d3,stroke:#16a085,stroke-width:2px;
    classDef pink fill:#f38181,stroke:#e74c3c,stroke-width:2px;
    classDef red fill:#ffe5e5,stroke:#c0392b,stroke-width:2px;
    classDef gray fill:#f5f5f5,stroke:#95a5a6,stroke-width:2px;
    class D blue
    class T green
    class O green
    class F green
    class C teal
    class R orange

A useful personal evaluation needs three components:

A representative task, or small task set, drawn from work you genuinely do.
Consistent test conditions: the same prompt, tools, constraints, and access for every model.
A rubric written before testing that defines a good result in terms specific enough to score consistently.

Write the rubric before seeing the outputs

A pre-written rubric prevents a polished interface, confident tone, or preferred answer from quietly changing the criteria after the fact. It also makes disagreements easier to inspect and reproduce.

The evaluation I designed

I regularly use AI models for technical and scientific problems in professional engineering work and personal projects. One capability matters repeatedly: taking an ambiguous question, reasoning through it carefully, and producing an explanation or artifact that a non-specialist can understand.

Public benchmarks address pieces of that requirement but not the whole combination. GPQA Diamond tests scientific accuracy without testing explanatory design. Coding benchmarks test functional output without testing whether the artifact teaches. Preference benchmarks measure whether readers like a response, not whether its reasoning is rigorous. None of the public benchmarks reviewed for this series measures this exact combination in one task.

I therefore designed one prompt that asked each model to analyze an intentionally ambiguous scientific question and create an interactive visual explanation as a self-contained web page.

Artifact requirements
- A specified number of visualizations.
- Interactive controls and quantitative plots.
- An explicit chain from assumptions to conclusions.
Process requirement
- A plain-text visualization plan before the artifact.
Delivery requirement
- One self-contained HTML file that could be inspected directly.

The question was deliberately unusual:

The test scenario

Who would win—100 duck-sized horses or one horse-sized duck?

The scenario is a familiar internet thought experiment, so it should not be treated as resistant to training-data overlap. Its value comes from being deliberately underspecified: a useful response must expose assumptions, decide which physical effects matter, and communicate uncertainty. The absurd premise does not remove the need for scientific care.

One important physical concept is the square–cube law. (1) As an object grows in linear size, its volume increases faster than its surface area. For a living creature, that changes bone loading, muscle capacity, and heat management. Applying the law to an imaginary scaled animal still requires biological assumptions, so it supports a discussion of likely constraints rather than an exact prediction.

If linear dimensions scale by a factor of k, surface area scales by k² while volume and mass scale roughly by k³, assuming similar shape and density.

The scoring rubric

The rubric separates four capabilities so that strength in one area cannot conceal weakness in another.

Criterion	Weight	What it measures
Multi-Step Compliance and Attention	20	Whether the model follows detailed constraints and preserves the required output sequence
Domain Rigor and Epistemology	30	Scientific correctness, treatment of ambiguity, and honesty about assumptions
Pedagogical and Information Design	25	Whether a non-specialist can follow the reasoning and learn from the visuals
Software Engineering	25	Whether the artifact runs, is well structured, and manages dependencies cleanly

---
title: "Rubric Dimensions Keep Tradeoffs Visible"
---
flowchart TB
    C["Compliance<br/>20"] -->|"weights"| R["Rubric"]
    D["Domain Rigor<br/>30"] -->|"weights"| R
    P["Pedagogy<br/>25"] -->|"weights"| R
    E["Engineering<br/>25"] -->|"weights"| R
    R -->|"guides"| J["Judgment"]
    classDef blue fill:#e3f2fd,stroke:#0066cc,stroke-width:2px;
    classDef green fill:#e5ffe5,stroke:#388e3c,stroke-width:2px;
    classDef yellow fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
    classDef orange fill:#ff9f43,stroke:#e67e22,stroke-width:2px;
    classDef teal fill:#95e1d3,stroke:#16a085,stroke-width:2px;
    classDef pink fill:#f38181,stroke:#e74c3c,stroke-width:2px;
    classDef red fill:#ffe5e5,stroke:#c0392b,stroke-width:2px;
    classDef gray fill:#f5f5f5,stroke:#95a5a6,stroke-width:2px;
    class C yellow
    class D green
    class P blue
    class E teal
    class R orange
    class J gray

Explore the four tabs to see how each dimension turns a broad quality judgment into observable checks.

Compliance · 20Rigor · 30Pedagogy · 25Engineering · 25

Core question
- Did the system follow detailed constraints and preserve the required output sequence?
Checks
- Plan appeared before the artifact.
- Required sections and formats were preserved.
- Earlier instructions survived the long generation task.

Core question
- Was the scientific reasoning correct, explicit, and honest about uncertainty?
Checks
- Relevant physical principles were applied.
- Assumptions and ambiguity were visible.
- Formulas were defensible rather than invented for convenience.

Core question
- Could a non-specialist follow and learn from the artifact?
Checks
- Visual flow and captions carried the reasoning.
- Interactive elements built understanding.
- The information hierarchy did not depend on specialist knowledge.

Core question
- Did the artifact run reliably and remain maintainable?
Checks
- No build process was required.
- Code and dependencies were well managed.
- Layout and interactions worked as intended.

Results from three systems

I ran one evaluation on systems using Gemini 3.1 Pro, Claude Sonnet 5, and GPT-5.6 Terra with the same user-visible prompt, reference pack, approval message, and declared constraints. (1)

Hidden system instructions, product implementations, and model-specific defaults could still differ. “Same user-visible conditions” does not mean the underlying systems were identical.

Model	Total /100	Compliance /20	Rigor /30	Pedagogy /25	Engineering /25
Gemini 3.1 Pro	74	6	24	22	22
Claude Sonnet 5	62	6	15	24	17
GPT-5.6 Terra	68	6	12	25	25

Open the three generated artifacts

---
title: "One Test, Three Different Strength Profiles"
---
flowchart TB
    S["Same User-Visible<br/>Test"] --> G["Gemini<br/>Rigor 24-30"]
    S --> C["Claude<br/>Pedagogy 24-25"]
    S --> T["Terra<br/>Engineering 25-25"]
    G --> F["Compliance<br/>6 of 20"]
    C --> F
    T --> F
    classDef blue fill:#e3f2fd,stroke:#0066cc,stroke-width:2px;
    classDef green fill:#e5ffe5,stroke:#388e3c,stroke-width:2px;
    classDef yellow fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
    classDef orange fill:#ff9f43,stroke:#e67e22,stroke-width:2px;
    classDef teal fill:#95e1d3,stroke:#16a085,stroke-width:2px;
    classDef pink fill:#f38181,stroke:#e74c3c,stroke-width:2px;
    classDef red fill:#ffe5e5,stroke:#c0392b,stroke-width:2px;
    classDef gray fill:#f5f5f5,stroke:#95a5a6,stroke-width:2px;
    class S blue
    class G green
    class C orange
    class T orange
    class F red

The table supports direct comparison. The tabs below provide the evidence profile for each system.

Gemini · 74/100Claude · 62/100Terra · 68/100

Strongest dimension — domain rigor
- Gave the most complete treatment of square–cube scaling.
- Used an assumption matrix and comparative scorecard effectively.
Important limitation
- Stated some biological consequences more confidently than the hypothetical supports.
Compliance failure
- Omitted the plan-first sequence and moved the main uncertainty to an appendix.

Strongest dimension — pedagogy
- Interactive charts connected parameter changes to intuitive outcomes.
- Information architecture was easy for a non-specialist to follow.
Important limitation
- Calculated momentum and kinetic energy but omitted square–cube structural constraints.
Compliance failure
- Placed the plan inside the HTML instead of before it.

Strongest dimensions — engineering and pedagogy
- Produced a responsive, zero-dependency page with SVG, vanilla JavaScript, and dark mode.
- Met every engineering criterion in this rubric.
Important limitation
- Used an invented “toy model” instead of a defensible scientific relationship.
Compliance failure
- Embedded the visualization plan inside the educational artifact.

What the results reveal

Finding 1 — sequencing failed across all three systems
- Every system placed the planning output in the wrong location.
- The shared failure matters for workflows that require planning and execution to remain distinct.
- One run cannot identify whether prompt interpretation, product behavior, or model reliability caused it.
Finding 2 — surface quality and scientific rigor were independent
- The strongest engineering output received the lowest domain-rigor score.
- The strongest scientific analysis shared the same low compliance score as the other systems.
- A single composite impression would have hidden both differences.
Finding 3 — the useful choice depends on the work
- Gemini provided the strongest domain-rigor evidence in this task.
- Terra provided the strongest front-end engineering evidence.
- Claude provided a strong instructional structure.
- None distinguished itself on compliance.

Why this is not a universal ranking

The evaluation used one intentionally demanding task, one recorded artifact per system, and a rubric tailored to my needs. It reveals differences worth investigating for that use case; it does not establish a general ordering of the three systems.

Limitations of this comparison

Keep the evidence boundary visible

This is a worked example of a personal evaluation, not a controlled research study.

Sampling limits
- One task and one recorded artifact per system.
- No repeated trials or statistical uncertainty.
Review limits
- No documented blinded grading or inter-rater agreement.
System limits
- Hidden instructions, tools, inference settings, and coordinating software may differ.
Decision limits
- Latency and cost were not assessed.

What a fully auditable record should publish

Exact product and model versions.
Run dates, settings, tools, and retry policy.
Complete conversations and generated artifacts.
Score sheets, reviewer identities or roles, and disagreement handling.
Any exclusions, reruns, or post-processing.

The full guide publishes the prompt and scoring framework. Until the remaining audit record is available, the named results should be read as an illustrative case study.

A public benchmark comparison would not have produced these conclusions. Public scores may be accurate for their own tasks and conditions while remaining incomplete for a specific decision.

Run your own evaluation

You do not need a research laboratory or a large task suite. Three to five representative tasks with explicit criteria can expose differences that headline benchmark scores miss.

Use this checklist to start:

Open the supporting method and rubric

Public evaluation will continue to evolve. New tests will appear, providers will optimize against existing ones, and the boundary of reliable capability will move. A small, repeatable personal evaluation keeps model selection grounded in the work that matters to you.

References and further reading

Open the complete reference list

Evaluation methodology

Biological and physical sources

Model announcements

Project guides

Reference guide for this series—the evaluation rubric, personal-evaluation method, and benchmark context.
Full guide: Reading AI Model Benchmarks and Building a Personal Evaluation—the complete prompt, worked evaluation, and primary sources.

Revisit Part 1 or Part 2.