How to read an AI benchmark without being fooled

Every model launch comes with a bar chart. The vendor's new model is on top. Here is how to decide whether that means anything for you.

Who ran it

Vendor-run numbers are measured under conditions the vendor chose: prompt format, number of attempts, tool access, reasoning budget. Independent evaluations (academic groups, third-party leaderboards, your own tests) are the only numbers that compare models under the same conditions. If a chart has no independent source, read it as a claim, not a result.

What was measured

• Static knowledge tests (multiple choice over science, law, medicine) saturate: top models score so close to the ceiling that differences are noise, and questions leak into training data over time.
• Coding and agentic tasks (fix this real bug, complete this multi-step job) are harder to game and closer to work people pay for. Note whether the score is pass@1 (one attempt) or best-of-many.
• Preference leaderboards rank models by which answer humans prefer in blind pairs. They measure likeability and formatting as much as correctness.

The details that change the answer

• Reasoning budget. Many models score higher when allowed more "thinking" tokens. A score at the maximum setting costs several times more per task than the default setting.
• Tools. A model allowed to run code or search scores differently from one that cannot. Compare like with like.
• Contamination. If the test questions were public before the model was trained, the model may have seen them. New or private test sets are more trustworthy.
• Error bars. A two-point lead on a 500-question test is usually within noise. Few vendor charts show confidence intervals.

What to do instead

Build a small evaluation set from your own work: 50 to 200 real tasks with known-good answers. Run every candidate model on it at the settings you would use in production, and record cost and latency alongside accuracy. That table will disagree with the launch chart more often than you expect, and it is the only one that predicts your results.