Every model launch arrives with a chart. Somewhere in that chart is a benchmark the model wins, and somewhere else is a benchmark it loses quietly. Learning to read the card takes ten minutes and saves a lot of bad purchasing decisions.
Check what is averaged
A single composite score hides which tasks were included. If the composite is dominated by multiple-choice knowledge questions, it says little about writing quality, tool use, or long-document reasoning. Ask for the per-task breakdown, and look for the task closest to your own work.
Look for contamination disclosure
Public benchmarks leak into training data. A model that has seen the test set scores well without being better. Responsible cards state how they filtered, or report held-out results. Silence is not proof of cheating, but it is a reason to discount the number.
Separate capability from behaviour
- Capability: can it do the task at all?
- Behaviour: will it do the task the way you need, at the length, tone and format you asked for?
Most disappointment in production comes from behaviour, not capability. Benchmark suites rarely measure it.
Read the failure examples
Good cards include cases where the model fails. Those sections are more informative than the score table, because they show the shape of the error: overconfident, verbose, refuses too often, or silently truncates.
Check the cost of being right
A model that is slightly more accurate at ten times the price per call is often the worse choice. Divide accuracy by cost, then by latency. The winner changes depending on whether your workload is interactive or batch.
The question to ask a vendor
Which of these benchmarks resembles my task, and what does your model score on the subset I care about? A vendor who can answer that has done the work. One who cannot is quoting a chart.
Bottom line: benchmarks are a filter, not a verdict. Use them to build a shortlist, then test the shortlist on your own twenty examples.
