Explore AI
Why the published numbers don't describe your machine
Benchmark scores are real measurements of things that may have nothing to do with what you are doing.
Every model release arrives with a table of scores. Most of them are honest. Almost all of them are misleading, for reasons that have nothing to do with dishonesty.
Four reasons a score does not transfer
It was measured at full precision. Your copy is quantised. See what Q4 costs you. Nobody publishes the quantised score.
It was measured on hardware you do not have. Throughput figures assume a datacentre GPU. Speed on your machine is a different number entirely, governed by memory bandwidth rather than compute.
Benchmarks are contaminated. Test sets have been public for years. A model can score well on questions it has effectively seen before. This is not always deliberate, and it does not stop being a problem.
Training on the test. The strongest evidence is not a rumour: model cards sometimes disclose it, and scores that jump suspiciously on one benchmark while others stay flat deserve scepticism.
Which benchmarks are worth anything
| Benchmark | What it measures | Honest use |
|---|---|---|
| Human preference arenas | Which answer people preferred, blind | The best available proxy for "feels good to use" |
| Coding task suites | Whether real code changes pass real tests | Genuinely useful, and hard to fake |
| Knowledge exams | Multiple-choice recall | Saturated. Nearly every current model scores near the ceiling. Ignore it. |
| Vendor-reported scores | The vendor's chosen comparison | Directionally useful, never decisive |
If you look at one thing, look at blind human preference and at coding suites. Both are expensive to game.
The only benchmark that matters
Your task, on your hardware, at your quantisation.
Fifteen minutes of testing beats any leaderboard:
1. Take twenty real examples of the work you actually want done. 2. Run them through two or three candidate models at Q4. 3. Score them yourself, blind if you can manage it. 4. Note the speed, not just the quality — a model that is 30% better and 5× slower may be the worse choice.
Twenty examples is enough to separate a model that works for you from one that does not. It is also the only measurement in this whole field that takes your hardware into account.