How to Actually Compare AI Models (Beyond Benchmarks)
Benchmarks are noisy proxies for the model that helps you decide. Here's a practical framework to compare AI models on the work you actually do.
Public benchmarks tell you which model scored highest on a fixed test set six months ago. They don't tell you which one will help you decide the thing you're deciding this week. Here's the framework we use to compare models on real work.
Step 1: Write your own 5-question eval
Not 500. Five. Pick five questions from your actual last month of work — a decision, a comparison, a piece of writing you edited, a hard email, one thing you got wrong. Those five questions predict how a model will do for you better than any leaderboard.
Step 2: Score on dimensions, not overall
- Faithfulness. Does it stick to the facts you gave it?
- Calibration. Does it hedge when it should?
- Surfacing the dissent. Does it tell you the counter-argument, or hide it?
- Voice. Would you send this?
- Latency and cost. At your volume, does it matter?
Step 3: Run the same prompt through more than one model
The fastest way to see the difference between two models is to run the same input through both and read the outputs side by side. Not the benchmark scores — the actual outputs. Most "model X is smarter" claims disappear when you look at the two answers next to each other.
Step 4: For real decisions, make them argue
If the question matters, don't pick a winner from two static outputs. Have them debate. What model A misses, model B will attack. What both miss, a synthesis pass will surface.
Example prompt to paste in the composer: