Short answer: a model that tops the leaderboard can still be worse at the job you use it for. Before you switch, take about 30 real tasks from your own work, run both models on them, and count three things: right answers, confident wrong answers, and cost per finished task. It takes an afternoon and saves you from finding the regressions in front of a customer.
This week's example
On 28 October Google announced Gemini 4 Carbon. VentureBeat reports that the independent tester Artificial Analysis put it, at high reasoning, first on its Intelligence Index v5.0 with 59 points, ahead of Claude Opus 5.5 at 58 and GPT-6 Astra at 57.
The same report shows the trade-off. On Artificial Analysis's factual-knowledge test, AA-Omniscience, Carbon answered 58% of questions correctly against 48% for the previous Gemini model, Argon. Its hallucination rate rose from 4% to 11%. So it knows more, and it also guesses more often when it doesn't know.
Neither number is a verdict. Both are reminders that "better" is a bundle. If your team uses AI to draft emails, more knowledge helps. If it reads purchase orders and fills in quantities, a model that guesses more often is a worse model for you, whatever its rank.
(Google is opening Carbon to "a subset of cyber defenders" first, so nobody reading this is switching to it this week. The pattern will repeat with the next release, though.)
Why a leaderboard can't choose for you
Artificial Analysis is open about this. Its methodology page says the index "may not apply directly to every use case" and describes it as a primarily text-based, English-language suite. Multilingual ability is measured separately.
For an Indian business that matters. Your real inputs might be a scanned GST invoice, a WhatsApp voice note in Kannada, or a customer email that switches between Hindi and English halfway through. None of that is what the headline score measures.
Price is the other trap. VentureBeat points out that identical per-token prices don't mean identical costs per finished task. To compare those, you need to know how many tokens each model uses, how often it retries and how long it runs. A cheaper-looking model that needs two attempts per invoice costs you more than it seems.
Build a test set from your own work
OpenAI's own guide to evaluations lists "vibe-based evals" among its anti-patterns: using "it seems like it's working" as the test. Production data and historical data are among the sources it suggests, and it says test data should include typical cases, edge cases and adversarial cases, labelled by human experts.
For a 20 to 200-person company, that translates into something simple:
- Pick one job the AI does today. Sorting enquiries, extracting invoice fields, drafting replies, summarising calls. One job per test set.
- Pull 30 real examples from the last few months. Take them from your inbox, your shared drive or your CRM, with personal details removed.
- Write down the right answer for each one. Ask the person who does this job by hand. This is the slow part, and the most valuable.
- Include the awkward ones. About 20 normal cases, 7 messy ones (blurry scans, mixed languages, missing fields), and 3 where the right answer is "I can't tell from this".
Keep the set in a plain spreadsheet: input, expected answer, and one column per model you test.
Score it the way your business feels it
Run each model on all 30 and mark every answer as one of three things:
- Right. It matches what your person wrote.
- Said it didn't know. Annoying, but safe. Someone picks it up.
- Confidently wrong. This is the one that hurts, because nobody notices until a customer or an auditor does.
Then work out the cost of the run divided by the number of right answers. That's your cost per finished task. Compare that number when you weigh two models; the price list won't tell you.
A model that gets 26 right and 4 "don't know" is usually a better choice for operations work than one that gets 27 right and 3 confidently wrong. For drafting marketing copy you might decide the other way. Either way, you're deciding with your own numbers. Our earlier post on where AI goes wrong in operations covers the checks to keep in place after you switch.
When to run it again
Keep the spreadsheet and rerun it whenever something changes underneath you:
- a vendor releases a new model or retires the one you use;
- your tool quietly moves you to a newer version;
- pricing changes. Argon's API prices double on 1 December 2026, to $4 per million input tokens and $20 per million output tokens, under Google's developer guide as reported by VentureBeat, so anyone using Argon through the API has a reason to check cost per task this month;
- you change the prompt or the instructions.
OpenAI's guide calls this continuous evaluation: run the tests on every change and grow the test set over time. In practice, every time the AI gets something wrong in real use, add that example to the sheet. Six months in, you'll have a test that knows your business better than any leaderboard.
Arthat AI
Working on something similar at your company?
Arthat AI builds these automations for Indian businesses every week. If you want to talk through whether this fits your situation, we'll give you a straight answer — even if the answer is "don't build it."
