How to Evaluate a New AI Model Without Starting From Scratch
Public benchmarks shortlist a model. A small task-specific eval tells you whether it deserves a larger test.
- Public benchmarks measure general capability but cannot reveal whether a new model handles the specific failure modes that matter in your production system, such as dropping critical qualifications or ignoring output length constraints.
- A structured smoke test of Gemini 3.5 Flash vs. 3.6 Flash across eight papers found that 3.6 Flash preserved more required facts (28/55 vs. 24/55) but violated the 180-word limit in six of eight runs compared to just one violation for 3.5 Flash, revealing a direct coverage-vs-constraint trade-off a single quality score would have hidden.
- Both models systematically converted a scoped 71.4x arithmetic-energy estimate in the BitNet paper into an unqualified efficiency claim across all repeated runs, identifying a stable failure category that more sampling cannot fix and that warrants a permanent regression test.
- The recommended evaluation structure is a sequential three-stage process — a 5–10 case smoke test to find blockers, a 50–100 case discovery set to resolve open trade-offs, and a larger statistically powered decision set — rather than a single large benchmark run.
- Migration thresholds (critical-fact recall, unsupported claims, qualification preservation, length adherence, latency, and cost per acceptable output) must be defined before seeing results, and every confirmed production failure should be added as a permanent regression case to make each future model evaluation faster and more targeted.
A new model dropped this morning. The chart looks strong, the price looks better, and someone on your team is already asking whether you should switch.
Public benchmarks tell you a model is generally capable. They cannot tell you whether it works inside your system.
Say your team uses Gemini 3.5 Flash to summarize technical papers and wants to test Gemini 3.6 Flash. The real question is not whether it can summarize a paper. It is whether it keeps the critical findings, retains the qualifications that change how those findings read, obeys your output limit, and meets your latency and cost needs.
You do not need a large benchmark to start. You need a small set built around the failures that would make the model unusable. A small test cannot approve a migration, but it can tell you whether the candidate deserves a bigger evaluation, which failures it must cover, or whether to stop now.
To make this concrete, I ran it. Gemini 3.5 versus 3.6 Flash, eight public AI papers, four risks. Buried results, lost qualifications, missing information, and figures that need context. Both got the same PDFs, prompt, tools, and 180-word limit. Four higher-risk papers ran three times, for 32 outputs.
The goal was not to crown a winner, but to show how a team frames the test, turns failures into cases, reads the errors, and decides what to do next.
Public benchmarks tell you where to look. A task-specific smoke test tells you whether to keep looking.
The process runs as one loop.
Why the public benchmark is not your answer
A public benchmark measures a model against someone else's task, scoring, and definition of success. Your system may care about something else. A summarization benchmark rewards broad coverage. Your product might fail when the model drops one condition buried in a limitations section.
Public scores also hide uneven behavior. A model can improve overall while regressing on the one category you depend on, or gain accuracy by writing longer and ignoring a format rule your product treats as fixed.
You do not ship benchmark scores. You ship outputs users accept, reject, edit, or act on. Only tests shaped around those outputs tell you whether the model belongs in your system.
Step 1: Reuse before you rebuild
Start with what production already gave you. A team that summarizes documents likely has the documents, the prompts, approved summaries, editor corrections, claims fact-checkers removed, and outputs rejected for length or tone.
That is the start of your evaluation set. A draft that omitted the comparison baseline becomes a qualification case. A summary that treated an estimate as measured becomes a contextual-figures case. Reuse production errors before generating artificial ones.
For this public walkthrough I could not use private documents, so I used eight public papers chosen to reproduce the same failures. The source changed, the method did not.
Before running either model, I wrote an answer card per paper. Not one ideal summary, but the facts and qualifications an acceptable summary had to preserve, the claims it must not make, and the one critical failure that would sink it.