How to Evaluate a New AI Model Without Starting From Scratch

A new model dropped this morning. The chart looks strong, the price looks better, and someone on your team is already asking whether you should switch.

Public benchmarks tell you a model is generally capable. They cannot tell you whether it works inside your system.

Say your team uses Gemini 3.5 Flash to summarize technical papers and wants to test Gemini 3.6 Flash. The real question is not whether it can summarize a paper. It is whether it keeps the critical findings, retains the qualifications that change how those findings read, obeys your output limit, and meets your latency and cost needs.

You do not need a large benchmark to start. You need a small set built around the failures that would make the model unusable. A small test cannot approve a migration, but it can tell you whether the candidate deserves a bigger evaluation, which failures it must cover, or whether to stop now.

To make this concrete, I ran it. Gemini 3.5 versus 3.6 Flash, eight public AI papers, four risks. Buried results, lost qualifications, missing information, and figures that need context. Both got the same PDFs, prompt, tools, and 180-word limit. Four higher-risk papers ran three times, for 32 outputs.

The goal was not to crown a winner, but to show how a team frames the test, turns failures into cases, reads the errors, and decides what to do next.

Public benchmarks tell you where to look. A task-specific smoke test tells you whether to keep looking.

The process runs as one loop.

Why the public benchmark is not your answer

A public benchmark measures a model against someone else's task, scoring, and definition of success. Your system may care about something else. A summarization benchmark rewards broad coverage. Your product might fail when the model drops one condition buried in a limitations section.

Public scores also hide uneven behavior. A model can improve overall while regressing on the one category you depend on, or gain accuracy by writing longer and ignoring a format rule your product treats as fixed.

You do not ship benchmark scores. You ship outputs users accept, reject, edit, or act on. Only tests shaped around those outputs tell you whether the model belongs in your system.

Step 1: Reuse before you rebuild

Start with what production already gave you. A team that summarizes documents likely has the documents, the prompts, approved summaries, editor corrections, claims fact-checkers removed, and outputs rejected for length or tone.

That is the start of your evaluation set. A draft that omitted the comparison baseline becomes a qualification case. A summary that treated an estimate as measured becomes a contextual-figures case. Reuse production errors before generating artificial ones.

For this public walkthrough I could not use private documents, so I used eight public papers chosen to reproduce the same failures. The source changed, the method did not.

Before running either model, I wrote an answer card per paper. Not one ideal summary, but the facts and qualifications an acceptable summary had to preserve, the claims it must not make, and the one critical failure that would sink it.