Google's Gemini Enterprise Agent Platform Now Catches Silent AI Failures in Production

Shipping an AI agent is the easy part. Knowing whether it's working, and catching when it quietly breaks in production, is the hard part. Google just made that significantly easier: Agent and Model Evaluations in the Gemini Enterprise Agent Platform are now generally available.

One engine, dev to prod

Agent quality needs to be measured during development against the cases you wrote, and after launch against the tasks the agent actually performed. The core promise here is a single evaluation engine that runs in both contexts with identical metric definitions. When you use consistent quality scoring on local experiments and live traffic, a score drop in production points to a problem with the agent rather than with the measurement. Without that guarantee, you can't tell whether a regression is real or just a measurement artifact.

What's in the box

The platform ships with more than 20 pre-built metrics spanning quality, safety, grounding, agent tool use and trajectory, and reference-based scoring for tasks like summarization and translation. The metrics fall into two families:

• Computation-based metrics — deterministic, ground-truth comparisons: ROUGE for summarization, BLEU, MetricX, and COMET for translation, exact match for extractive QA.
• LLM-as-a-judge metrics — model-graded scores for things that resist formulas: task success, tool use quality, safety, hallucination, grounding, and trajectory quality.

The standout addition is adaptive rubrics, a workflow co-developed with Google DeepMind. Instead of applying one generic judge prompt to every input, an adaptive rubric creates case-specific pass/fail tests from the eval case definition, the developer instruction, and the tool declarations, then grades traces against those rubrics and returns a verdict with rationale per rubric. A "good summary" of a legal document and a "good summary" of a recipe deserve completely different criteria, and this is the mechanism that enforces that distinction.

You can also define your own code-based or LLM-as-a-judge metrics and store them in one versioned, org-wide registry so scoring stays consistent and comparable over time. A code-based metric is a Python function, useful for exact text matches, JSON-shape checks, or any logic you can express in code.

Simulation without the scripting pain

Writing eval cases by hand is slow and tends to miss edge cases. The evaluation service can generate them through three tools:

• Case generator: seeds synthetic eval cases from the agent's instructions and tools, so you're not starting from a blank page.
• User simulator: you define a persona and a short conversation plan, and the simulator plays that user across a full multi-turn exchange, letting you evaluate real back-and-forth without scripting each reply.
• Environment simulator: stands in for the systems your agent calls. Point it at a tool, give it the response you want (mocked data, a forced error, added latency) and it intercepts that call during the run, so you can test how the agent handles a failing or slow backend without touching production.

Production monitoring without a data pipeline

An agent that passes your entire test suite can still drift a week after launch, when faced with inputs nobody wrote a case for. Online monitors address this directly. If you already collect traces and sessions through Cloud Trace, you can evaluate them with one click in the traces UI. Set up a continuous monitor and you get scores over time in built-in dashboards, plus drift alerts for email, Slack, or other channels. Built-in sampling and targeted filters keep costs manageable by letting you skip grading every single request.

Where you can plug it in

Evaluation is available through the Agent Platform SDK and the REST API. Entry points include:

• Agent Platform SDK — programmatic access with the full API surface
• agents-cli — makes eval a first-class command alongside the ones you already use to deploy agents and inspect telemetry (ADK-Python agents today)
• ADK — evaluation is built into the framework, so you can define eval sets and run them locally as you develop, including inside pytest for CI
• Evals Worksheet — a web UI for reviewing runs without writing code
• Coding agents — a reusable skill that walks Claude Code and similar tools through the full agent-quality flywheel

Experiments run client- or server-side. Server-side keeps every artifact in Cloud Storage, making runs auditable and reproducible. For large jobs, issue clustering groups eval failures into interpretable clusters against your own taxonomy of failure reasons.

What it costs

Pricing is consumption-based. For model-based metrics, charges apply only for the prediction costs of the underlying judge model, billed on input tokens from your evaluation dataset and the judge's output tokens. Code-based and computation metrics carry no additional cost. Server-side runs incur standard Cloud Storage fees for artifacts. New customers get up to $300 in free credits to try Agent Platform and other Google Cloud products.

The GA release also adds enterprise security features and regional availability — see the regions and security table for specifics. The consistency guarantee across offline and online contexts is what separates this from a standalone eval toolkit: it gives you the foundation to treat agent quality like any other software metric, something you measure continuously, alert on, and improve over time.