Google's Gemini Enterprise Agent Platform Now Catches Silent AI Failures in Production
Google's agent evaluation suite hits GA with 20+ metrics, adaptive rubrics, and live production monitoring -- all on one consistent engine
- GA launch: Agent and Model Evaluations in Gemini Enterprise Agent Platform are now generally available for dev and production use.
- 20+ metrics: Pre-built metrics cover quality, safety, grounding, tool use, trajectory, summarization, and translation -- plus custom code or LLM-judge metrics.
- Adaptive rubrics: Co-developed with Google DeepMind, generates case-specific pass/fail criteria per eval input instead of a one-size-fits-all judge prompt.
- Online monitoring: Continuous evaluation on live production traffic with score-over-time dashboards and drift alerts via email or Slack.
- Simulation tools: Case generator, user simulator (multi-turn), and environment simulator (mock backends) reduce manual test-writing.
- Pricing: LLM-judge metrics billed on token usage; code-based and computation metrics are free; new accounts get $300 in free credits. See pricing.
Shipping an AI agent is the easy part. Knowing whether it's actually working -- and catching when it quietly breaks in production -- is the hard part. Google just made that significantly easier: Agent and Model Evaluations in the Gemini Enterprise Agent Platform are now generally available.
One engine, dev to prod
Agent quality must be measured during development against the cases you wrote, and after launch against the tasks the agent actually performed. The core promise here is a single evaluation engine that runs in both contexts with identical metric definitions. When you use consistent quality scoring on local experiments and live traffic, a drift in production points to a problem with the agent rather than with the way it was measured. That's a subtle but important guarantee -- without it, you can't tell if a score drop is a regression or just a measurement artifact.
What's actually in the box
Start from more than 20 pre-built metrics spanning quality, safety, grounding, agent tool use and trajectory, and reference-based scoring for tasks like summarization and translation. The metrics break down into two families:
- Computation-based metrics -- deterministic, ground-truth comparisons: ROUGE for summarization, BLEU, MetricX, and COMET for translation, exact match for extractive QA.
- LLM-as-a-judge metrics -- model-graded scores for things that can't be reduced to a formula: task success, tool use quality, safety, hallucination, grounding, and trajectory quality.