NVIDIA Open-Sources SkillEvaluator to Test Whether AI Agent Skills Actually Work
NVIDIA open-sourced SkillEvaluator, a three-tier framework that stress-tests agent skills with quality gates, deduplication, and live sandboxed runs.
- NVIDIA open-sourced SkillEvaluator, an Apache-2.0 framework for evaluating agent skills end to end.
- Three independent tiers: static validation, embedding-based deduplication, and live sandboxed agent runs.
- Tier 3 measures Skill Lift, the behavioral delta between agent runs with and without the skill.
- Scores on five dimensions plus pass@k reliability, with CLI, JSON, HTML, and Markdown reports.
- Ships GitHub Actions and GitLab CI templates for merge-gating skills on pull requests.
- Works keyless for quality checks; LLM rubric and Tier 3 need a provider and Docker sandbox.
NVIDIA has open-sourced SkillEvaluator, an Apache 2.0 framework for checking whether AI agent skills are safe, distinct, and effective. It combines static validation, similarity analysis, and sandboxed agent runs so developers can evaluate a skill before adding it to a catalog or shipping it to users.
Agent skills are folders containing Markdown instructions, metadata, scripts, and reference files that extend coding agents such as Claude Code and Codex. As these packages multiply, maintainers need automated checks for insecure code, exposed credentials, duplicated guidance, and instructions that fail to improve task results. SkillEvaluator follows the community Agent Skills specification and forms part of NVIDIA’s broader Verified Skills pipeline.
Three gates answer different questions
| Tier | Question | Checks |
|---|---|---|
| 1: Validation | Is the skill safe and well formed? | Schema, quality, security, personally identifiable information, licensing, and script checks, plus optional LLM-based review. |
| 2: Deduplication | Does the skill repeat existing guidance? | Embedding similarity within a skill and across a collection or catalog. |
| 3: Live evaluation | Does the skill improve agent performance? | Sandboxed task runs with and without the skill, followed by comparative scoring. |
Each tier has its own entry point and can run without the preceding stages. Tier 1 uses
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
