NVIDIA's SkillEvaluator Proves Coding Agent Skills Boost Correctness by 41 Points

NVIDIA just dropped hard numbers on a question every team building coding agents has been arguing about: do packaged skills actually make agents better, or are they just another prompt-engineering fad? The company benchmarked more than 300 of its verified skills across 30+ NVIDIA products and published SkillEvaluator, the open-source tool it used to run the tests.

The setup is deliberately boring in a good way. For each evaluation case, an agent harness runs twice: once with the verified skill installed and once without it. Each run executes in its own isolated sandbox with the same prompt, model, task inputs, and grading criteria. The only variable is the skill itself, and the delta between the two runs is what NVIDIA calls Skill Lift.

What a skill actually is

NVIDIA verified Skills are packaged, signed capability descriptors that tell an agent exactly what an NVIDIA product does, when to invoke it, and how to call it. Think of them as pre-written playbooks that ship with instructions, examples, and tool-calling guidance so the agent does not have to rediscover how to use, say, CUDA-X or Jetson memory optimization by grepping through docs.

The skills are distributed through plugins for Claude Code, Codex, and Cursor, and the same skills are available through Skills.sh, ClawHub, and Hermes Hub. SkillEvaluator itself is the layer that certifies which ones are worth shipping.

The three-tier gate

Before a skill gets published, SkillEvaluator runs it through three checks that each answer a different question:

• Tier 1 (Safety and structure): Static checks for schema and frontmatter validation, quality scoring, security scanning for prompt injection and data exfiltration, secret and PII detection, license checks, and script linting.
• Tier 2 (Distinctiveness): Uses embedding similarity to identify duplicated guidance inside a single skill and overlapping coverage across the catalog.
• Tier 3 (Live evaluation): Runs a live evaluation with an agent against generated tasks, once with the skill installed and once without, inside an isolated sandbox, and measures the difference.

Tier 3 is where the real work happens. It uses Harbor, an open-source framework for running agent evaluations in repeatable, isolated environments. SkillEvaluator handles the Harbor setup, turning evaluation cases into tasks, running the agents in sandboxes, collecting the results, and calculating the skill's impact.

The numbers

Baselines without skills were mediocre. Average baseline scores ranged from 39 to 46 out of 100 across Correctness, Discoverability, Effectiveness, and Efficiency, indicating substantial room for improvement. Security was the exception, with an average baseline score of 97. With skills installed, the picture changes sharply:

Dimension	Without skill	With skill	Lift
Correctness	46	87	+41
Discoverability	42	82	+40
Effectiveness	39	78	+39
Efficiency	43	78	+35
Security	97	98	+1

Both harnesses show consistent, meaningful gains. The difference between them is expected, given different default system prompts, context handling, and tool-calling implementations. Claude Code averaged +34 lift across all dimensions versus Codex at +29.

The catch nobody wants to talk about

Skills are not a free lunch on tokens. NVIDIA published two contrasting single-attempt examples that expose the variance: the jetson-optimize-memory skill reduced tokens from 617,306 to 142,540 (76.9%) and execution time from 474.9 to 220.0 seconds (53.7%). Conversely, cuopt-install increased tokens from 25,227 to 55,582 (120.3%) and execution time from 34.0 to 41.1 seconds (20.8%). A skill can double your token bill even while lifting correctness, so measuring per-skill economics matters.

The other honest caveat: Skill Lift varies far more across products than across harnesses. Claude Code and Codex differ by about 5 points on average. But per-product Skill Lift ranges from roughly +2 to +46. The domain and the quality of the eval set matter more than which agent framework you pick.

Trying it yourself

The workflow is two commands. Generate an evaluation dataset for a skill:

skillevaluator create-eval-dataset ./my-skill --full

Then run the with/without comparison inside Docker:

skillevaluator tier3 evaluate ./my-skill \
  --agents codex \
  --env-mode docker

SkillEvaluator converts the cases into a Harbor task bundle, runs each case with and without the skill, grades both runs, and produces the scores and Skill Lift. Full workflow details are in the SkillEvaluator quickstart docs.

Why this matters beyond NVIDIA's catalog

Anyone shipping a coding agent has been guessing whether a system prompt tweak, a tool description rewrite, or a retrieval index actually helped. SkillEvaluator formalizes that measurement into a reproducible A/B test with sandbox isolation, which is the piece most in-house evals skip. Partners are already integrating it: OpenClaw is piloting SkillEvaluator for official organizations on ClawHub. The integration runs Tier 3 evaluations and displays with-skill and without-skill results in an Evals tab, and Nous Research wired an advisory scanner called SkillSpector into the Hermes Agent install flow.

If your team is building or curating skills for agents, the tool is worth pointing at your own catalog before shipping. The 300-skill benchmark suggests most well-scoped skills clear a real bar, but it also suggests roughly one in ten will burn tokens without meaningfully changing outcomes, and you will not know which is which without measuring.