NVIDIA's TAO 7 Lets Claude Code Train Vision AI in Plain English
NVIDIA TAO 7 turns your coding agent into a vision model trainer with LLM-guided AutoML that finds good configs 2x faster
Takeaways
- NVIDIA TAO 7 is now generally available, turning coding agents (Claude Code, Codex) into natural-language interfaces for vision model training.
- LLM-guided AutoML finds strong hyperparameter configs up to 2x faster than traditional search, with 6 new algorithms and W&B integration.
- Any HuggingFace CV or VLM model can now be fine-tuned (full or LoRA) on local NVIDIA GPUs with no Python required for basic workflows.
- Data Enhanced Fine-Tuning (DEFT, developer preview) autonomously diagnoses model failures and selects data strategies to fix them.
- The TAO Skills Bank bundles 56 skills across models, data, platforms, and end-to-end workflows under an Apache-2.0 license.
- Supports local Docker, NVIDIA Brev, SLURM, and Kubernetes backends; free to use, pay only for compute.
NVIDIA TAO 7 is now generally available, and it represents a fundamental rethink of how you interact with a model training toolkit. Instead of editing YAML configs or memorizing CLI flags, you describe what you want in plain English to your coding agent, and TAO handles the rest. The toolkit ships as a plugin for Claude Code and Codex, exposing every training workflow as a structured skill contract the agent reads to plan and dispatch jobs.
From CLI to conversation
TAO 7 adds a suite of agent skills and tools for fine-tuning vision AI models with coding agents and natural language prompts. The core idea is that the entire toolkit, including model training, data prep, hyperparameter search, and deployment, is now surfaced as a skill bank that any compatible coding agent can consume. You install the plugin, and from that point on your agent knows how to run every TAO workflow.
The skill bank works with Codex, Claude Code, and NemoClaw. It is a plugin published to coding-agent marketplaces that exposes every model, data transform, platform backend, and application workflow as a SKILL.md contract the agent reads to plan jobs. Crucially, most skills require zero Python: the agent constructs docker run commands directly, so the only hard dependencies are Docker and an NVIDIA GPU.
Four things that are actually new
- Agent skills: Fine-tune open models, including Cosmos 3, using natural language with any coding agent. The skill bank bundles 56 skills across model training, data preparation, platform backends, and end-to-end application workflows.
- LLM-guided AutoML: Automated hyperparameter optimization using the AutoML agent achieves the highest accuracy and converges quickly with LLM-generated reasoning. This converges 2x faster toward accuracy targets. Six new search algorithms are included: BOHB, BFBO, ASHA, PBT, DEHB, and Hyperband-ES, with Weights & Biases integration.
- HuggingFace fine-tuning: Fine-tuning is now available for any HuggingFace CV or VLM model, supporting both full fine-tuning and LoRA on local NVIDIA GPUs inside an NGC PyTorch container. LoRA (Low-Rank Adaptation) is a technique that fine-tunes only a small fraction of a model's parameters, making it much cheaper than full retraining.
- Data Enhanced Fine-Tuning (DEFT): This is currently in developer preview. It is an autonomous workflow that diagnoses where a model is failing, then selects the right data strategy to fix it, closing the loop between evaluation and training without manual intervention.
How it works under the hood
TAO is built around three components: a skill bank that publishes every TAO capability as a SKILL.md contract; a coding-agent plugin that makes those skills available to Claude Code, Codex, and other agents; and the TAO Execution SDK, which dispatches jobs to your chosen compute backend. When you prompt your agent, it reads the relevant skill contract, figures out what inputs are missing, asks you for them, and then fires off the job.
The AutoML upgrade is where the LLM angle becomes concrete. Rather than running a blind grid search over hyperparameters (trying every combination exhaustively), LLMs offer much greater flexibility in the search space over traditional methods by using prior knowledge about what configurations tend to work. TAO's AutoML agent reasons about training logs and proposes the next configuration to try, which is why it converges faster than classical approaches.
What you can run it on
TAO supports a wide range of deployment backends. Jobs can run locally or on managed cloud infrastructure:
- Local Docker on your own workstation (single GPU, no SDK needed)
- NVIDIA Brev and DGX Cloud Lepton for managed GPU cloud
- SLURM clusters for on-prem multi-node training
- Kubernetes via kubeconfig
Agent skills streamline the data pipeline process with tools for data ingestion, auto-labeling, and conversion to formats optimized for NVIDIA TAO. TAO also standardizes deployment with NVIDIA DeepStream skills for all supported models on edge or cloud.
What it covers and what it doesn't
TAO 7 is squarely focused on computer vision and vision-language models. It supports over 100 pre-trained models and covers tasks including image classification, object detection, instance and semantic segmentation, depth estimation, 3D perception, pose estimation, and change detection. New in 7.0.1 are NVPanoptix3D for 3D panoptic reconstruction, CLIP fine-tuning with SigLIP2 and RADIO-CLIP support, and Cosmos-Embed1 for visual retrieval.
If you need to train LLMs, this is not the right tool. For training and fine-tuning Large Language Models, NVIDIA points to NeMo instead. TAO is also currently limited to NVIDIA GPUs; there is no CPU-only or AMD path. And while DEFT (the autonomous data diagnosis workflow) sounds compelling, it is still in developer preview, so production use should be treated with caution.
Getting started in two commands
Install the skill bank into Claude Code with:
/plugin marketplace add https://github.com/NVIDIA-TAO/tao-skills-bank.git#7.0.1
/plugin install tao-skills@tao-skill-bank
For Codex, a one-liner handles everything:
curl -fsSL https://raw.githubusercontent.com/NVIDIA-TAO/tao-skills-bank/main/scripts/install-codex-agents.sh | bash
You will need an NGC_KEY for container pulls and an HF_TOKEN for gated HuggingFace models. Export those in your shell before launching the agent session and they are picked up automatically. The official docs cover the full walkthrough including environment variables and example prompts. The toolkit is free to use; you pay only for the compute you run jobs on.
Why this matters beyond the headline
The real shift here is that NVIDIA is betting on the coding agent as the new interface for MLOps tooling. Previous TAO versions required you to hand-edit spec files and invoke a CLI. TAO 7 treats the agent as the primary user, and the human as the one setting goals. That is a meaningful architectural change, and it aligns with where the broader ecosystem is heading: tools like Claude Code and Codex are increasingly being used not just for software development but for orchestrating compute-heavy workflows.
The LLM-guided AutoML is also worth watching. Hyperparameter search has historically been one of the most expensive and opaque parts of production CV development. Using an LLM to reason over training logs and propose configurations, rather than sampling blindly, is a practical application of LLM reasoning that goes beyond code generation. If the 2x convergence claim holds across diverse datasets, it could meaningfully reduce the GPU hours required to ship a production vision model.