Build the API chain first

Query-first systems such as ToolBench and ToolACE sample APIs, invent a hypothetical instruction, and ask an agent to find a valid sequence of calls. Their depth-first search explores one sequence until it reaches a dead end, backtracks, and tries another. Failed branches consume model calls and often leave the pipeline with shallow or unusable trajectories.

ToolGrad begins with executable API calls and gradually extends them into a workflow. Once the chain is complete, an LLM writes the corresponding user request and final answer. Generating language from a known sequence is a bounded, one-shot task, while discovering that sequence requires open-ended exploration.

Execution reports steer each step

ToolGrad adapts the textual-gradient concept from TextGrad, where an LLM critic returns plain-language feedback that guides revisions much as numerical gradients guide weight updates. In ToolGrad, execution reports provide the feedback used to construct longer API workflows.

Four modules turn that feedback into the next verified step, with each module handling one decision in the generation loop:

1. API Proposer: reduces a sampled API set to a small group of candidates that could extend the current workflow.
2. API Executors: run candidate calls in parallel and record their arguments, outputs, and failures.
3. API Selector: compares the reports and chooses the strongest successful call to append.
4. LLM Updater: revises the synthetic user request and model response to account for the expanded workflow.

Each completed workflow yields a user request, an executed API chain, and a final response. Because the framework tests every appended call, the resulting trajectory is execution-valid within its generation environment.

A 99.8% yield on ToolBench

Google evaluated ToolGrad against a standard depth-first-search pipeline using the ToolBench collection of more than 16,000 real-world APIs. The paper reports that ToolGrad generated longer, more complex workflows with a higher pass rate and lower generation cost.

The reported annotation pass rate reached 99.8%, meaning almost every generated example passed the pipeline’s checks and remained available for training. That figure measures dataset-generation yield. Downstream benchmarks separately measure whether fine-tuned models can select and call unfamiliar tools correctly.

A small dataset reaches unfamiliar tools

To measure downstream performance, the researchers used Gemini 2.5 Flash-Lite to create ToolGrad-500, then fine-tuned Gemma 3 models with 1 billion, 4 billion, and 12 billion parameters. Evaluation used the BFCL, whose tool set differs from ToolBench. The test therefore measures transfer to unseen tools rather than recall of the generation catalog.

According to the paper, ToolGrad-12B finished within 0.1 points of Gemini 2.5 Pro, exceeded the listed Claude and GPT models, and led the open tool-use models included in the comparison. Benchmark revisions, model versions, and evaluation settings remain relevant when interpreting those rankings.

Gemma 3 12B also surpassed Gemini 2.5 Flash-Lite, the model that generated ToolGrad-500, on the reported evaluation. Execution filtering supplies the student with successful call sequences, while the generator handles the associated natural-language request and response.

From an API catalog to training data

A production implementation needs more than a list of endpoint names because ToolGrad must execute and assess each proposed call. Teams adopting the pattern should prepare:

• Machine-readable API definitions with parameters, types, authentication requirements, and expected outputs.
• An isolated execution environment with test credentials, rate limits, retries, timeouts, and detailed logs.
• Selection criteria that reward valid, coherent calls and useful dependencies between workflow steps.
• Dataset checks for prompt quality, duplicate trajectories, sensitive data, unsafe actions, and argument correctness.
• Held-out evaluations using tools and requests excluded from the generation process.

Teams can focus generation on specific API families, workflow lengths, or dependency patterns. That makes the approach suitable for internal catalogs where production traces are sparse or expensive to annotate.

Execution leaves semantic gaps

Successful API execution confirms that selected arguments worked in the test environment. Separate checks must cover whether the workflow addresses a realistic user need, follows authorization rules, handles side effects safely, and produces a useful final answer.

Backward-generated requests may also differ from production language because the model writes them to fit an existing chain. Prompt diversity, human review, and evaluation against real traffic can reveal synthetic phrasing or workflows that users rarely request.

The reported results come from ToolGrad-500 and BFCL, so they establish data efficiency under the paper’s evaluation conditions. Deployment still requires domain-specific testing for tool selection, argument accuracy, multi-step recovery, latency, and security.

A cheaper route to specialized agents

For developers maintaining an internal API catalog, ToolGrad offers a practical route to targeted post-training with a relatively inexpensive generator model. A team can execute verified workflows in a sandbox, convert them into supervised examples, and fine-tune a compact model without manually writing every trajectory.

The GitHub repository contains the implementation, and the paper describes the generation pipeline and experiments. ToolGrad’s central contribution is a constrained synthesis process that verifies API calls as a workflow grows, then asks a language model to describe the completed task.