Skip to main content
Back

ToolSpec Makes LLMs Generate Tool Calls 4.2x Faster Without Any Training

A training-free speculative decoding method exploits JSON schemas and past tool calls to cut tool-calling generation latency by up to 4.2x.

  • ToolSpec accelerates LLM tool calling by up to 4.2x using training-free speculative decoding
  • Combines schema-aware drafting via a finite-state machine with retrieval of similar historical tool calls
  • Tool-call generation accounts for up to 96% of end-to-end latency on Qwen2.5-72B, not tool execution
  • Outperforms Eagle-2, Eagle-3, PLD, Token Recycling, and SAM-Decoding across four benchmarks
  • Plug-and-play with only ~6% drafting overhead; code available on GitHub
  • Speedup correlates strongly (r=0.90) with model's JSON schema adherence during generation

Tool calling has become the connective tissue between LLMs and the outside world, but it carries a hidden cost that rarely shows up in benchmarks: generation latency. A new paper introduces ToolSpec, a training-free speculative decoding method that leverages the strict structure of JSON tool calls and reuses past invocations as drafts, achieving up to 4.2x speedups over vanilla autoregressive decoding.

The core idea is refreshingly practical. Tool calls look nothing like free-form text, so treating them like free-form text during decoding wastes cycles. ToolSpec exploits that mismatch, and the authors have released the code as a plug-and-play module.

Where the time actually goes

The paper opens with a measurement that reframes the problem. Most prior work on speeding up tool use focuses on tool execution, parallelizing API calls or overlapping them with generation. On ToolBench, however, generating tool calls with Qwen2.5-14B-Instruct accounts for roughly 80% of end-to-end inference latency and is about 4x larger than the time spent executing the tools. Scale the model up and it gets worse: tool-calling generation latency reaches up to 96% of end-to-end latency for Qwen2.5-72B-Instruct.

The bottleneck is not the network round-trip to the calculator or search engine. It is the LLM slowly emitting JSON tokens, one at a time.

Two patterns worth exploiting

The authors identify two structural properties of tool-call traces that make them ideal for speculative decoding:

  • Schema adherence. Tool calls are JSON with a fixed shape. Both the LLaMA-Instruct and Qwen2.5-Instruct model families achieve over 99% accuracy in following the required tool-call schema. Most tokens (braces, quotes, field names) are effectively predetermined once you know which tool the model picked.
  • Repetitive invocations. Users tend to hit the same tools over and over. On API-Bank, the average number of calls per tool exceeds 10.95.