Skip to main content
Back

Nanjing University's AutoTool Beats Larger Vision Models by Knowing When to Skip Tools

AutoTool teaches multimodal models to decide when NOT to use tools, cutting inference time by up to 44.9% while boosting accuracy

  • AutoTool teaches multimodal LLMs to decide whether to invoke tools per query, rather than always calling them.
  • Uses two special tokens (<tool_on> / <tool_off>) learned via RL (GRPO) with mode-specific reward functions.
  • Cuts inference time by up to 44.9% on POPE and training time by 20.3% vs. DeepEyes baseline.
  • Achieves 21.8% accuracy gain on V* benchmark over the Qwen2.5-VL-7B base model.
  • AMB (Adaptive Mode Balancing) prevents training collapse into text-only reasoning by dynamically adjusting reward coefficients.
  • Code and model available at github.com/MQinghe/AutoTool; accepted at ICML 2026.

There's a quiet assumption baked into most tool-augmented multimodal AI systems: more tools, more often, equals better answers. A new paper from researchers at Nanjing University challenges that assumption head-on, showing that indiscriminate tool use can actually hurt both performance and efficiency. Their solution, AutoTool, teaches a vision-language model to decide on its own whether a given question even needs a tool in the first place.

The work arrives at a moment when the field is racing to give multimodal large language models (MLLMs) the ability to "think with images" -- a paradigm popularized by OpenAI's o3 and pursued by open-source projects like DeepEyes and OpenThinkIMG. These systems let models zoom into image regions, run object detectors, or call OCR engines mid-reasoning. The problem is that none of them ask whether the tool call was actually necessary.

When the zoom-in makes things worse

The core insight is simple but underappreciated. When a model is asked about the spatial relationship between a person and a car in a scene, it needs a global view of the image. Zooming into just the car region -- which is exactly what a tool-always-on model tends to do -- strips away the context needed to answer correctly, introducing hallucinated responses. Conversely, when identifying a specific object among many similar candidates in a dense, high-resolution image, the zoom-in tool is genuinely essential.

Examples showing when tool use is beneficial vs. unnecessary in visual question answering

Prior work like DeepEyes uses reinforcement learning to teach models how to invoke tools correctly, but the reward structure implicitly encourages tool use on every query regardless of difficulty. The result: DeepEyes calls the zoom-in tool on nearly every sample, even trivial ones. This inflates training time to 44.9 hours and bloats inference costs across the board.