---
title: "Nanjing University's AutoTool Beats Larger Vision Models by Knowing When to Skip Tools"
subtitle: "AutoTool teaches multimodal models to decide when NOT to use tools, cutting inference time by up to 44.9% while boosting accuracy"
url: "https://alphasignal.ai/news/nanjing-university-s-autotool-beats-larger-vision-models-by-knowing-when-to"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-03T00:00:00.000Z"
updated: "2026-06-12T00:57:21.030Z"
type: "Paper"
topics:
  - "Reasoning"
  - "Post Training"
subtopics:
  - "Test Time Compute"
  - "Rl"
upvotes: 2068
source_url: "https://arxiv.org/abs/2605.19852"
comments: 0
truncated: true
---

# Nanjing University's AutoTool Beats Larger Vision Models by Knowing When to Skip Tools

> AutoTool teaches multimodal models to decide when NOT to use tools, cutting inference time by up to 44.9% while boosting accuracy

There's a quiet assumption baked into most tool-augmented multimodal AI systems: more tools, more often, equals better answers. A new paper from researchers at Nanjing University challenges that assumption head-on, showing that indiscriminate tool use can actually hurt both performance and efficiency. Their solution, [AutoTool](https://github.com/MQinghe/AutoTool), teaches a vision-language model to decide on its own whether a given question even needs a tool in the first place.

The work arrives at a moment when the field is racing to give multimodal large language models (MLLMs) the ability to "think with images" -- a paradigm popularized by OpenAI's o3 and pursued by open-source projects like DeepEyes and OpenThinkIMG. These systems let models zoom into image regions, run object detectors, or call OCR engines mid-reasoning. The problem is that none of them ask whether the tool call was actually necessary.

## When the zoom-in makes things worse

The core insight is simple but underappreciated. When a model is asked about the spatial relationship between a person and a car in a scene, it needs a global view of the image. Zooming into just the car region -- which is exactly what a tool-always-on model tends to do -- strips away the context needed to answer correctly, introducing hallucinated responses. Conversely, when identifying a specific object among many similar candidates in a dense, high-resolution image, the zoom-in tool is genuinely essential.

![Examples showing when tool use is beneficial vs. unnecessary in visual question answering](https://arxiv.org/html/2605.19852v2/x2.png) 

Prior work like DeepEyes uses reinforcement learning to teach models **how** to invoke tools correctly, but the reward structure implicitly encourages tool use on every query regardless of difficulty. The result: DeepEyes calls the zoom-in tool on nearly every sample, even trivial ones. This inflates training time to 44.9 hours and bloats inference costs across the board.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning](https://arxiv.org/abs/2605.19852)
- [AutoTool GitHub Repository](https://github.com/MQinghe/AutoTool)
- [AutoTool Full Paper (HTML)](https://arxiv.org/html/2605.19852)
- [DeepEyes: Incentivizing Thinking with Images via Reinforcement Learning](https://arxiv.org/abs/2505.14362)
- [Awesome Multimodal Reasoning - GitHub](https://github.com/The-Martyr/Awesome-Multimodal-Reasoning)