---
title: "NC State's Tomcat Agent Reads Vague Human Commands as Well as People"
subtitle: "A new benchmark tests whether LLM agents can decode vague human instructions in a shared task, and top models match human performance."
url: "https://alphasignal.ai/news/nc-state-s-tomcat-agent-reads-vague-human-commands-as-well-as-people"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-16T00:00:00.000Z"
updated: "2026-08-23T21:01:18.528Z"
type: "Paper"
topics:
  - "Agents"
  - "Reasoning"
subtopics:
  - "Computer Use"
  - "Tool Use"
  - "Test Time Compute"
upvotes: 1870
source_url: "https://arxiv.org/abs/2507.02935"
comments: 0
truncated: true
---

# NC State's Tomcat Agent Reads Vague Human Commands as Well as People

> A new benchmark tests whether LLM agents can decode vague human instructions in a shared task, and top models match human performance.

When a teammate says `Can you get that?` while gesturing at a table full of tools, you figure out what they mean without a formal spec. Getting an AI agent to do the same requires what researchers call Theory of Mind, the ability to model what another party is thinking. A team from North Carolina State University and TU Delft built a task and an agent to test whether today's large language models can pull it off in a live collaborative setting, and the results land surprisingly close to human performance.

The paper introduces [Instruction Inference](https://arxiv.org/abs/2507.02935), a benchmark where an agent has to figure out a human collaborator's unspoken goal from vague commands, along with [Tomcat](https://github.com/fardinsaad/Tomcat-LLM), an LLM-based agent that reasons about that goal step by step. Existing ToM benchmarks mostly rely on narrative or conversational scenarios; Instruction Inference instead evaluates whether an agent can infer a principal's intended goal from clear and unclear instructions and act appropriately in a shared task environment.

## The gap between literal and pragmatic

Most instruction-following benchmarks assume the human says exactly what they mean. Real collaboration works differently. People use indirect language, leave out details they assume are obvious, and rely on shared context. Traditional autonomous agents, lacking that context, tend to interpret commands literally and fail when those commands are incomplete or ambiguous.

![Diagram showing an LLM agent processing unclear instructions using pragmatic context and theory of mind](https://arxiv.org/html/2507.02935v3/motivation-unclear-2.png) 

The authors ground their design in Grice's cooperative principle from linguistics, which says communication follows maxims about quantity, quality, relation, and manner. When speakers break these maxims, listeners infer what they really meant. Incomplete instructions omit critical information required to determine the intended action, violating the Quantity maxim. Ambiguous instructions yield two or more possible interpretations, violating the Manner maxim. The team focuses on these two categories because they commonly arise in cooperative settings where collaborators assume shared context and task understanding.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Theory of Mind in Action: The Instruction Inference Task in Dynamic Human-Agent Collaboration](https://arxiv.org/abs/2507.02935)
- [Tomcat-LLM GitHub Repository](https://github.com/fardinsaad/Tomcat-LLM)
- [Theory of Mind in Action (arxiv HTML)](https://arxiv.org/html/2507.02935)
- [ADS listing for Theory of Mind in Action](https://ui.adsabs.harvard.edu/abs/2025arXiv250702935S/abstract)