Skip to main content
Back

LLMs Lose 60 Points of Accuracy When Reading Human Non-Verbal Cues

A new benchmark reveals LLMs lose up to 60% accuracy when inferring meaning from silence, gestures, and facial expressions alone

Paper, Full Access
  • LLMs lose up to 60 percentage points of accuracy when inferring meaning from non-verbal cues like silence, gestures, and facial expressions versus verbal responses.
  • The paper is the first systematic evaluation of LLMs on dialogue composed exclusively of non-verbal signals, covering silence, facial expressions, and movements.
  • Six LLM families from 3B to 100B+ parameters were tested; larger open models generally perform better, but API models like GPT-4o and Claude underperform relative to their scale.
  • Models fail in three main ways: literal interpretation, over-abstract interpretation, and outright misinterpretation of non-verbal cues.
  • Few-shot in-context learning (adding worked examples to the prompt) meaningfully improves non-verbal pragmatic inference, though gains vary by model family.
  • No code or model is released; findings are directly applicable to conversational AI, social robotics, and multimodal pipelines that pass non-verbal descriptions to LLMs.

Ask an LLM to read between the lines of a spoken conversation, and it does reasonably well. Ask it to interpret what someone means by staying silent, shrugging, or raising an eyebrow, and it falls apart. A new paper from researchers at Yonsei University and Korea University presents the first systematic evaluation of LLMs on purely non-verbal dialogue, and the results expose a fundamental gap in how these models understand human communication.

The finding is striking: LLMs struggle to infer underlying meaning from non-verbal responses, with accuracy dropping by up to 60 percentage points compared to verbal ones. That is not a minor regression. It suggests that the pragmatic competence LLMs have built up over trillions of tokens of text is far more fragile than leaderboard scores imply.

Words are only half the story

Pragmatics is the branch of linguistics concerned with how context shapes meaning beyond the literal words spoken. When someone asks "Is this a good time?" and you sigh heavily without answering, the sigh carries a complete communicative act. Non-verbal behavior remains a fundamental component of human communication, especially when deliberately utilized in isolation to convey indirect meanings.

LLMs have demonstrated remarkable capability for understanding semantics, but they often struggle with understanding pragmatics. Prior work has probed implicature, presupposition, irony, and metaphor, but always in the context of words. This work provides the first systematic evaluation of LLMs' ability to infer underlying intent from dialogue responses composed exclusively of non-verbal signals.

The practical stakes are real. As LLMs get embedded into conversational agents, social robots, and AI assistants, they will increasingly encounter descriptions of human behavior that go beyond speech. A model that cannot recognize that silence after "Did you enjoy the meeting?" signals discomfort will produce tone-deaf responses.

The benchmark: three flavors of silence