Skip to main content
Back

NoRA Benchmark Exposes How Frontier AI Fails Basic Social Reasoning

NoRA is a new video benchmark that tests whether AI agents can justify social actions from first-person video, not just pick from a menu of choices

Paper, Full Access
  • New benchmark: NoRA tests whether VLMs can generate and justify socially appropriate actions from first-person video, without multiple-choice scaffolding.
  • Core innovation: Models must produce a fact-reason-action support graph, linking visible scene facts to normative principles to proposed actions.
  • Dataset: 1,420 annotated egocentric video clips split into HumanGold-190 (human-annotated) and LLMSilver-1230 (LLM-annotated) sets.
  • Key finding: 12 frontier VLMs (GPT, Gemini, Grok, Qwen, Kimi) can spot plausible actions and scene facts but consistently fail to bind actions to the correct supporting evidence.
  • Evaluation: Three metrics -- action alignment, factual grounding, and support binding -- aggregated into a single grounded reasonableness score; no model saturates the benchmark.
  • Why it matters: Shifts AI safety evaluation from "did the model pick the right answer" to "can it justify the right answer for the right visible reasons."

As AI agents move from answering questions to operating in the physical world alongside people, a critical question emerges: can they reason about what is socially appropriate to do next, and can they explain why? A new benchmark called NoRA (Normative Reasoning for Action) is designed to find out, and the early results are sobering.

LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe behavior, yet existing approaches either assess normative judgment in text alone or reduce it to choosing among a fixed set of candidate actions. NoRA argues both approaches miss the point entirely.

The gap that multiple-choice can't measure

In practice, agents are never handed a menu of options; they must identify a reasonable action from scratch, grounded in visible facts and supported by inspectable reasons. This is the core insight driving the benchmark. When a robot or AI assistant is helping someone in a kitchen, a park, or a hospital corridor, there is no dropdown list of socially acceptable next moves. It has to figure that out on its own, and it has to be able to show its work.

Prior work in this space, like EgoNormia, already revealed alarming gaps. Current state-of-the-art VLMs lack robust grounded norm understanding, scoring a maximum of 54% on EgoNormia, with performance across norm categories indicating significant risks of safety and privacy when VLMs are used in real-world agents. But EgoNormia still relied on multiple-choice questions. NoRA takes the next step by removing the answer choices entirely.

Examples of first-person normative reasoning scenarios showing multiple-choice limitations and incorrect model reasoning

How NoRA works

NoRA is a visual first-person video benchmark that requires models to generate candidate next actions and justify each through an explicit fact-reason-action support graph. Think of this graph as a structured chain of evidence: the model must point to a visible fact in the scene, connect it to a normative principle (a reason), and then link that to the action it recommends. Every step must be traceable.