GAIATrace Opens the Black Box Inside Multi-Agent AI Pipelines
GAIATrace and Vidur-Agent give researchers the first token-level window into how complex multi-agent AI systems actually behave under real workloads.
Takeaways
- GAIATrace is the first token-level trace dataset of complex multi-agent AI systems, capturing full reasoning tokens, task structures, and all LLM activities across hundreds of tasks.
- Vidur-Agent is a trace-driven simulator that replays GAIATrace to enable cheap, reproducible infrastructure experiments without re-running live agents.
- Reasoning tokens make up 47-70% of all generated tokens in the Main-LLM -- a cost that most existing datasets and simulators completely ignore.
- Tool calls are not the dominant overhead for complex general tasks; LLM inference (especially Sub-LLM prefill and Main-LLM decode) dominates instead.
- The two systems studied -- MiroThinker and OWL -- show that architectural choices like ReAct-style history accumulation significantly affect KV cache hit rates and token costs.
- Both GAIATrace and Vidur-Agent will be fully open-sourced upon publication.
Everyone is building agentic AI systems. Fewer people actually understand what is happening inside them. A new paper from researchers introduces GAIATrace and Vidur-Agent, a paired dataset and simulator designed to finally shed light on the system-level behavior of complex, multi-model agentic pipelines -- the kind that orchestrate multiple LLMs, invoke dozens of tools, and run for hundreds of turns to complete a single task.
Agentic AI completes tasks through iterative planning, tool use, and reasoning based on observed outcomes -- but despite its popularity, its system-level behavior remains poorly understood, particularly for complex architectures, owing to highly non-deterministic execution, prohibitive evaluation costs, and limited visibility into proprietary models. In other words: these systems are expensive to run, hard to reproduce, and nearly impossible to study at scale. This paper attacks all three problems at once.
The black box problem with agentic systems
Most of what we know about how LLMs behave at the infrastructure level comes from chat-style workloads: a user sends a message, the model responds. Agentic workloads are fundamentally different. Agentic AI consists of autonomous pipelines in which LLMs plan, invoke tools, observe results, and iterate until a task is complete. The state space explodes: multiple models may be active simultaneously, tool calls interleave with reasoning steps, and a single user task can spawn hundreds of sub-queries across different LLMs.
More complex setups -- multi-agent and multi-model systems tackling general tasks with a broad range of tools -- remain largely uncharted, because bringing up such systems and running controlled experiments are expensive and challenging. Without a shared dataset or a way to replay executions cheaply, every research group studying agentic systems has to re-run everything from scratch, burning GPU budget and getting different results each time due to non-determinism.
Two artifacts, one goal
The paper introduces two complementary contributions. The first is GAIATrace, a token-level execution trace dataset. It captures two state-of-the-art agentic systems -- MiroThinker and OWL -- running GAIA, a benchmark composed of a heterogeneous mix of general-purpose tasks. GAIA (General AI Assistants) is a benchmark developed by Meta and Hugging Face that tests fundamental capabilities including multi-modal understanding, web browsing, reasoning, and complex problem-solving.