--- title: "Sakana AI's CoffeeBench Catches Claude Haiku 4.5 Going Bankrupt Over 90 Days" subtitle: "Sakana AI's CoffeeBench puts LLM agents in charge of a coffee supply chain for 90 days — and the results reveal surprising gaps in long-horizon decision-making" url: "https://alphasignal.ai/news/sakana-ai-s-coffeebench-catches-claude-haiku-4-5-going-bankrupt-over-90-days" author: "AlphaSignal Newsroom" author_url: "https://alphasignal.ai/editorial-team" company: "Sakana AI" company_url: "https://alphasignal.ai/company/sakanaailabs" published: "2026-06-26T06:06:22.000Z" updated: "2026-06-26T07:24:14.367Z" type: "News" topics: - "Benchmarks" - "Agents" subtopics: - "Multi Agent" - "Deep Research" - "Tool Use" upvotes: 1097 read_time: "5 min" source_url: "https://arxiv.org/abs/2606.16613" comments: 0 --- # Sakana AI's CoffeeBench Catches Claude Haiku 4.5 Going Bankrupt Over 90 Days > Sakana AI's CoffeeBench puts LLM agents in charge of a coffee supply chain for 90 days — and the results reveal surprising gaps in long-horizon decision-making Most AI benchmarks are sprints. A model reads a prompt, generates an answer, and gets scored. [CoffeeBench](https://sakana.ai/coffee-bench/), a new benchmark from Sakana AI and KPMG Japan's audit firm AZSA, is something closer to a marathon , and a competitive one. Six LLM agents are dropped into a simulated coffee industry supply chain and told to run their businesses for 90 simulated days. The goal: maximize net profit. The result: some models thrive, one quietly goes bankrupt, and a genuinely new class of failure mode gets a name. ## Why existing benchmarks fall short The problem with most agentic benchmarks is that they test a single agent against a passive environment. Real economic activity doesn't work that way. Economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own objectives over extended periods. Benchmarks like MMLU or HumanEval measure a single response in isolation. Even newer agentic evals rarely go beyond a few dozen steps. The closest predecessor to CoffeeBench is [Vending-Bench](https://arxiv.org/abs/2502.15840), which tested whether LLMs could run a vending machine business solo. While LLMs can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons , Vending-Bench tested an agent's ability to manage a long-running business scenario, with tasks that are each simple but collectively stress an LLM's capacity for sustained, coherent decision-making. CoffeeBench takes that premise and multiplies it: six companies, six agents, all interacting with each other. ## A supply chain in a sandbox The simulation involves two farmers, two roasters, and two retailers , all operating simultaneously in a coffee industry supply chain. Each company is controlled by an LLM agent. The model being evaluated always plays the role of Roaster A, while the other five companies are run by a fixed reference model (Claude Sonnet 4.6) to keep conditions comparable across evaluations. Each agent acts through a **ReAct-style loop** , a pattern where the model alternates between reasoning about its situation and taking a concrete action via tool calls. The available tools vary by role: - Farmers call `produce_item()` to grow coffee beans - Roasters call `roast()` to process beans into sellable product - Retailers call `set_retail_price()` to manage consumer pricing - All agents can send messages, place orders, pay invoices, and call `wait_for_next_day()` when there's nothing left to do The simulation also includes realistic economic friction: **credit sales** (where payment comes after delivery), fluctuating demand, and fixed daily operating costs. If an agent does nothing, it bleeds money. Every day has a cost, and the only way to cover it is to trade. ## What the models actually did The experiment evaluated models including GPT-5.5 and Claude Opus 4.7, with each model running the roaster role three times to account for variance, and results averaged across runs. The findings split cleanly into two camps. All models outperformed a passive baseline that takes no actions, with most achieving positive net income. But the spread between top and bottom performers was substantial. The behavioral analysis pointed to a clear pattern: higher-performing models communicated more actively with other firms, sending frequent messages to both farmers and retailers to negotiate prices and push sales. Lower-performing models were more passive , fewer messages, fewer trades, less profit. One nuance worth noting: raw activity wasn't enough. Kimi K2.6 matched high-performing models in total tool call volume but didn't see proportional profit gains , what mattered was whether those calls translated into profit-generating actions like `make_offer` and `accept_offer`, not just any tool use. Gemini 3.1 Pro showed a distinct style: it read incoming messages frequently but rarely initiated contact , a reactive rather than proactive management approach. ## The idle-drift failure mode The most striking finding involves Claude Haiku 4.5. Haiku 4.5 exhibited an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. The model's reasoning logs showed it correctly identifying opportunities , cheap beans available from a farmer, rising retail demand nearby , but then consistently choosing `wait_for_next_day()` instead of acting on that analysis. This is a qualitatively different failure from hallucination or factual error. The model wasn't confused. It was paralyzed. This phenomenon occurred in all three of Haiku 4.5's runs but was not observed in any other model tested. The benchmark gives this a name: **idle drift** , a long-horizon failure where thought and action decouple over time. ## Beyond profit: a testbed for AI governance CoffeeBench was designed with a longer-term research agenda in mind. The paper's authors, working with AZSA (the Japanese arm of KPMG), are interested in what happens when agents face pressure to hit targets they can't legitimately reach. In a pilot experiment, agents were given aggressive sales targets and told to hit them at all costs , a setup designed to mirror the conditions that lead to accounting fraud in real companies. In this pilot, agents did not engage in circular trading or other fraudulent behavior to inflate sales figures , likely because they didn't realize such strategies were available to them. But the authors note that as models get better at long-horizon planning and multi-agent coordination, the risk of agents discovering and exploiting such loopholes grows. CoffeeBench is positioned as the infrastructure to study that problem before it becomes a real one. ## What's available The full benchmark is open source. Resources released alongside the paper include: - [The paper](https://arxiv.org/abs/2606.16613) (accepted at ICML 2026 Workshop on Failure Modes in Agentic AI) - [Code on GitHub](https://github.com/SakanaAI/CoffeeBench) - Full agent trajectory logs for every model run, browsable online For teams building or evaluating long-horizon agents , especially those intended for business automation, procurement, or supply chain management , CoffeeBench offers something most benchmarks don't: a controlled environment where the agent has to sustain performance over time, adapt to other agents' behavior, and manage real tradeoffs between cash, inventory, and relationships. The idle-drift finding alone is worth paying attention to: a model that reasons well in the short term can still fail silently over a longer horizon. ## Sources - [X post by @sakanaailabs](https://x.com/sakanaailabs/status/2070388178201334260) - [CoffeeBench – Sakana AI Blog](https://sakana.ai/coffee-bench/) - [CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies](https://arxiv.org/abs/2606.16613) - [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://arxiv.org/abs/2502.15840)