Qwen's E-Commerce Bench Exposes How Badly AI Agents Fail at Running a Business

Running a business for a year involves noticing that a supplier keeps sneaking in price hikes, remembering that last summer's promotion drained your cash, and deciding whether to trust a vendor offering a suspiciously good deal. That is exactly the kind of drawn-out reasoning most agent benchmarks avoid, and it is what the Qwen team's new E-Commerce Bench tries to measure.

E-Commerce Bench is the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation. Agents start with a fixed budget and have to actually run the store, from sourcing to cash management, over the equivalent of 365 simulated days.

Why short-horizon benchmarks miss the point

Most agent evaluations chain together a handful of tool calls and call it a day. Long-horizon agentic tasks require LLMs to continually explore, learn from experience, and adapt their policies across evolving environments with long-range dependencies, sometimes over thousands of steps. The interesting question is whether a model gets better at a task after doing it fifty times against the same counterparty. Current models turn out to be surprisingly weak on exactly that.

Inside the simulation

The environment holds 6,886 products from real e-commerce data across 60 categories, served by 576 suppliers of which 152 are fraudulent across 5 scam types, plus a fixed year-long calendar of 8 promotions and 10 market events that continually reshapes demand. Much of it stays hidden, so the agent must learn which store types pay through action alone, across up to 4,000 turns per episode.

The benchmark's reproducibility comes from a deterministic design on both sides of the market. Customers follow a fixed demand model, and a Deterministic Negotiation Kernel fixes every price, concession, and accept/reject decision while an LLM only renders those decisions as dialogue. The supplier's numbers are pinned down by rules; a separate LLM writes the sales-rep dialogue on top. A lucky sample cannot swing a negotiation, so score differences reflect agent behavior rather than random variation.

Each agent receives a merchant account, ¥100,000, and the 2026 calendar. It can open up to four stores across twelve types, research the market, negotiate with suppliers, price goods, fulfill orders, and handle returns. Looking things up costs the same simulated minutes as acting, forcing real tradeoffs about what to pay attention to. Revenue does not land instantly: money flows through an escrow pipeline from sale to platform wallet to bank, while every expense hits the bank the day it is incurred. Ten of the 90 evaluation episodes ended in bankruptcy despite otherwise profitable plans.

How 18 models actually performed

The team evaluated 18 models, 8 proprietary and 10 open-weight, across 7 vendor families including Anthropic, OpenAI, Google, Qwen, GLM, Kimi, and DeepSeek, with 5 independent episodes each for 90 total runs.

The headline numbers show enormous spread. GPT-5.6 Sol earns the most, growing the ¥100,000 opening stake into ¥1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency. Among open-weight models, Qwen3.8-Max-Preview leads with ¥416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders. The best-to-worst ratio across all 18 models is roughly 1,264×.

Seven axes, no clear winner

Year-end assets alone hide most of what the benchmark exposes. Six other dimensions sit alongside profit: negotiation quality, fraud avoidance, cash-flow solvency, operational efficiency, execution, and learning over the horizon. When plotted together, six of the seven top vendor-family representatives fall below the 18-model median on at least one axis.

A systematic failure to learn from past prices

The benchmark introduces a metric called AnchorRatio: when an agent re-orders the same SKU from the same supplier, does it pay less than a random reshuffle of its own past quotes would predict? Below 1.0 means the agent got cheaper over time. Above 1.0 means it anchored on an early price and paid more than chance would produce.

Measured over 8,647 re-orders across 2,230 repeat supplier-SKU pairs, fifteen of eighteen models miss their own permutation null by more than two standard deviations in the expensive direction. That is a systematic unrevised anchor on an early price, not merely a failure to improve. Only Qwen3.8-Max-Preview beats its own shuffled ordering by a statistically significant margin. Two suspected culprits: context eviction destroys the tool results carrying earlier prices, and the persistent memory store available to agents is barely used.

What developers can do with it

E-Commerce Bench makes "long-horizon" mean something specific and measurable:

• Deferred consequences, so cash-flow mistakes surface weeks after the decision that caused them.
• Adversarial counterparties across 5 fraud types, including VIP fee extortion, quantity bait-and-switch, and fake urgency.
• Repeated interactions with the same supplier, where anchoring and memory management become measurable.
• A deterministic environment, so two models can be compared on identical years without noise.

Practical uses include stress-testing agent memory systems, evaluating whether a model's negotiation prompts translate into real bargaining leverage, and probing how tool-heavy agents balance information-gathering against action under a time budget. The code and data are released under Apache-2.0, and the full paper covers the negotiation kernel and metrics in detail.