Qwen, in collaboration with Taobao & Tmall Group, has released E-Commerce Bench — the first open benchmark in which an LLM agent with an initial capital of ¥100,000 manages an online store for a full year: negotiating with suppliers, setting prices, and managing inventory and cash flow. In a leaderboard of 18 models evaluated across seven dimensions, there is no dominant leader: final capital proved to be a poor proxy for business acumen, and agents will need to be chosen based on their capability profile rather than a single number.
What Happened
Qwen and Taobao & Tmall Group published E-Commerce Bench along with a preprint on arXiv, and also released the code, anonymized data, and leaderboard under the Apache-2.0 license. The environment is built on anonymized data from a real platform: the catalog contains 6,886 products across 60 categories, 576 suppliers operate in the market, including 152 fraudsters with five deception schemes; 8 promotions and 10 inevitable market events are scheduled for the year. The conditions are strict: the agent's working day is limited to 600 minutes, and revenue is received via escrow only nine days after shipment. 18 models participated in the run, each in five 365-day episodes: GPT-5.6 Sol earned the most — ¥1,431,425, a 14.3x increase — but ranked only 16th out of 18 in resilience to fraudsters. In 10 out of 90 episodes, the agent went bankrupt.
Context
The motivation for this development was the accumulated problem of agent evaluation: short scenarios and one-dimensional scores do not show how a model behaves when decisions accumulate over weeks and months. The authors' key methodological decision is a deterministic negotiation core: the reserve price and acceptable concessions are defined by product properties, not sampling, while the LLM merely voices the decisions of this core for both sides of the market. Without such a core, an LLM supplier could be persuaded to sell below cost, and the evaluation would turn into a jailbreak contest rather than a measurement of the agent's business qualities. Determinism makes the results reproducible, so the difference between models is explained by the agent itself, not by the randomness of the environment. The seven evaluation axes, including profitability, negotiation quality, resilience to fraudsters, solvency, and learning on the long horizon, are built as a capability profile rather than a single number. The authors propose this recipe — a deterministic core plus multi-dimensional evaluation on the long horizon — as a general template for future long-horizon benchmarks.
Why This Matters for the Industry
For the industry, the main point here is not the leaderboard, but the transferable method and ready-made infrastructure. An open stand with a deterministic core and multi-dimensional evaluation provides a template for reproducible long-horizon evaluations for in-house agent pipelines, including procurement, logistics, and financial operations, and the open license allows runs to be integrated into regression tests with each model update. The results are sobering: 6 out of 7 vendor families are below the median on at least one axis, meaning one-dimensional agent scores systematically mask profile weaknesses, and a model for business process automation should be chosen based on its profile rather than an aggregate score. The most profitable and the most cautious models on this stand are different systems, so vendors will likely have to publish multi-dimensional capability profiles instead of a single score. The expected trajectory for the coming months is derivative environments in adjacent domains, integration of runs into the CI of agent teams, and a race for the resilience-to-fraudsters axis, which currently remains one of the most difficult for all participants.
Why This Matters for Users
Teams building agent frameworks for automating procurement, pricing, inventory, and cash flow get a realistic open stand with constraints on time budget and liquidity. The repository can be cloned and run today: run your model through annual episodes, compare it with the leaderboard, and get a profile of strengths and weaknesses instead of a single number. The environment's constraints are useful as a checklist when designing your own agent systems: time limit for the working day, revenue delay, commissions, returns, reputation, and fraudsters — these are pressures that should be modeled before an agent enters a real business loop. For customers and buyers of agent solutions, this provides a selection criterion: request a multi-dimensional capability profile rather than a demo video. However, it is important to remember that this is an evaluation tool: it does not yet provide economic value in production and is not a ready-made means of autonomous store management.
What Is Still Unknown / Limitations
External validity has not been proven: this is a simulator with a deterministic core and anonymized data, and the transfer of results to live platforms with non-deterministic counterparties is not confirmed in the available materials. An open question is whether the correlation between the leaderboard and real business will be maintained, or whether models will become so tailored to a specific environment configuration that transferability will weaken; versions of models and pipelines optimized for the benchmark are expected to appear. It is also unknown how quickly other teams will transfer the deterministic core recipe to their domains. Finally, choosing a model by profile does not yet bring economic value in production: today's value lies in regression evaluations and model comparison, not in autonomous operations.
Sources
- Qwen Blog — E-Commerce Bench: Long-Horizon Operations, Multi-Dimensional Evaluation
- arXiv — E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation (arXiv:2608.30730)
- GitHub — QwenLM/E-CommerceBench: code, data, and leaderboard (Apache-2.0)
Author
Look at AI, editorial team