Agents in e-commerce: what works today, measured

In a typical online shop, customer service comes down to a dozen recurring scenarios and a long tail of unusual cases. Owners ask us the same two things: how much of it can be handed to a machine, and how far the machine can be trusted. Both now have numbers behind them, from benchmarks where agents serve simulated shop customers and from one deployment that published its promise and its later accounts.

A chatbot answers, an agent reaches for tools

Anthropic, a company that builds models, meaning programs that generate text, separates the two in its engineering write-up from December 2024. A workflow is a system where the model and its tools are led along a path written in advance in code. An agent is a system where the model directs its own process and its own use of tools, keeping control over how the job gets done. A tool here means a function in your system that the model may call: fetch the order, change the delivery address, issue a refund.

The difference is about permissions. The same model with access only to a knowledge base is a page of frequently asked questions dropped into a chat window. Give it write access to the order system and it becomes something you have to hold to a standard, like any other worker.

• Read: order status, customer history, stock levels.
• Write: a CRM note, a carrier claim, a corrected document.
• Rules: what the agent may do on its own, and what needs a person’s approval.

How many cases an agent closes first time

The benchmark closest to e-commerce is called tau-bench and was published in June 2024. It works simply. A simulated customer holds a conversation with the agent, the agent has the tools of its industry plus a written policy, and at the end the state of the database is compared with the annotated goal state. Points come from the state of the system after the conversation. Well-turned sentences earn nothing.

The second version, tau2‑bench from June 2025, measures three industries: retail, an airline and telecoms. In retail, Claude 3.7 Sonnet passes 79% of cases and GPT‑4.1 passes 74%. On the airline the same models fall to 50% and 56%, and on telecoms to 49% and 34%. Retail is the easiest of the three measured industries, and roughly one case in five still ends badly.

The number that decides: the same case four times

A success rate over a single attempt says little about what a customer will see, because in support the same case returns hundreds of times a day. The authors of tau-bench therefore introduced a measure called pass^k: the share of tasks the agent passes in all k independent runs. At k of one it is the ordinary success rate. At k of four it asks whether the agent can handle the same case four times in a row.

In retail, Claude 3.7 Sonnet drops from 79% to 60% and GPT‑4.1 from 74% to 53%. On telecoms the same Claude falls from 49% to 25%, losing half its score. The original 2024 paper reported something harsher still: a GPT‑4o agent averaged over 60% task success, and across eight runs it dropped below 25%. That is the honest number of this field, and it rarely reaches a slide.

The agent that clicks through a browser itself

A second kind of agent gets no ready-made tools. It gets a browser and clicks through pages the way a person does. That was measured in 2025 across 300 real tasks on 136 websites, every run scored by people, in a COLM paper titled “An Illusion of Progress?”. The title is the most useful hint in it.

The two best systems, OpenAI’s Operator and Claude Computer Use 3.7, finish 61.3% and 56.3% of tasks. The other four stop at around 30%. The split that matters more for you is by task length. On short tasks, the ones a person completes in five steps or fewer, Claude 3.7 reaches 90.4% and Operator 83.1%. Difficulty rises mainly with the number of steps.

The authors did one more thing worth knowing before you talk to a vendor. They built a crude agent able only to run searches. On WebVoyager, an older and widely quoted benchmark, it passes 51% of tasks. On the new set it passes 22%: about 50% of easy tasks, 18% of medium and 3% of hard. Some of the loud results out there are measuring how easy the tasks were.

What to hand over first

• Short read-only cases: order status, expected delivery date, whether a size is in stock. Five steps or fewer is where measured systems reach 90%.
• Reversible operations: an address change before dispatch, a return label, a note on a record. A mistake costs a minute of someone’s time.
• Drafting answers for a person: the agent gathers data from three systems and proposes the wording, your team sends it. An error never reaches the customer.
• For people: money going out above an agreed threshold, disputed complaints, cases with feelings in them, and anything that cannot be undone with one click.

That split follows from the numbers. Since success falls with the length of the task and with the number of repeats, the permission boundary belongs where the cost of a mistake starts growing faster than the time saved. Anthropic recommends two plain guardrails: pauses for human confirmation at set checkpoints, and a hard cap on the number of steps after which the agent stops. It also warns about errors compounding once an agent runs on its own across many steps.

One deployment that published both sides of the ledger

Klarna, the Swedish payments company, launched an assistant built with OpenAI in February 2024 and published results a month later. 2.3 million conversations, two thirds of all customer service chats, “the equivalent work of 700 full-time agents”. Time to resolve a case fell from 11 minutes to under 2 minutes, repeat enquiries fell by 25%, coverage reached 23 markets and more than 35 languages, and satisfaction was “on par with human agents”. Estimated effect on 2024 profit: 40 million dollars. The release also carries a line that is easy to miss: customers can still choose to interact with live agents if they’d prefer.

A year on, the same things appear in the quarterly reports and look more modest. Klarna’s first quarter 2025 report gives customer service cost per transaction: 0.32 dollars in the first quarter of 2023, 0.25 a year later, 0.19 in the first quarter of 2025. A 40% fall over two years, and by the company’s account with no drop in consumer satisfaction after the assistant went live.

There is also a number that never reaches a slide. In the third quarter 2025 report the cost line “customer service and operations” stands at 53 million dollars against 44 million a year earlier: up around 20%, on revenue up 28%. Automation improved the unit cost and did not remove the department. Those are two different things, worth separating before you sign off a budget on the strength of a headline about 700 jobs.

What these numbers do not promise

Every one of these results comes from someone else’s environment. The shop in tau2‑bench has a dozen tools and a clean database, while yours has a five-year-old integration and three free-text fields where the sales team keeps its notes. The customer in that test is simulated by a model and behaves better than a real one. In building Online‑Mind2Web, 47% of the tasks drawn from an older set turned out to be invalid or to carry outdated solutions, because the websites had moved on. Your website moves on the same way.

The only test that counts is your own queue. Take the last hundred cases of one type, write down what “resolved” means, and put each of them through the agent several times. That result tells you whether the agent can answer a customer or should for now work for your team. If you want to run it on your own data, write to us.