How you know the agent actually works

An ordinary software test rests on one assumption: the same input produces the same output. Check once that a tax function turns a thousand pounds into two hundred, and the test holds for years. A language model gives no such guarantee. Two runs of an identical request can take different routes, reach for a different tool and finish on a different sentence. A test comparing answers character by character raises false alarms on day one.

Since the wording cannot be checked, two other things are: the outcome and the repeatability. The outcome is whether the database holds what it should once the conversation ends. Repeatability is whether the same case ends the same way on a second run, and it is the part a demo never shows.

One attempt is not enough to conclude anything

In 2024 the company Sierra published τ-bench, a public set of tasks in which an agent talks to a simulated customer and uses tools. Grading ignores wording: it compares the database state at the end of the conversation with a recorded target state. The authors added pass^k, the share of cases passed on all k consecutive attempts. Their summary: the best systems of the time solved fewer than half the tasks, and retail pass^8 fell below 25 per cent.

The results table in the τ-bench repository gives exact figures. The strongest agent measured there, built on Claude 3.5 Sonnet from October 2024, passes 69.2 per cent of retail cases on one attempt and 46.2 per cent when the same case must succeed four times running. In the airline domain, where the rules are stricter, it is 46.0 per cent on one attempt and 22.5 per cent on four. The paper describes 115 retail tasks and 50 airline ones.

How many cases are enough

In 2024 Evan Miller of Anthropic set out how to compute the uncertainty in these measurements, treating a test set as a statistical sample. One figure cools the enthusiasm: detecting a three percentage point difference between two versions, at 80 per cent power and 5 per cent significance, takes roughly 969 questions. A smaller set is not useless, but it catches only a large difference.

The same paper gives two ways down from that number. The first is paired comparison: take the difference question by question rather than scoring each version separately, which removes the variation that comes from some questions simply being harder. The second is running each question several times: one attempt to two cuts the variance by a third, four attempts by half, against a ceiling of two thirds that repetition alone cannot pass.

• Thirty real production queries, five of them cases the system has already failed.
• A one-sentence description of a good answer, written before you see the output.
• Every case run at least three times, scored on the share that passed every time.
• The same set run on both versions on the same day, because a paired comparison is far more sensitive.

Four ways to grade one answer

Grading is a separate decision from the set of cases, and most projects use several methods at once.

• Fixed rules, meaning a mechanical comparison. OpenAI’s documentation calls this a string check grader: one when the output matches the reference, zero otherwise. Cheap, instant, suited to formats and categories.
• Factual assertions, meaning whether the right order number appears and the database state after the conversation matches the expected one. That is how τ-bench grades.
• A model as judge, meaning a second model scoring the answer against a rubric. Anthropic’s documentation says the judge should be a different model from the one that produced the answer.
• A human grading a sample. The slowest and dearest option, so reserve it for what no rule can describe.

The same Anthropic page states a principle that sounds wrong and is practical: more questions graded automatically, even at slightly lower signal, beat a handful graded by hand. Hand grading does not scale to a daily run, and a set that runs quarterly catches nothing in time.

How far a model judging a model can be trusted

That question has a measured answer. The team behind MT‑Bench and Chatbot Arena collected judgements from 58 experts, about 3,000 expert votes and 30,000 conversations from the open arena, then checked how often a model acting as judge agreed with a person. Counting only non-tied votes, GPT‑4 agreed with a human expert 85 per cent of the time, while two human experts agreed 81 per cent of the time. Two random judges would agree 50 per cent.

The same paper marks the limits. Agreement tracks the gap between the answers compared: near 100 per cent when one is clearly better, about 70 per cent when they are close. The authors name three biases in the judge: a preference for the answer shown first, for the longer answer, and for its own answers. Shown the model’s reasoning after disagreeing with it, people called that reasoning sound in 75 per cent of cases and changed their choice in 34 per cent.

So a model judge suits a daily screen and obvious differences, not the question of whether version A beats version B by two points. That needs a human sample or a rule.

Regressions, or the fix that breaks three other cases

The commonest way an AI deployment quietly degrades: a customer reports one wrong case, somebody adds a sentence to the prompt, that case starts working, and three others stop. Without a set that runs on every change nobody notices, because nobody is checking those three any more. A case, once added to the set, stays in it for good, including after it looks solved.

The test set itself carries bugs. SWE-bench, a public benchmark built from real GitHub issues, holds 2,294 tasks, and its SWE-bench Verified variant is 500 of them checked by hand by engineers for whether they can be solved at all. If a well known benchmark needs that hygiene, your thirty cases will need a review too.

What to log from day one

Evaluation cannot be added retrospectively with nothing to build cases from. Logging costs a few hours at the start and decides whether there is anything to analyse later.

• The complete input, including attached documents and the conversation history.
• The prompt version and exact model name, or you cannot reconstruct what changed.
• Every tool call and what the tool returned.
• The complete output, and whether the user accepted, corrected or abandoned it.
• A run identifier, the duration and the token count, your billing unit.

What the measuring itself costs

Less than people expect, as long as you count tokens rather than guess. Anthropic’s pricing page gives a worked example: one customer support conversation averages about 3,700 tokens, and ten thousand of them on Claude Haiku 4.5 cost about 37 dollars. Two hundred cases run five times each is a thousand such conversations, roughly 3.70 dollars for the whole exercise.

Then there is the judge: grading every answer with a second model means paying twice, and a judge dearer than the system can cost more than the run itself. Anthropic lists 1 dollar per million input tokens and 5 per million output for Claude Haiku 4.5, 2 and 10 dollars for Claude Sonnet 5, and batch mode, for work that can wait, halves both. The real cost of evaluation is the time people spend writing down what a good answer looks like, not the API bill.

What this measurement does not promise

A test set measures what you put into it and nothing else. It will not catch a situation nobody anticipated, because that situation is not in the set. A score of 85 per cent on your own set is not 85 per cent on live traffic: real customers write differently from whoever wrote the cases. The τ-bench and MT‑Bench figures come from 2023 and 2024 models, so read them as a description of the phenomenon. The drop across repeated attempts is real; its size in your case will differ.

The only measurement that settles anything is your cases, your definition of a good answer, and two versions compared on the same set on the same day. If you want to see what that looks like for a process of yours, write to us. Thirty real queries and an hour agreeing what success means is usually enough to start.