Building and operating AI9 min read

Evaluating an AI Product

How to define quality, build representative cases, inspect agent behaviour and use evaluation evidence in product decisions.

An AI component is tested against representative cases before a release control.
On this page
  1. 1.Start with the decision
  2. 2.Build cases from the work
  3. 3.Decide how to grade
  4. 4.Work through a case
  5. 5.Inspect the path as well as the result
  6. 6.Use the result to decide
  7. 7.Learn from use
  8. 8.Try it: draft and calibrate an eval

A document-processing team has a promising prototype. It extracts the right fields from five invoices in a demo. Before a pilot, the team needs to know what happens with missing fields, conflicting totals and documents the system was never meant to process.

An evaluation, or eval, is a repeatable way to ask those questions. It combines cases, an expected outcome or rubric, a way to assess the result and a record of what happened. The first version can be small and manually reviewed. Its job is to help the team make a decision, not to produce an impressive aggregate score.

Start with the decision

Write down the task, who experiences it and the consequence of a mistake. Then decide what the team needs to learn. Can the system extract the correct amount? Does it identify missing information? Can it use a tool without exceeding its authority? Different questions call for different cases and checks.

For the hypothetical invoice workflow, a case might contain a document, the fields to extract and the rules for flagging discrepancies. The expected behaviour could require a correct total, a reference to where it appears and a flag when information is missing. A finance specialist should help define what makes the result acceptable.

The first eval cannot capture every possible request. Give it enough variety to expose the important risks, record where the sample is thin and add cases as the team learns.

Build cases from the work

Use representative tasks, known failures, examples from domain experts and cases where the correct behaviour is to ask, refuse or escalate. Include different user groups and important exceptions. Preserve the input, relevant context, expected behaviour and why the case matters, subject to data permissions and privacy controls.

Synthetic examples can help explore variations, but they inherit assumptions from whoever or whatever created them. Check them against real work and expert judgement. Keep a separate set for comparing changes so that repeated tuning on familiar examples does not give a false sense of progress.

A capability set explores what the system cannot yet do well. A regression set protects behaviour the team already relies on. Some cases can move from one purpose to the other as the system improves, but the distinction should be clear when results are interpreted.

Decide how to grade

Different aspects of a task need different checks.

MethodUseful forLimitation
Deterministic checkStructured output, permissions, known rules, tests and observable final stateCannot judge every ambiguous or qualitative outcome
Human reviewDomain judgement, new failures and calibrating other gradersTakes time and reviewers can disagree
Model-based graderApplying a clear rubric at a larger scaleCan make systematic mistakes and needs calibration against human judgement

Choose a method that provides useful evidence for the decision. For consequential work, use independent checks where possible and inspect disagreements. Two reviewers may disagree because the rubric is unclear, the source evidence conflicts or more than one answer is acceptable. Resolve the underlying issue before treating the score as authoritative.

Assign an owner to maintain the quality definition and involve the people who understand the task. Preserve enough context to explain why a case passed or failed.

Work through a case

Suppose the team wants to test an internal tool that prepares invoices for a person to review. In this hypothetical workflow, the system may extract and flag information, but it cannot approve payment. The team agrees that a reviewer must be able to find the source of each extracted amount and that a conflicting total must be flagged, not silently corrected.

One ordinary case contains an invoice with a clear supplier, invoice number and total. The expected result has the correct fields and identifies where the total appears. A deterministic check can compare the extracted number with the labelled value; a person can check whether the source reference is useful.

An exception case contains a printed total of $110 and line items that add to $100. The tool returns $110 with a correct source reference, but does not flag the discrepancy. A reviewer marks the extraction as correct. Another marks the task as failed because the discrepancy was missed. Both are describing something real: the value was copied accurately, but the workflow did not draw attention to a condition the reviewer needed to resolve.

The team separates those checks in the rubric. Field extraction passes; discrepancy handling fails. The finance specialist confirms that the exception belongs in the review queue. The team adds the case to its regression set after fixing the behaviour and checks similar cases with missing tax or incomplete line items. These examples illustrate a process, not an observed test result or a complete invoice standard.

For this pilot, the team had agreed that known discrepancies must be flagged before the tool is used on that class of invoice. It therefore keeps the pilot to a narrower invoice type covered by its tests, with a person screening for exceptions before processing and reviewing every result afterwards. Conflicting-total cases remain manual. A later expansion needs a retest and evidence from actual review work. The offline result alone cannot establish whether the tool saves time or will handle every invoice in the pilot.

Inspect the path as well as the result

Separate inspection stations evaluate an AI system's execution trace and final outcome.

An agent can reach the right answer after accessing information it should not have seen, making unnecessary tool calls or taking a risky action. Conversely, a reasonable path can end in an incomplete result. Inspect the final state and the trace separately.

Depending on the workflow, check tool selection, permissions, repeated calls, recovery after errors, escalation, latency and cost. Test attempts to manipulate the system through untrusted material. A prompt instructing an agent to behave safely does not replace independently enforced permissions and action limits. The security playbook covers these tests in more depth.

Use the result to decide

Compare the candidate system with a relevant baseline. Look at important slices, not only the average: a change may improve common cases while making a rare, consequential case worse. Repeat tests when model behaviour varies and report enough detail to understand the uncertainty.

What a small result can tell you

Imagine a test set of 50 invoices: 45 routine examples and five with conflicting totals. A team compares two versions on the same set. These numbers are illustrative; the cases are deliberately selected, not a random sample of all invoices.

ResultVersion AVersion B
Routine cases passed45 of 4545 of 45
Conflicting-total cases flagged0 of 53 of 5
All cases passed45 of 5048 of 50

Version B improves on the selected exceptions, but it still misses two of them. Its overall result of 48 out of 50 would hide that failure if the team looked only at the total. Neither result tells the team the rate of errors across every real invoice: the mix and number of cases were chosen for the test, and five exception cases give limited evidence about the variety of exceptions in practice.

If the model can vary between runs, repeat the same case and record each attempt along with the configuration. A single pass shows that the system succeeded once under those conditions; it does not establish that it will do so reliably. Repeated attempts on the same five examples help examine variability, but they do not create five new independent invoice types or replace broader coverage.

The team should decide whether missed discrepancies are acceptable for a supervised pilot, what a reviewer must check, and which cases remain manual. Add representative cases with domain help and compare changes with an appropriate baseline. Use production observation and customer research to answer questions the offline set cannot.

Set a quality threshold before the decision, informed by failure consequence, alternatives and the ability to review or recover. A team may decide that a system is suitable for an internal draft but not for sending a response. An offline pass does not establish customer adoption, security, sustainable cost or operational readiness.

If the evidence is insufficient, narrow the pilot, improve the test or delay the commitment. If the result is acceptable, decide who will monitor the workflow and what would cause it to pause.

Learn from use

After release, inspect sampled work, corrections, incidents, tool failures and user feedback. Policies, inputs and integrations can change even when the model does not. A correction can suggest a useful new test case, but it is not automatically a correct label: verify the context and protect any sensitive data before adding it.

When a failure appears, determine whether to repair the source, change a tool or permission, revise the workflow, add an eval case or narrow the product boundary. Retest the affected behaviour. Keep the suite small enough that someone can understand and maintain it, while covering the risks the team has chosen to own.

Try it: draft and calibrate an eval

Describe a real task and its current authority. Use the invoice example above if you cannot share an actual workflow. The prompt produces candidate cases, not an evaluation suite ready for a release decision.

Prompt
Help me draft evaluation cases for this AI workflow.

Task and intended user: [describe them]
What the system may read or do: [list its actual permissions]
What a useful result requires: [describe the quality standard]
Consequences of an error: [describe them]
Examples or rules I am permitted to share: [add them or say none]

Propose a small set of ordinary cases, important exceptions and cases where
the system should ask, refuse or hand off. For each, provide an input,
relevant context, expected behaviour, reason for inclusion and a possible
grading method. Identify rules or expected answers that need a domain
expert. Do not invent policy, label generated cases as synthetic and do not
claim that they represent real usage or establish a release threshold.

Take two proposed cases to someone who understands the work. Independently grade the same sample result, then compare your reasons. If you disagree, check the source and clarify the rubric before choosing a label. Replace or supplement generated cases with permitted examples from real work, and keep a separate comparison set. Decide the release threshold with the accountable team; a model cannot establish it from the prompt alone.

v3.1 · Updated Oct 2026