The product job5 min read

Run Discovery as a Learning System

How to investigate customer needs and technical capabilities, choose a test and keep evidence connected to decisions.

Different forms of customer evidence converge into a clearly framed product problem.
On this page
  1. 1.Follow the customer and the capability
  2. 2.Keep a problem record
  3. 3.Make the bet explicit
  4. 4.Choose evidence for the decision
  5. 5.Connect the evidence to a decision
  6. 6.Try it: challenge a product bet

A polished prototype can make an idea feel settled. The team may be discussing colours and launch dates while still unsure whether anyone needs the proposed workflow.

Before investing further, decide what you need to learn and what evidence could change your mind. That is the work of discovery.

Follow the customer and the capability

Two questions can move at different speeds. A customer may have lived with a difficult workflow for years, while a change in model capability makes a new approach worth testing. A model may also perform an impressive task that nobody needs done.

Keep a record of customer problems and the evidence behind them. Separately, track useful tasks that are not yet reliable enough for the intended use. For each possible match, identify the user, the current alternative, the technical limitation and a test that would justify revisiting it.

In the hypothetical support workflow, staff may spend more time locating policy than writing a reply. The next test might ask whether the system can retrieve the current rule and identify exceptions. A fluent draft alone would not answer that question.

Keep a problem record

Capture the affected person and situation, the observed problem, current alternatives, supporting and contradictory evidence, the outcome that would matter and the next unresolved question. Include source references where the team has permission to retain them.

A feature request belongs in this record as evidence. Ask what work sits behind it, but remain open to the requested feature being the right answer. Understanding the customer's work covers how to investigate urgency and alternatives.

Make the bet explicit

Write down what you observed, what you think it means, the intervention you want to try and the outcome you expect. Add the evidence that would justify the next investment, and what would lead you to change direction or stop. Agree on a threshold or decision rule before seeing the result where practical. If you change it later, record why.

A failed solution does not automatically invalidate the problem. If people are waiting for support because staff cannot find current policy, an assistant that drafts faster may fail while the underlying wait remains worth addressing. The next bet could focus on finding and checking the policy instead.

Interpret evidence in context. Interest, repeated use, payment and a customer outcome answer different questions. A positive interview does not establish retention; an offline evaluation cannot establish adoption. Choose the least costly test capable of informing the decision, not simply the easiest test to run.

Choose evidence for the decision

UncertaintyA useful starting pointWhat it does not establish
Problem significanceObservation, conversations, support or behavioural evidenceThat a particular solution will be adopted
InteractionA sketch or prototype tested with usersProduction readiness or sustained demand
Model capabilityRepresentative tasks, a rubric and an evaluationThat the workflow is useful or economical
Adoption and operationsA bounded pilot with relevant controlsThat every customer or task will behave the same way

Label what an artefact shows and what it simulates. A realistic prototype can expose an interaction problem, but its polish can also make untested assumptions look like facts.

An AI evaluation, often called an eval, tests a system against a set of cases and a defined standard. It helps answer whether the system can perform those cases; the evaluation chapter explains how to choose and assess them.

Evidence progresses from a single comment to observed behaviour, testing and measured outcomes.

Connect the evidence to a decision

After a test, record the result, its limitations and the decision. A failed solution may leave the underlying customer problem worth exploring; it may also reveal that the problem was less important than expected.

Keep a short delivery queue with clear owners and enough context to proceed. Review active problems when meaningful evidence arrives. Ask which uncertainty is now most important, whether the next test addresses it and which work should stop.

AI can help retrieve, compare or organise evidence. Check the source, the sample and any conclusion before relying on a generated summary.

Try it: challenge a product bet

Use this when a proposed feature is gathering momentum but the team has not agreed what it needs to learn. Provide only information you are permitted to share. For practice, use a fictional situation.

Prompt
Help me examine a product bet before my team commits to a solution.

The problem or request: [describe it]
What we have observed: [evidence, sources and dates where available]
The solution we are considering: [describe it]
Constraints and commitments: [what is known]

Separate observations from assumptions. Identify missing or contradictory
evidence. Suggest at least two plausible approaches, including a simpler
non-AI approach where appropriate. Identify the uncertainty most likely to
change our next decision and a small test for it. Explain what the test can
and cannot establish, and what result might lead us to change or stop.

Do not invent customer evidence, numbers or consensus. Challenge the
proposed solution without assuming it is wrong.

Check the observations against your sources and add anything the tool missed. Discuss the strongest alternative with your team. You still need to decide which test is feasible and whether its result would change your decision.

v3.1 · Updated Oct 2026