AI Product Metrics

TL;DR

• Most teams measure the model (evals) or the business (COGS). They skip the product layer: adoption, trust, and value delivery. That middle layer is where AI features live or die.
• Deployed is not adopted. Adopted is not valued. Track the funnel from feature shipped to measurable user outcome, or you'll mistake motion for progress.
• A weekly AI product review covering five metrics (adoption gap, trust calibration, escalation patterns, value delivery speed, cost efficiency) gives you the signal to invest, iterate, or kill.

---

Teams that ship AI features typically measure two things well: model quality (through evaluation frameworks) and unit economics (through COGS modelling). Both matter. Neither tells you whether the feature is actually working for users.

The product layer sits between model and business metrics. It answers the question every PM should be able to answer at any point: "Is this AI feature delivering value to users, and how do I know?" If your answer involves eval pass rates or inference costs but not adoption, trust, or time-to-value, you're measuring the engine and the fuel bill but not whether anyone is getting where they need to go.

I explored the full three-layer metrics framework (output, quality, economics) in How to Measure an AI Product. This chapter focuses on the product measurement discipline that connects those layers to decisions.

The adoption funnel: deployed is not adopted

A feature that exists in your product is deployed. A feature that users interact with is adopted. A feature that changes a measurable user outcome is valued. These are three different things, and most teams only track the first.

The adoption gap

Track a three-stage funnel for every AI feature:

Deployed. The feature is live. 100% of eligible users can access it. This is the number teams report when they say "we shipped AI."

Adopted. Users have interacted with the feature at least once in the measurement period. For inline AI, this might mean accepting a suggestion. For destination AI, it means navigating to the feature and completing a query.

Valued. The feature has measurably improved an outcome the user cares about: task completed faster, error rate reduced, decision made with better information. This requires connecting AI feature usage to downstream outcomes, which requires instrumentation most teams skip.

The gaps between these stages tell you different things:

A 10% adoption rate after three months is not "early days." It's a signal that the feature doesn't fit the workflow. Diagnose which gap is wide and address it, or kill the feature.

Measuring adoption across the autonomy spectrum

Adoption means something different at each level of autonomy:

Copilot features (suggestions the user accepts or dismisses): measure acceptance rate. A code completion tool with a 30% acceptance rate is useful. One with 8% is noise. Track acceptance rate by suggestion type, not in aggregate, because a blended number hides underperforming categories.

Co-driver features (agent proposes, human confirms): measure confirmation rate and edit depth. If users confirm the agent's proposal 80% of the time with no edits, the feature is earning trust. If they confirm 80% of the time but edit 60% of proposals before confirming, the feature is creating work, not saving it.

Supervised autopilot features (agent executes, human monitors): measure intervention rate and intervention cause. A ticket triage agent with a 5% intervention rate is working. One with a 5% intervention rate where all interventions are the same error type has a fixable gap. One with a 25% rate where interventions are varied has a systemic quality problem.

Full autopilot features (agent executes autonomously): measure outcome quality through sampling and downstream impact. If the agent processes invoices overnight, sample 2% the next morning and grade them. Track the downstream metric (processing time, error rate reported by finance) to confirm the autonomous workflow delivers equivalent or better outcomes.

Trust calibration

Trust is the product metric that determines whether AI features compound in value or plateau. Users who trust the AI appropriately use it for the right tasks and verify the right outputs. Users who over-trust it miss errors. Users who under-trust it do everything manually anyway.

Override rate as a trust signal

When users review AI output, how often do they change it? The override rate is one of the richest signals you have.

Below 5% in high-stakes domains: Rubber-stamping. Users have stopped checking. This is dangerous in regulated environments and means your human-in-the-loop is a formality, not a safeguard. Either the AI is genuinely that good (verify with independent audits) or users have disengaged from the review process.

10% to 30%: Healthy calibration. Users engage with the output, make targeted corrections, and trust the AI where it's reliable. This range typically correlates with the highest user satisfaction.

Above 50%: The AI output isn't useful as a starting point. Users spend more time fixing than they save by not starting from scratch. At this rate, the feature is creating an AI detour rather than eliminating work.

Track override rate by output type and user segment. Power users and novices will override at different rates for different reasons. Aggregate override rate is a starting point. Segmented override rate is actionable.

Confidence acceptance correlation

Plot two variables: the AI's stated confidence and whether the user accepted the output. In a well-calibrated system, high-confidence outputs are accepted at high rates and low-confidence outputs trigger more overrides.

If users override high-confidence outputs frequently, either the confidence scoring is miscalibrated (the model thinks it's right when it isn't) or users don't trust the confidence signal (they've learned to ignore it). Both problems need different fixes.

If users accept low-confidence outputs without checking, your UI isn't communicating uncertainty effectively. Revisit your confidence indicators.

Autopilot graduation rate

For features designed to move up the autonomy spectrum, track how many users have graduated from copilot to co-driver, or from co-driver to supervised autopilot. A feature where 40% of users have opted into higher autonomy after three months is building trust. One where 95% of users stay at the initial level isn't.

Graduation should be user-initiated (they choose to increase autonomy) and data-informed (you show them their personal accuracy stats to build confidence in the promotion). Forced graduation destroys trust.

Value delivery speed

How quickly does a new user get measurable value from an AI feature? This metric matters because AI features have a trust deficit that traditional features don't. Users expect traditional features to work. They expect AI features to fail. Every second of delay or confusion reinforces the scepticism.

Time-to-first-value. Measured from the user's first interaction with the AI feature to the first outcome they'd describe as valuable. For a document summarisation feature, this might be 30 seconds. For an agentic workflow, it might be a completed task cycle.

Inline AI features should deliver value in under 10 seconds. If a user has to configure, prompt, or wait longer than that for a first positive experience, adoption will plateau.

Session efficiency. For repeated interactions, track whether users get faster over time. A user who takes 5 minutes to complete a task with AI assistance in week 1 should take 2 minutes by week 4, as they learn the feature's strengths and limitations. If the time is flat or increasing, the feature isn't learnable. Users are fighting it rather than mastering it.

Escalation intelligence

Escalation rate (what percentage of tasks the AI routes to a human) is covered in the metrics framework blog post. The product discipline goes deeper.

Escalation clustering. Group escalations by cause. If 40% of escalations share the same root cause (a specific query type, a data format the model can't parse, a policy rule it misinterprets), that's your highest-ROI improvement target. Fix one pattern and the escalation rate drops measurably.

Escalation trend by cohort. New features escalate more. That's expected. The signal is the slope. If a feature's escalation rate drops from 30% in week 1 to 15% in week 4 to 8% in week 8, the system is learning (through HITL feedback, prompt refinements, and eval-driven improvements). If the rate is flat at 25% across all weeks, the improvement loop is broken.

Escalation cost. Every escalation has a cost: the human reviewer's time, the latency penalty for the user, and the opportunity cost of the reviewer not doing higher-value work. Calculate escalation cost per task and add it to your cost-per-task formula. A feature that looks margin-positive at the inference layer can be margin-negative when escalation cost is included.

The weekly AI product review

Model evals run continuously. Business metrics get reviewed monthly or quarterly. Product metrics need a weekly cadence, because AI features change behaviour faster than traditional software (prompt drift, model updates, input distribution shifts) and the feedback loops are shorter.

Five questions for the weekly review

Run this review for every AI feature in production:

1. Is adoption moving? Check the deployed → adopted → valued funnel. Compare week-over-week. If adoption is flat for three consecutive weeks, the feature needs intervention or a decision to kill.

2. Is trust calibrated? Check override rate and confidence acceptance correlation. Look for sudden shifts, which usually indicate a model update, a data quality change, or a UX issue introduced in a recent release.

3. Are escalations improving? Check escalation rate trend and cluster analysis. Identify the top escalation pattern and assign it as the week's improvement target.

4. Is value delivery getting faster? Check time-to-first-value for new users and session efficiency for returning users. Both should trend down over time.

5. Is it economically sustainable? Check cost per task including escalation cost. Compare to the ceiling established in the Definition of Ready. If cost is trending up, identify whether the cause is model, volume, or escalation.

Who owns what

The PM owns the review. Every metric has a clear primary owner who is responsible for diagnosis and action, and a secondary owner who provides the data or expertise.

Connecting metrics to decisions

Metrics that don't inform decisions are vanity metrics. Every product metric should connect to a specific decision type.

Invest when the adoption funnel is healthy (>50% adopted, >30% valued), trust is calibrated (override rate 10-30%), escalations are declining, and cost is within ceiling. This feature is working. Expand its scope or graduate it up the autonomy spectrum.

Iterate when adoption is moderate but the valued stage is weak (users try it but don't get lasting value), or when escalation patterns reveal specific fixable gaps. The feature has potential but needs targeted improvement.

Kill when adoption is below 10% after three months despite discoverability fixes, or when the cost ceiling is persistently breached with no path to improvement, or when override rates stay above 50% across all user segments. Not every AI feature should survive. The discipline to kill underperforming features frees resources for features that compound.

The weekly review forces this decision cadence. A feature that sits in "iterate" for more than two months without measurable improvement should be re-evaluated as a "kill" candidate.

The anti-pattern: the deployment dashboard

The team ships twelve AI features in six months. Leadership asks for a dashboard. The PM builds one showing feature count, total AI interactions, and average response time. The dashboard looks great. The numbers go up every month.

Nobody knows which features users actually value. Nobody tracks whether users who try the AI features continue using them. Nobody measures whether the AI output changes any downstream outcome. The dashboard reports activity, not value.

Six months later, a customer survey reveals that users find three of the twelve features genuinely useful, ignore six entirely, and actively avoid three because the AI gets it wrong often enough to be unreliable. The dashboard never surfaced any of this. It was measuring deployment, not adoption. Counting interactions, not outcomes.

The fix: kill the deployment dashboard. Replace it with the adoption funnel, the trust metrics, and the weekly review. Measure fewer things, but measure the things that tell you whether the AI is working for users, not just working in production.