top of page
搜尋

Evals Across an AI Product Team: Three Perspectives and the Essential Role of UX Research

  • Ying Chen
  • 8月3日
  • 讀畢需時 10 分鐘

已更新：8月9日

A high-level look at AI evaluation across product management, data science, and UX research, as well as the essential role UXR plays in defining what “better” means.


Welcome to Signals to Solutions, a newsletter about understanding products through user behavior, product strategy, and UX research. For this first article (originally published here), I want to begin with a sharing my perspective from reading and synthesizing select industry and academic sources about a word that appears in almost every conversation about building AI products: Evals.


Large language models are probabilistic by nature. Ask the same question twice, and you may receive two different answers. Both answers can sound plausible, but neither is guaranteed to be accurate, useful, or appropriate for the specific context of the user. This creates a sharp break from conventional software, where the same input is generally expected to produce the same output. So in the software, a bug either reproduces or it does not. But with an AI system, “Does this work?” is rarely a simple yes-or-no question. Instead, AI product teams are seen to evaluate the system’s behavior across many examples, user situations, and product contexts. They need to determine where the system performs reliably, where it fails, and whether changes genuinely improve the experience. And from the industry and academic research that I have referenced, I understand that evals is one set of approach that can help teams examine those questions.


However, one set of limitations from my readings that I find is this: different functions within an AI product team can use the word “eval” to mean different things.


Let’s think about an example together. Imagine a product roadmap meeting where someone says:

“We need to ‘eval’ this before we ship.”

Everyone agrees. The product manager may be thinking about launch criteria and whether the feature is ready for release. The data scientist may be thinking about model metrics or an online experiment. The UX researcher may be thinking about observing real users attempting meaningful tasks with the system. The team has agreed on the word, but not necessarily on the question(s) being answered. One way to interpret this is that AI response quality is multidimensional, and that a single evaluation method may not capture everything a team would want to understand about what could be improved and how.


For the purposes of this article, I focus on three perspectives that can each contribute to understanding whether an AI product is improving: product management (PM), data science (DS), and UX research (UXR). However, this is not intended to be an exhaustive taxonomy of the functions involved in AI evaluation.


Below, I outline at a high level how “evals” may be approached by these different functions, and how their perspectives could complement one another when assessing and improving AI responses. However, since this is still an evolving methodology, new methods and perspectives may be introduced in the future.


Product Managers' AI Evals: What does success mean for the product?


Industry sources point out that for product managers, AI evals help translate a broad product promise into concrete success criteria. For example, a team may be building an AI assistant that helps people summarize complex documents, and a PM may ask:


  • What types of documents should the assistant support?

  • What must it do reliably before launch?

  • How should quality change across different user use cases?

  • What signals will tell us whether the feature is creating value after launch?


These questions connect model behavior to the product’s intended purpose. So, a PM-led evaluation process might include representative test cases, quality thresholds, risk categories, and post-launch monitoring. Together, these can become an eval suite or a pre-release checklist that the team revisits as the product evolves.


One potential benefit of developing AI evals early is that the process can encourage a team to make its definition of “good” more explicit. Without that definition, two people may look at the same response and reach entirely different conclusions. One person may see a harmless wording issue, while the other may see a trust-destroying issue.


At the same time, a predefined eval set is necessarily bounded by the success criteria the team has identified and chosen to measure. A fixed eval set may gradually become outdated as the product evolves. It can continue producing reassuring scores while user expectations and real-world use cases move elsewhere. For that reason, it may be useful to revisit the direction and composition of evals as the product and its use cases evolve.


Data Scientists’ Online & Offline Evals: Is the model and its responses performing better?


Data science evals focus on measuring model or system performance statistically, as explained by industry sources. These evaluations are often commonly divided into offline and online methods.


Offline evals test a model against existing datasets, labeled examples, or predefined benchmarks. They can be useful because they can be run quickly, repeatedly, and without exposing users to an untested system.


A data science team might measure:


  • Accuracy or task-completion rates

  • Hallucination or factual-error rates

  • Latency and cost

  • Performance across different categories of requests


Offline evals are especially useful during development because teams can compare multiple prompts, models or agent configurations before deciding which version should reach users. However, offline success does not guarantee product success. A model can perform well on a static benchmark and still struggle with ambiguous user requests or changing user behavior.


This is one reason online evals can provide a useful additional perspective.


Online evals typically assess a system using live users or production traffic. Using methods such as A/B tests, DS can compare different versions of a model or experience and measure effects on user behavior.


Even then, online experiments provide only a partial view: they measure the outcomes a team has instrumented, within a particular experimental period and context. For example, a two-week A/B test may identify changes in clicks, task completion, or repeated usage. It may not reveal user trust slowly eroding after several confidently wrong answers. It may not show users becoming overly dependent on the system or quietly abandoning it because correcting its mistakes takes too much effort.


Taken together, data science evals can provide evidence about measurable changes in system performance, while additional methods may still be needed to understand how those changes are interpreted or experienced by users.


UX Researchers’ User Evals: Does the system work for people in context?


After learning from industry UX researchers’ online sharings and academic researchers’ publications, I find that UX researchers may use User Evals (or “UX Evals”) to examine what happens when people use an AI product in the context of real-world goals and tasks. For example, a UX researcher may investigate:


  • Whether users understand what the AI can and cannot do

  • How users decide whether to trust an answer

  • What users do when the system is uncertain or wrong

  • Which failures feel minor, and which fundamentally damage user confidence

  • Whether the system helps users achieve the outcome they actually care about


A user brings their own goal, expectations, previous experiences and real-world consequences to their use of the AI product. They interact with the full system, not only a single AI response. In this way, User Evals can add context to whether something appeared to fail, including where, how, and why a difficulty emerged in the user’s experience.


What happens when User Eval is skipped?


Two illustrative failure patterns help show what user-focused evaluation may add:


A feature may perform well on PM and DS evals, but still struggle in real-world use


Consider a scenario where the AI system is accurate, fast, and performs well across the PM and DS’s eval suite. Yet usage declines after the first few weeks.


The problem may not be response quality in isolation. For example, the AI may not fit naturally into the user’s workflow. Users may also need to provide substantial context or verify many details, which could contribute to lower retention. In such a case, the available model or system metrics may look healthy, while the product still does not become sufficiently useful in the user’s actual workflow.


The right AI answer may be delivered in the wrong way to the user


For example, an AI system could provide factually correct information, but communicate it with excessive confidence or unnecessary verbosity. A real user may then leave the interaction trusting the product less than before because of its tone. This type of damage can also compound across multiple user interactions, damaging the user’s trust with the system. Once the user stops trusting the system, they may ignore good outputs, double-check everything, or stop using the product entirely.


These kinds of problems may be difficult to understand from isolated response evaluation alone, particularly without observing how users interact with the system over time or in context.


UX Research can contribute to defining what “better” means to the user


AI product evaluation is sometimes described as a sequence such as:


  • First, build automated evals.

  • Then monitor production performance.

  • Afterwards, run A/B tests.

  • Finally, let users test the product and review the outputs.


This sequence can be useful, but it also raises an earlier question: before a team can evaluate whether an AI response has improved, it needs some basis for deciding what the system should be evaluated against.


For instance, what does a harmful failure look like in this particular domain for the user? Which errors are merely inconvenient, and which are trust-destroying? When do users want the system to act independently, and when do they want more control? What makes an answer feel useful rather than simply correct?


These can be technical and product questions, but they can also be empirical questions about how people interpret, use, and respond to AI systems in real-world contexts.


This is why I find the idea of UX research contributing at a kind of “stage zero” of AI evaluation useful: UXR can help surface user behaviors, expectations, and failure modes that may later inform what other parts of the evaluation system measure. I use “stage zero” here as a framing rather than as a formal or universally adopted stage of the eval process.


Through methods such as usability studies and user interviews, UX researchers can help identify dimensions of response quality that appear to matter to users in particular contexts. Depending on the product and context, these may include not only factual accuracy, but also dimensions such as clarity and an appropriate level of confidence for an AI response to fit within an existing user workflow.


In this sense, UX research may contribute not only by evaluating an AI experience after it has been built, but also by informing some of the human-centered criteria that an evaluation system might incorporate.


UX research may also contribute later in the product lifecycle. Once teams automate parts of the evaluation process (perhaps using classifiers or an LLM-as-a-judge), UX researchers can investigate whether those metrics still represent what matters to users. Without that kind of periodic validation, there could be a risk that a team could improve a metric without knowing whether it still corresponds closely to an experience users value.


How the three functions could work together


From this perspective, I think the goal is not to decide which function owns the “real” definition of evals. Rather, each function may be contributing evidence toward a somewhat different question:


  • Product management: Are we meeting the product’s success criteria, and are we ready to ship?

  • Data science: Is the model or system performing better according to measurable evidence?

  • UX research: Does the product help people accomplish meaningful goals in context?


One potentially useful approach is to connect these perspectives rather than treat them as separate evaluation efforts.


UX researchers can contribute evidence about user needs, failure patterns, and trust boundaries, which may help a team articulate what a better AI response could look like for particular users and contexts. Product managers can incorporate those findings into product priorities, requirements, and release decisions. Data scientists can then help operationalize selected criteria through datasets, experiments, and metrics.


The resulting evidence can then return to UX researchers, who can investigate whether the evaluated dimensions and metrics continue to align with users’ experiences. Those findings may, in turn, inform another round of product criteria or automated eval refinement. In practice, this could create a more continuous evaluation loop rather than a one-time test. From a UXR perspective, User Evals can therefore offer one way to examine whether “better” is still pointing toward outcomes users actually value.


For UXRs, I think this may be an especially useful area of contribution: not only assessing an experience after it has been built, but also bringing user evidence into discussions about the human-centered criteria by which AI products are designed, measured, and improved.


Notes:

  • This article is a high-level synthesis of selected industry and academic sources on AI evals, written from my current perspective as a UX researcher learning from experts in an evolving field. It is not intended to propose a definitive taxonomy, methodology, or division of responsibilities.

  • Since the field continues to develop and evolve, I believe terminology and practices can vary substantially across organizations and may continue to change.

  • I would be grateful for comments on additional perspectives or examples that add to the framing presented here.

  • This article was republished on August 9, 2026, after additional referenced sources were added and minor revisions were made for clarity.



If this is content you like to read, feel free to follow/subscribe to my Substack newsletter to receive the latest writing in time: Signals to Solutions


Sources Referenced


UX / User Evals — industry & academia:

Product Managements’ AI Evals:

Data Science’s Online vs. Offline Evals:


 
 
 

留言

這篇文章不開放留言。請連絡網站負責人了解更多。

© 2026 by Ying Chen. All Rights Reserved.

bottom of page