What is User Evaluation

Does the product change users' thoughts, feelings, knowledge and behaviour towards the development outcome?

Once an AI system is reliable (Level 1) and engaging (Level 2), we must ask the deeper question: Is it actually working? In the development sector, "liking" a product is not a proxy for impact. Level 3 evaluates the "stepping stones" of change—the intermediate cognitive and affective shifts that predict long-term life improvements in health, education, or livelihoods.

Note: The levels are not strictly sequential. Teams may run Levels 1, 2, and 3 in parallel, and revisit earlier levels as new results emerge

---

Core Concept: Intermediate Outcomes

Level 3 measures how an "adequate dosage" of your AI product shifts the user across several dimensions. We look for changes in the following constructs:

Cognitive outcomes: Are users learning?

Are they gaining new knowledge or updating beliefs? Do they demonstrate improved skills or decision-making ability as a result of engaging with the product?

• Constructs to measure: comprehension, knowledge acquisition and retention, belief updating, critical evaluation of information, metacognitive awareness (e.g., accurate calibration of what one does and does not know), perceived clarity, complexity of reasoning during/following interaction.

Affective outcomes: How does the product make users feel?

Do users report feeling supported, motivated, and capable after interactions, or are there indications of confusion, anger, or emotional distress?

• Constructs to measure: mood, emotional valence and arousal, frustration, confusion, emotional granularity, felt support, sense of safety, sense of belonging, perceived empathy, trust, or comfort interacting with AI.

Behavioural outcomes: Is the user doing something different?

Are users taking small but meaningful actions that predict longer-term development?

• Constructs to measure: application of new information, intent to try recommended behaviors, observable shifts in interaction patterns(e.g., asking more complex questions, prompt sophistication) that proxy for longer-term development outcomes, help-seeking behavior.

Motivational outcomes: Is the user motivated to change?

Does the product energize or deplete a user’s drive to pursue goals, learn, or act independently?

• Constructs to measure: intrinsic motivation, curiosity, self-efficacy, perceived autonomy, goal commitment, persistence (not the same as perseveration), dependency (i.e., reduced willingness to attempt tasks without AI assistance).

Social and relational outcomes: Is the user connecting with people?

Does use of the product affect the user’s human-to-human relationships and broader social functioning?

• Constructs to measure: social displacement (substitution of AI for human interaction), loneliness, perceived social support, quality of interpersonal communication, willingness to engage with others, trust in human versus AI sources of social support and information.

Well-being outcomes: Is the user flourishing?

Does using the product provide broader and more distal effects on users’ overall quality of life and psychological health beyond momentary feelings/mood?

• Constructs to measure: life satisfaction, meaning and purpose in life, flourishing, burnout (i.e., in professional contexts), perceived agency/control over one’s environment.

---

Individual vs. System Outcomes

You may have noticed that in Level 3, we focus on individual outcomes, rather than the broader community or system. This is a practical choice, not a philosophical one. While changing social norms (e.g., how a village views vaccination) is often the ultimate goal of development, measuring those shifts requires slower, more extensive fieldwork (as discussed in Level 4).

In contrast, individual changes can be immediate. You can observe if a user is learning or motivated right now, even if the full impact of an AI product requires changes in social dynamics. This makes individual metrics the fastest, most sensitive signals for rapid product iteration.

Our recommendation: Do not ignore social dynamics, but do not let them slow down your experimentation. Measure what you can see today (i.e., the user’s behavior) and until you are ready for a full impact evaluation, use lightweight proxies to track social effects (e.g., asking "Did you share this advice with a neighbor?"). In-app questions about off-app behaviour can complement information about user behaviour.

Further reading

Level 3 vs. Traditional User Research

User research plays a critical role across the product lifecycle (e.g., Discover, Explore, Test, and Listen). However, in Level 3 we will focus on quantitative user research that captures intermediate outcomes at scale. These outcomes are sometimes observed in the product logs described in Level 2, but more often captured in surveys or automated analysis of user text, voice recordings, and other digital traces. At Level 3, our goal is to cheaply track psychological “states” and “traits” (e.g., cognitive, affective, and behavioral) across a large sample of users, for use in product monitoring and rapid-cycle experiments. We want to understand user shifts in psychology or behavior, but without needing to conduct bespoke qualitative research every time a new feature is being tested (even though at least one such qualitative evaluation should be conducted for the overall GenAI solution). Ideally, qualitative user research contributes to the theory of change and informs every stage of the evaluation framework:

Level 1 (AI system)

Interviews help define the "Golden Dataset" by gathering realistic user questions, identifying edge cases, and defining "ideal" answers based on needs expressed by real users.

Level 2 (Product)

Interviews and direct observation can contextualize engagement data. If an A/B test reveals a drop in retention, qualitative research helps diagnose the underlying friction or confusion.

Level 3 (User)

Interviews or focus groups can validate that intermediate outcome metrics (e.g., user-reported confidence) actually correlate with real-world behavior changes.

Note: This playbook can be used directly with AI tools like Claude, ChatGPT, and NotebookLM. Explore how >

---

💬 Want to suggest edits or provide feedback?