Who is most involved in this level of evaluation?

Execute	Support
Product Managers	Data Scientists
Directly responsible for product metrics at this level. Works cross-functionally to prioritize the most promising hypotheses to test.	Apply evaluation methods with the proper measurement tools. Ensure accuracy and availability of product metrics (data pipelines).

---

Key Motivation

Technical performance (Level 1) does not guarantee user adoption. Level 2 evaluation is critical because:

Value Validation

If users stop interacting, they likely see no value, meaning the intervention cannot achieve its intended life outcomes.

Continuous Improvement

It transforms product development from opinion-driven to data-driven through iterative cycles and A/B testing.

Safety & Risk Management

Monitoring user signals allows for controlled rollouts of experimental features, preventing negative reactions from reaching your entire user base at once.

---

Why is this level of evaluation important?

An AI system that produces perfect responses is worthless if users do not use it. Once you deploy your AI system as a product (e.g., a chatbot or app), you must track a few critical user signals, like:

Engagement

How many users are using the product?

Retention

How likely are they to continue using it?

If users never engage—or stop interacting because they see no value—they are unlikely to change their behavior in ways that improve their life outcomes.

Like AI system evals, Level 2 evaluation is a continuous, iterative cycle, not a one-time exercise. We track user interaction metrics over time, and look for unexpected drop-off or intended improvements, for example when a promising new feature is released as part of an A/B test. Product evaluations are critical for iterative improvement, but they can also be a matter of safety. Suppose you have an experimental new feature in your chatbot – but you’re not sure how people will react. It might be risky to roll out this new feature to all users, all at once.

Note: This playbook can be used directly with AI tools like Claude, ChatGPT, and NotebookLM. Explore how >

---

💬 Want to suggest edits or provide feedback?