> For the complete documentation index, see [llms.txt](https://eval.playbook.org.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://eval.playbook.org.ai/product-analytics/overview/overview.md). # What is Product Evaluation Does the overall product engage and retain users? An AI system that produces perfect responses is worthless if no one uses it. Level 2 Evaluation moves beyond technical accuracy to measure the "digital traces" users leave behind. By tracking how users move from their first interaction to long-term habit formation, we can ensure the product actually delivers value in the real world. {% hint style="warning" icon="circle-info" %} **Note:** The levels are not strictly sequential. Teams may run Levels 1, 2, and 3 in parallel, and revisit earlier levels as new results emerge {% endhint %} *** #### Core Concept: The User Funnel To evaluate the product, we "instrument" the application to automatically track specific user actions, as they progress through four distinct stages. The resulting log data allow you to track users as they move through the User Funnel: from their first interaction (Activation), to regular usage (Engagement) to long-term commitment or habit formation (Retention). We prioritize "Time to Success" (solving the user's problem) over "Time on Device" to ensure we are optimizing for welfare rather than just addiction. In the tech sector, companies might track "clicks" and "purchases" as users move through a website. By analyzing logs, you can then identify which content or products are likely to bring users back to the website over time, or how different web experiences affect browsing time. In the development sector, we need to track actions that signal user intent, and estimate the value returned to users in response. * For an AI Agronomist, instead of tracking page views you might track whether a farmer uploads a photo of a diseased crop, listens to audio advice to completion, or shares content with another person. * For an AI Tutor, you might track if a student completes a quiz, how many follow-up questions they ask in a single session, or if they return to the app the night before an exam. By analyzing these "digital traces," we can identify exactly where users lose interest. Does the farmer drop off because the photo upload takes too long? Does the student quit because the AI’s first response was too complex? Below are the four stages of the user funnel with goals and metrics:

Stage	Goal	Key Metric Example
Acquisition	Bring users into the ecosystem.	New User Count, Cost Per User (CAC)
Activation	Ensure users find "First Value."	Activation Rate, Time to Activate
Engagement	Measure depth and frequency of use.	Active Users (DAU/WAU), Interaction Depth
Retention	Build long-term habits/commitment.	Stickiness (DAU/MAU), Retention Rate

{% hint style="success" icon="star" %} **Note:** **This playbook can be used directly with AI tools like Claude, ChatGPT, and NotebookLM.** \ [***Explore how***** >**](/additional-resources/using-the-playbook-with-ai-tools.md) {% endhint %} ***

💬 Want to suggest edits or provide feedback?

{% embed url="" %}