# Introduction {% hint style="info" %} This playbook is a living playbook. We’ll keep updating this playbook and collaborating more deeply with specialists to co-create shared evaluation tools, refine methodologies, and support their practical use in real-world settings. {% endhint %} The use of generative AI (GenAI) tools in low- and middle-income countries is multiplying – from AI-powered math tutors for children to digital advisory tools for farmers. While studies have shown that AI-powered applications can improve human and economic development outcomes (e.g., [Henkel et al., 2024](https://arxiv.org/abs/2402.09809)), others warn of harmful uses (e.g., [Bastani et al., 2024](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486)). The common thread is that outcomes depend critically on how AI is designed and used. **Evaluations can help developers build more impactful AI tools,** **yet there has been little agreement on what evaluation actually means.** Organizations often find themselves caught between two extremes of evaluation: tech teams prioritize product performance, often overlooking impact, while impact evaluators focus on outcomes but may neglect the underlying technology. **These foci are often pursued in isolation, even though they are inherently complementary.** This playbook offers a unified framework to bridge the gap, laying out how to evaluate GenAI services in the development sector: what evaluation should include, and the standard practices implementers should follow. It is organized around a four-level framework:
Cover image
:gear-code:

Level 1 - Model evaluation

Does the AI system perform as intended?level-1-model-evaluation
:box-isometric:

Level 2 - Product evaluation

Does the overall product engage and retain users?overview
:user:

Level 3 - User evaluation

Does the product change users’ thoughts, feelings, knowledge, and behavior towards the development outcome?overview
:chart-column:

Level 4 - Impact evaluation

Do users with access to the product improve development outcomes?overview
Even if the boundaries may sometimes be blurry in practice, the four levels form a logical progression. Users are unlikely to stay engaged (Level 2) if the GenAI system fails to perform (Level 1), and development outcomes are unlikely to improve (Level 4) if users disengage or their feelings, knowledge, and behaviors are harmed (Level 3). The central element of this framework is continuous evaluation. Unlike earlier rule-based tools, GenAI performance is highly sensitive to underlying models, training data, prompts, context, configuration, and other parameters. This complexity demands new evaluation methods. Moreover, AI’s core inputs evolve far faster than earlier technologies. Amidst this continuously evolving technology, continuous evaluation enables rapid iteration, maintains expected behavior, and improves performance and impact over time.
***
💬 Want to suggest edits or provide feedback? {% embed url="" %}