> For the complete documentation index, see [llms.txt](https://eval.playbook.org.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://eval.playbook.org.ai/social-impact/running-the-evaluation/common-pitfalls-to-avoid.md). # Common pitfalls to avoid IEs are high-investment undertakings, both financially and operationally, although strategies exist to address both financial and operational constraints. Avoiding a few predictable errors can significantly improve the value – and credibility – of your results. While these issues face many non-AI impact evaluations, here we have tried to capture the ways these risks are manifesting differently in early impact evaluations of AI products. {% hint style="info" icon="window-frame" %} #### Pursuing an IE too early They are most useful when your product is mature enough to test and when the decision stakes are high enough to justify the effort. In general, consider an IE when: * [x] **Levels 1–3 are strong**: The model performs well, users engage meaningfully, and early evidence suggests improvements in knowledge, attitudes, or behavior. * [x] **You are preparing to scale**: Funders or policymakers are considering wider adoption, and therefore evidence; cost-effectiveness or cost-benefit estimates, would be helpful to inform the decision. Conversely, scale-up plans may be in progress and present an opportunity for evidence gathering. * [x] **You have bandwidth**: Implementing an IE is a lot of work for both the research team and implementer; doing it well takes time and effort. You do not need to run an IE if your product is still in early design or usage is too inconsistent to expect impacts. In such cases, Level 3 evaluations—focused on user cognition and behavior—are more appropriate. Once you have confidence that the theory of change is working, you can and should revisit an impact evaluation. {% endhint %} {% hint style="info" icon="window-frame" %} #### Insufficient evaluability planning Although IEs are usually run later, credible and cost-effective evaluation requires early design choices. Building in features like holdout groups, staged rollouts, or embedded randomization from the start (also useful for A/B testing) preserves the ability to estimate causal effects without disruptive redesigns. Even if a full IE is premature, these choices create opportunities for credible inference later and reduce evaluation burden. Funders assessing scale readiness should look for signs of early evaluability. {% endhint %} {% hint style="info" icon="window-frame" %} #### Neglecting Independence and Transparency Rigorous IEs require expertise. We recommend working with an **independent evaluator**—such as an academic partner, a research or research-and-policy organization (e.g., J-PAL, IPA, or an LMIC-based evaluation firm), or a third-party M\&E firm (e.g., IDinsight, Laterite)—to strengthen technical quality and the perceived independence of the impact evaluation. Being clear on your evaluation goals (as discussed above) will help you choose among evaluator options. At a minimum, we suggest: * [x] **Clarifying roles**: who builds the product, who runs the study, who communicates findings * [x] **Pre-registering the design**: on platforms such as the AEA RCT Registry, EGAP, or RIDIE * [x] **Sharing results transparently**: Disclose all findings, including null or negative results, and make methods and materials publicly available where feasible to support reproducibility and sector-wide learning. {% endhint %} {% hint style="info" icon="keyboard-brightness-low" %} #### Being underpowered Even real impacts can go undetected in underpowered studies. For AI products, low uptake—especially early on—is a key risk. Overly optimistic uptake assumptions can leave treatment groups too small to detect effects. Set realistic expectations by piloting uptake with groups similar to the intended treatment population, involve skeptics in planning, and use recent Level 2 evaluations to inform assumptions. As discussed earlier, you track your target population and key sub-groups across all four evaluation stages. At Level 4, it is critical to have sufficient sample size to detect statistically significant, programmatically meaningful effects, including differences across groups. This challenge is not AI-specific but applies to any sub-group analysis; however, AI interventions may see groups participate in different ways and at different rates. Insights from Levels 1–3 should inform Level 4 sample design and outcome measurement. If budget allows, keep samples and outcomes broad enough to detect unintended positive or negative effects not flagged earlier. {% endhint %} {% hint style="info" icon="window-frame" %} #### Mismanaging transparency Impact evaluations should build confidence by involving credible, independent investigators, sharing data where appropriate, and pre-specifying key measures and analyses. But transparency should not come at the expense of adaptability. Researchers and implementers need to have an agreement on what they are evaluating, and if e.g. the intervention is to remain static or not and if not what data will be available to understand the dynamic nature of the intervention. Given how quickly AI interventions can change, establish mechanisms for early and ongoing coordination during implementation. {% endhint %} {% hint style="info" icon="mobile-screen" %} #### Letting product evolution obscure the analysis If the product may change during the study, pre-specify how changes will be handled analytically. One option is to freeze a version for the trial; if that is not feasible, define and log substantive changes, tag version exposure, and use this metadata to test for improvements or degradations over time. While this creates risk, it is also an opportunity for GenAI evaluations. Unlike analog interventions—where changes often went unobserved—version tracking, and even repeating Levels 1–3 evaluations after major updates, can enable much richer analysis during the impact evaluation period. {% endhint %} {% hint style="info" icon="square-exclamation" %} #### Underestimating the risks of attrition Attrition—through disengagement or loss to follow-up—can seriously weaken power and interpretability. In digital interventions, only a small share of sign-ups may engage, and drop-off is easy. Plan for this: track engagement early, power studies accordingly, and use passive data where possible. If attrition is unavoidable, pre-specify how it will be handled and report it transparently. Use Level 2 and 3 data to monitor attrition early, adjust design, and link it to version tracking to understand who drops out and when. {% endhint %} {% hint style="success" icon="star" %} **Note:** **This playbook can be used directly with AI tools like Claude, ChatGPT, and NotebookLM.** \ [***Explore how***** >**](/additional-resources/using-the-playbook-with-ai-tools.md) {% endhint %} ***
💬 Want to suggest edits or provide feedback? {% embed url="" %}