Do I need a Process Evaluation?
While PEs can help diagnose bottlenecks and inform refinements, conducting a PEs for each levels 2-4 is not a requirement to advance your program. However, here are some situations under which a PE would be especially valuable:
When the user is not the beneficiary. Levels 1-3 focuses on the user. Tools targeting frontline workers as users - midwives, teachers, extension agents - typically aim to benefit someone else downstream: a pregnant woman, student, farmer, or patient. If the beneficiary does not receive, understand, or act on the information, impact will not follow. Therefore, you may want to construct funnels for each stakeholder in the delivery of your intervention, program, or social service.
When the product influences actors outside the program to behave in a way that can impact outcomes. AI products can shift the work of people who never touch your product, intervention, or program. For example, teachers may shift from lecturing to coaching when students begin using a tutoring app that delivers core instruction– or they may become less motivated to teach a topic if they believe the app has “taken over,” resulting in reduced teacher effort or preparation). If routines, responsibilities, or incentives shift, these changes can be understood, supported, and/or mitigated via a PE.
When the product is implemented in relation to other programmatic systems already in place. Digital products rarely operate alone: they must either fit into existing systems (e.g., data or reporting systems) or transform them. For example, if an AI product flags patients that should follow-up with a provider, this information should flow into existing health record systems and be used by the providers who see them. If data cannot be synced, matched to the right person, or acted on, outcomes do not improve, even when the tool itself is used correctly. A PE supports investigating these factors.
If a program moves to a Level 4 trial without incorporating a process evaluation it risks finding null or even negative impacts for an otherwise promising product. Without insight into implementation, it becomes difficult to determine whether the results reflect a failure of the product itself or weaknesses in how it was delivered and integrated into the broader program system.
Last updated
Was this helpful?