What does it take to do a process evaluation?
Process evaluations systematically and empirically describe what happens during program implementation, then carefully compare this description to ex-ante expectations in the implementation plan and theory of change. In brief, it proceeds along the following lines:
Program theory of change: Construct user funnels for all stakeholders (human actors) that interact with the AI product and influence development outcomes. Explicitly articulate what each actor must do differently for the AI intervention to work. Make sure to surface the assumptions that link each step: what needs to be true for inputs to lead to activities, for activities to generate outputs, and for outputs to translate into outcomes. These assumptions are often where the chain breaks.
Adopt an implementation framework: Established Implementation Science (IS) frameworks such as CFIR (Consolidated Framework for Implementation Research) (Damschroder et al., 2009) offer pre-specified domains for systematically identifying the factors that shape whether an intervention takes hold in a real-world setting. These and similar frameworks have been applied extensively to digital interventions (e.g. Greenhalgh et al. 2017). PEs of AI-enabled tools can draw on this body of work to systematically assess how characteristics of the broader intervention, the implementing organization (e.g., leadership engagement, infrastructure), and the external environment (e.g., policy, incentives) shape real-world implementation dynamics.
Use mixed methods: Combine routine and administrative data, structured process indicators, document review, qualitative (interviews, observations, focus groups) and quantitative methods (representative surveys), to interrogate weak links in the theory of change.
Iterate on programme design (beyond model or product): Treat findings as input into redesigning training, supervision, accountability structures, and integration with existing systems.
Stage-gate the next level of evaluation: Make satisfactory implementation improvements an explicit precondition for proceeding to the next level of evaluation.
Last updated
Was this helpful?