# Define guardrail metrics and measure potential harm As you reach Level 3 evaluations, you are not just measuring if your product is working; you want to measure if it is causing harm. While Level 2 metrics track usage, Level 3 is your opportunity to use direct interviews and surveys to track unintended consequences. A central concern is that AI models and agents can empower or disempower users. In evaluating social impact, user agency can be a critical guardrail for AI products. There is a risk that "helpful" AI agents might actually undermine development, creating dependency for users or communities rather than building capabilities. Therefore, you will want to track whether your tool is improving or reducing users’ agency. For instance, we recommend measuring agency in two ways:
Cover image
Subjective Agency (Internal-Facing)This captures users’ beliefs and perceptions of their own capabilities (e.g. Albert Bandura’s Social Cognitive Theory), measured through qualitative or survey methods. Ask users about their sense of self-efficacy: do they believe they can solve the problem on their own now?https://images.unsplash.com/photo-1515463626042-123ab67dcaa7?crop=entropy&cs=srgb&fm=jpg&ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHw0fHxSZWZsZWN0aW9ufGVufDB8fHx8MTc3MzIzNzE5OXww&ixlib=rb-4.1.0&q=85
Objective Agency (External-Facing)These are the capabilities required to plan, navigate, execute, and reflect on personal goals (e.g. Amartya Sen’s Capability Approach). Does the user have the skills to act? Test their ability to plan and execute goals without the AI's help. Are they learning the underlying logic, or just copy-pasting answers?https://images.unsplash.com/photo-1603804449564-2ad32f24d17e?crop=entropy&cs=srgb&fm=jpg&ixid=M3wxOTcwMjR8MHwxfHNlYXJjaHwyfHxwbGFufGVufDB8fHx8MTc3MzE1MTc1NXww&ixlib=rb-4.1.0&q=85
***
💬 Want to suggest edits or provide feedback? {% embed url="" %}