Outline

• Introduction: Defining the “Asymmetry Problem” in technology and service evaluation.
• Key Concepts: Understanding Institutional Power vs. User Agency and the “Evaluation Gap.”
• Step-by-Step Guide: Implementing power-aware evaluation frameworks.
• Examples: Case studies in Algorithmic Management and EdTech.
• Common Mistakes: The pitfalls of “User-Centricity” that ignore systemic power.
• Advanced Tips: Incorporating participatory design and adversarial testing.
• Conclusion: Shifting from “user satisfaction” to “power equity.”

Bridging the Gap: Why Evaluation Frameworks Must Account for Power Dynamics

Introduction

Most modern evaluation frameworks are built on a flawed assumption: that the provider and the user engage on equal footing. Whether it is an enterprise software platform, an algorithmic hiring tool, or a public health app, traditional metrics—such as Net Promoter Score (NPS), task completion rates, and churn—often focus on the experience of the user while ignoring the position of the user.

When a system provider holds significant institutional, legal, or financial power over the end-user, the user’s “choice” to adopt or use the system is often coerced by circumstance. If your evaluation framework ignores this dynamic, your data is not just incomplete—it is fundamentally deceptive. To build truly ethical and functional systems, we must evolve our frameworks to account for the structural power imbalance inherent in the provider-user relationship.

Key Concepts

To understand why power dynamics matter, we must distinguish between User Agency and Systemic Pressure. In a market-based relationship, users can easily opt out. In an asymmetric relationship, users face high switching costs or, worse, existential consequences for non-compliance.

The Asymmetry Gap: This occurs when the provider defines the metrics of “success” in a way that minimizes their liability while maximizing user dependency. For example, an employer-mandated productivity tracker might report “high engagement” because employees are forced to use it to keep their jobs. The evaluation framework registers this as success, whereas the reality is a captured audience.

Power-Aware Evaluation: This is a methodology that treats power dynamics as a primary variable. It asks not just “Does this feature work?” but “Does this feature expand or contract the user’s autonomy?” It shifts the focus from efficiency metrics to equity and accountability metrics.

Step-by-Step Guide: Implementing a Power-Aware Framework

Integrating power dynamics into your evaluation strategy requires a shift from passive observation to active inquiry. Follow these steps to audit your current framework:

1. Map the Power Gradient: Identify the dependency level. Is the user using your system because they want to, or because they are required to by an employer, government entity, or lack of alternatives? Map the potential “harm” a user incurs by leaving or complaining.
2. Diversify Your Feedback Channels: Standard surveys favor those who are comfortable with the system’s design. Implement “negative feedback” protocols. Actively solicit accounts of users who have been frustrated or harmed by the system, specifically targeting those with the least institutional influence.
3. Audit Success Metrics for Coercion Bias: Review your primary KPIs. If a metric improves when the user has fewer options, that metric is a measure of dependency, not quality. Replace these with metrics that measure user agency, such as the ease of data portability or the ability to opt-out of specific system features without penalty.
4. Establish a Redress Mechanism: An evaluation framework is not just about measuring; it is about responding. Create a clear path for users to contest automated decisions or system constraints. The existence of this path itself is a powerful evaluation metric.
5. Implement Longitudinal Impact Studies: Evaluate how the system changes the user’s life or work over time. Does the system reduce the user’s workload, or does it lead to “scope creep” where the user is now managing the system rather than their own tasks?

Examples and Case Studies

The Algorithmic Management Case: Consider a delivery driver application. The provider evaluates the system based on “efficiency” and “delivery speed.” However, a power-aware framework would look at the “autonomy score.” If the driver is penalized for rejecting routes in an area where they feel unsafe, the system is exerting coercive power. An equitable framework would mandate that safety preferences outweigh algorithmic efficiency.

“True system quality cannot be measured by speed alone if that speed is gained at the expense of human agency.”

The EdTech Context: Schools often implement learning management systems (LMS) that track student activity. A standard framework might praise the “granularity of data” available to teachers. A power-aware framework, however, would evaluate whether that granular tracking creates a “panopticon effect,” where students feel they are being watched, ultimately stifling creative risk-taking. Success here should be measured by student comfort and engagement, not just the volume of data generated.

Common Mistakes

• Confusing Compliance with Consent: Just because a user clicked “Accept” on a Terms of Service agreement does not mean they entered the relationship voluntarily. Assuming that a signed contract equals valid consent is a common error that masks systemic coercion.
• Prioritizing the “Average” User: Evaluation frameworks often optimize for the median experience. However, the most critical power dynamics often reveal themselves at the extremes. If a system works perfectly for 90% of users but traps the bottom 10% in a cycle of exclusion, your framework is failing to capture the systemic harm.
• Ignoring the “Cost of Exit”: Many frameworks fail to account for how difficult it is for a user to stop using a service. If the “exit cost” is high, users will under-report dissatisfaction for fear of being “locked out” or blacklisted.

Advanced Tips

To move toward a truly mature evaluation model, consider Adversarial User Testing. Instead of testing for usability (how easy is it to use?), test for subversibility. How hard does the system make it for a user to protest, complain, or bypass unfair constraints? If the system is designed to be impossible to complain about, it is fundamentally anti-user.

Furthermore, incorporate Equity Impact Assessments. Before deploying a new feature, run a simulation: “How would this feature affect a user who has no alternative options?” If the feature forces this user into a disadvantageous position, the framework must trigger a mandatory review by an ethics committee or a diverse stakeholder panel, rather than just a technical product team.

Conclusion

We are long past the era where “user-centric design” is enough. A system that is easy to use but creates a toxic power dynamic is a failed system. To build better technology and services, we must acknowledge the inherent asymmetry between the provider and the user.

By shifting our evaluation frameworks to prioritize user agency, transparency, and accessible redress, we move from a model of manipulation to one of partnership. The goal of evaluation should not be to validate the system’s effectiveness for the provider, but to ensure that the system respects the humanity and autonomy of the person on the other side of the screen. When we account for power, we don’t just build better systems—we build more sustainable, ethical, and trustworthy digital ecosystems.