The Interpretability Gap: Why Auditors and End-Users Need Different Explanations

Introduction

The push for “Explainable AI” (XAI) has often been treated as a monolithic challenge—as if a single dashboard or technical report could satisfy everyone from the data scientist to the customer. However, this one-size-fits-all approach is a strategic failure. In high-stakes environments like banking, healthcare, and insurance, the needs of the individuals peering into the black box are fundamentally different.

Auditors are concerned with systemic fairness, regulatory compliance, and risk mitigation. They need to understand the global model—the broad logic that dictates how the system behaves across the entire population. Conversely, end-users are concerned with agency and fairness regarding their own outcomes. They require local, instance-specific justifications for why they were denied a loan, rejected for a claim, or flagged for a policy violation. Bridging this gap is not just a technical requirement; it is a prerequisite for earning public trust in automated decision-making.

Key Concepts

To understand why these groups have different requirements, we must first define the two tiers of interpretability.

Global Interpretability refers to the ability to explain the entirety of a machine learning model’s behavior. An auditor doesn’t care about a single transaction; they care about the distribution of decisions. They need to know: Does the model discriminate against protected groups? Is the model sensitive to noise? Does it adhere to legal requirements like the Fair Credit Reporting Act (FCRA)? Global explanations often involve feature importance rankings, sensitivity analysis, and partial dependence plots.

Local Interpretability focuses on a single “instance” or prediction. If a user is denied insurance, they don’t need a 50-page audit report on the model’s training data. They need to know the specific, actionable factors that led to that specific decision. Local explanations leverage techniques like SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) to highlight the specific variables—such as “insufficient credit history” or “high debt-to-income ratio”—that tipped the scale in that individual case.

Step-by-Step Guide: Designing an Interpretability Strategy

1. Segment Your Stakeholders: Identify exactly who needs to see the model output. Map each stakeholder to their primary concern (e.g., Compliance vs. Consumer Rights).
2. Implement Global Model Auditing: Utilize surrogate models or feature attribution methods to provide auditors with a holistic view of model biases and feature weights. This should be a static, recurring report.
3. Integrate Real-Time Local Explanations: Build an inference layer that generates explanations simultaneously with the model’s prediction. This allows your user-facing interface to pull “reason codes” directly from the decision engine.
4. Standardize the Vocabulary: Ensure that the terms used in an auditor’s global report map directly to the explanations provided to the end-user. If the auditor sees “Credit Utilization” as a risk factor, the end-user should not be told their rejection is due to “Financial Behavior.”
5. Establish a Feedback Loop: Create a mechanism for users to contest local decisions. When a user challenges a result, the system should allow human reviewers to look at both the local justification and the global policy to ensure consistency.

Examples and Real-World Applications

Case Study: Retail Banking Loan Decisions

In a loan approval system, the bank uses a complex ensemble model. The internal auditor uses a global explanation tool to ensure the model isn’t implicitly using zip codes as a proxy for race, a practice known as “redlining.” They analyze the model’s weightings across the entire applicant pool to satisfy federal fair lending audits.

Meanwhile, an end-user applies for a mortgage and is denied. The bank’s portal provides them with an instant, local explanation: “Your application was declined because your current credit card utilization is above 60%.” This is actionable—the user knows exactly what to fix to improve their chances next time. The auditor cares about the systemic rules of the model, while the applicant cares about the “why” behind their specific outcome.

Case Study: Healthcare Diagnostics

In a diagnostic AI system for radiology, the hospital compliance board (auditors) reviews global accuracy and false-positive rates across different demographic groups to ensure the tool is safe for widespread deployment. They need to see how the model generalizes across diverse patient populations. A patient (end-user), however, needs a local explanation: “This diagnosis was reached because of the shadow identified in the upper-right quadrant of the lung scan.”

Common Mistakes

• Dumping Data on Users: Providing a 100-page “explainability report” to an end-user is not transparency; it is obfuscation. Users need concise, plain-English reasons, not technical documentation.
• Ignoring Global Biases: Focusing entirely on local explanations can mask systemic, harmful biases that only become apparent when looking at the model’s global behavior.
• Inconsistent Messaging: If a local explanation implies a variable didn’t matter, but the global auditor report shows that same variable is a primary driver, the organization loses credibility and opens itself to litigation.
• Post-Hoc Rationalization: Attempting to invent reasons for a model’s decision after the fact, rather than using model-agnostic explanation tools that accurately map the decision path, leads to “hallucinated” explanations that do not reflect reality.

Advanced Tips

Adopt Contrastive Explanations: The most effective local justifications are often “contrastive.” Instead of telling a user why they were rejected, tell them: “You were rejected, but if you had paid off your personal loan, you would have been approved.” This provides significantly higher utility than a static list of factors.

Use Surrogate Models for Global Oversight: For incredibly deep, opaque neural networks, use a “surrogate model”—a simpler, interpretable decision tree that mimics the behavior of the black-box model. Auditors can use this surrogate to grasp the model’s global logic without needing to understand the millions of weights in the primary architecture.

Build Human-in-the-Loop Validation: Technology cannot solve the “right to explanation” entirely. Ensure that local explanations trigger a secondary human review when a user flags them as incorrect. This creates a data-rich environment for the next iteration of the model’s global training.

Conclusion

The distinction between global and local explanations is the dividing line between a system that is merely functional and a system that is fundamentally trustworthy. Auditors and regulators need to look at the forest to ensure the model isn’t growing in ways that violate ethics or the law. End-users need to look at the tree—their specific tree—to understand their situation and maintain agency over their lives.

By implementing these two layers of interpretability, organizations can satisfy the rigid demands of compliance while simultaneously providing the clear, actionable insights that customers demand. Moving forward, the goal should not be to build a “more explainable model,” but rather to build a tiered communication strategy that speaks to the specific needs of every individual interacting with your AI systems.