Navigating the “Reasonableness” of Model Feature Interactions in Regulatory Compliance

Introduction

In the era of black-box AI, the days of relying solely on predictive accuracy are over. For financial services, healthcare, and insurance sectors, the ability to build a high-performing model is no longer the primary hurdle; the challenge is explaining it to regulators. Increasingly, global oversight bodies—from the Consumer Financial Protection Bureau (CFPB) to EU AI Act auditors—require documentation of the “reasonableness” of model feature interactions.

But what does “reasonableness” actually mean in a mathematical context? It implies that the way your features influence one another must align with economic logic, clinical reality, or domain-specific expertise. If your model predicts that an applicant’s credit risk decreases as their debt-to-income ratio rises, you have a “reasonableness” problem. This article breaks down how to document, validate, and defend your feature interactions to meet modern regulatory expectations.

Key Concepts

To provide documentation that stands up to an audit, you must move beyond simple linear correlations. Feature interactions occur when the effect of one input variable on the target output depends on the value of another variable. For example, the impact of “length of credit history” on a loan default probability may vary significantly depending on the “total debt level.”

Reasonableness is the bridge between statistical significance and domain intuition. It is the assertion that a machine learning model’s learned logic does not violate causality or common sense. If a model detects a non-linear interaction, the documentation must explain why that relationship exists. Without this, regulators perceive your model as an unstable “black box” prone to discriminatory outcomes or erratic behavior in edge cases.

Step-by-Step Guide to Documenting Interactions

1. Initial Interaction Discovery: Utilize global and local interpretability tools. Use Partial Dependence Plots (PDPs) for single features and H-statistic or ICE (Individual Conditional Expectation) plots to identify where features are interacting.
2. Domain Expert Consultation: Present the identified interactions to subject matter experts (SMEs). Do not assume the model is correct because the training data is “clean.” Ask the SME: “Does it make sense that A and B interact in this specific way?”
3. Hypothesis Formulation: Draft a “Reasonableness Statement” for every significant interaction discovered. This should be a 2–3 sentence explanation grounded in your industry. (e.g., “The interaction between Age and Annual Income is observed because younger individuals with high incomes often carry higher student debt loads, affecting default probability.”)
4. Stress Testing and Sensitivity Analysis: Conduct “what-if” scenarios. Systematically perturb the values of interacting features to ensure the model output moves in a direction that remains economically or medically plausible.
5. Final Regulatory Narrative: Compile the findings into a Model Risk Management (MRM) document. Include the statistical plots, the SME validation sign-off, and the sensitivity testing results to demonstrate that the model behaves predictably under various input combinations.

Examples and Case Studies

Example 1: Mortgage Lending

A bank uses a gradient-boosted tree model to predict default. The model identifies a strong interaction: “Employment Duration” has a massive impact on the “Loan Amount” feature. The documentation explains: “In our portfolio, lower employment duration is penalized more heavily for high-value loans because these borrowers lack a long-term track record of stable cash flow to cover larger monthly commitments.” This is a documented, reasonable interaction.

Example 2: Healthcare Diagnostic Tool

A model predicts the likelihood of complications in surgery. It finds an interaction between “Age” and “Pre-existing Diabetes.” The documentation states: “The model assigns a higher risk multiplier to patients over 65 who have diabetes compared to younger patients with the same condition, reflecting clinical evidence that recovery resilience decreases significantly after age 65.”

Common Mistakes

• Ignoring “Proxy” Interactions: Often, an interaction is statistically reasonable but socially biased. If your interaction captures a proxy for a protected class (e.g., zip code interacting with income), it is not just unreasonable—it is a compliance violation.
• Over-reliance on Global Explanations: Global methods like SHAP summary plots are great, but they often average out interactions. Always supplement them with local analysis (e.g., individual case studies) to show you understand how features interact at the extremes of your dataset.
• Failure to Update Documentation: Models drift. If you retrain a model on new data, the interactions may change. A common failure is keeping the original “Reasonableness” documentation while the underlying logic of the model has shifted.
• Disregarding “Broken” Interactions: Sometimes, interactions occur due to data leakage or missing values being encoded in a way the model treats as a feature. If an interaction is purely a result of noise or data error, do not justify it—fix the data.

Advanced Tips

To elevate your documentation, leverage Monotonicity Constraints. If your industry requires that higher credit scores should always lead to lower interest rates, implement constraints in your model training. By forcing the model to respect monotonicity, you reduce the likelihood of “unreasonable” interactions appearing in the first place.

Furthermore, consider implementing Interaction Strength Metrics. Use techniques like Friedman’s H-statistic to quantify exactly how much of the model’s prediction variance is driven by interactions versus main effects. Presenting these metrics to regulators shows a sophisticated level of model governance that goes beyond the “black box” standard. Finally, always document the “Negative Evidence”—explicitly state interactions that you checked for but did not find, as this demonstrates a thorough audit process.

Conclusion

The requirement for documenting feature interactions is a call for transparency. Regulators are not asking you to abandon complex machine learning; they are asking you to be the master of it. By systematically identifying, validating, and explaining how your features influence one another, you transform a mysterious predictive engine into a robust, defensible, and compliant business asset.

Documenting reasonableness is not just a regulatory checklist item—it is an opportunity to improve the quality and reliability of your models. Start by mapping your interactions, validating them with domain experts, and grounding your documentation in the fundamental logic of your industry. When you can explain the why behind the how, you protect your organization from both regulatory scrutiny and model failure.