Documenting Model Feature Interactions: A Regulatory Compliance Guide

Introduction

In the evolving landscape of artificial intelligence and machine learning, the “black box” is no longer acceptable for high-stakes decision-making. Whether you are operating in fintech, healthcare, or insurance, regulatory bodies like the Federal Reserve (SR 11-7), the GDPR, and various state-level AI accountability acts are demanding more than just predictive accuracy. They are demanding transparency, accountability, and—most crucially—the documented reasonableness of how your model’s features interact.

When a model predicts a loan default or a health risk, regulators want to know: Are these variables interacting in a way that is logically defensible, or is the model picking up on noise and unintended biases? Providing documentation on feature interactions is the bridge between a high-performing model and a compliant, auditable asset.

Key Concepts: What is a Feature Interaction?

A feature interaction occurs when the effect of one input variable on the model’s outcome depends on the value of another input variable. Mathematically, it is the non-linearity that allows a model to go beyond simple additive effects.

For example, if you are modeling credit risk, “Annual Income” and “Debt-to-Income Ratio” are two separate features. An interaction effect exists if the impact of a high debt-to-income ratio on the probability of default is significantly worse for an individual with a low annual income than it is for someone with a high annual income.

Reasonableness, in this context, refers to the alignment of these interactions with domain expertise and economic reality. Regulators are not looking for complex math alone; they are looking for evidence that the relationships the model has learned mirror the physical or business realities they are intended to represent. If a model suggests that “Number of Children” has a massive, non-linear effect on interest rates that defies traditional financial logic, that is an unreasonable interaction.

Step-by-Step Guide to Documenting Interactions

Compliance is a proactive process. Follow these steps to build a robust documentation trail that satisfies auditors.

1. Identify High-Risk Interactions: Use sensitivity analysis to identify which pairs or groups of features drive the most variance in your model’s output. Focus your documentation efforts on these interactions first.
2. Select Interpretation Tools: Utilize industry-standard post-hoc interpretability methods. Partial Dependence Plots (PDPs), Individual Conditional Expectation (ICE) plots, and SHAP (SHapley Additive exPlanations) interaction plots are your primary tools for visualizing these relationships.
3. Define the Hypothesized Logic: Before examining the output, document what the relationship should look like based on subject matter experts (SMEs). This creates a “benchmark for reasonableness.”
4. Quantify and Visualize: Generate plots for the identified interactions. Ensure these visualizations are labeled clearly, showing the change in model prediction as the two features vary across their distributions.
5. Draft the Justification: Write a concise narrative linking the visual evidence to the business or scientific logic. Use phrases like, “The model demonstrates a higher sensitivity to X when Y is present, which aligns with our internal policy regarding [Risk Factor].”
6. Review for Bias: Check if any interaction disproportionately affects protected classes (e.g., race, gender, age). Documenting the lack of unreasonable disparate impact is just as important as justifying the business logic.

Examples and Case Studies

Fintech: Mortgage Underwriting

A mortgage lender uses a gradient-boosted tree model. During the model validation phase, the team discovers an interaction between “Zip Code” and “Employment Duration.” The model heavily penalized short employment durations in specific, predominantly minority neighborhoods, but not in others. Upon review, the team determined that the model was essentially using “Zip Code” as a proxy for socioeconomic status, leading to an unreasonable interaction that violated Fair Lending standards. By documenting this “reasonableness test,” the bank was able to justify removing the interaction and retraining the model to prioritize stable employment indicators regardless of geography.

Healthcare: Predictive Diagnostics

A hospital system uses a model to predict patient readmission. The model shows an interaction between “Age” and “Comorbidity Index.” The documentation identifies that as age increases, the weight of the Comorbidity Index on the risk score increases exponentially. The clinical team reviews this and confirms it matches medical consensus: older patients are indeed more vulnerable to complications from multiple, compounding illnesses. The documentation provides a clear, defensible “reasonableness” case for regulators, citing specific clinical literature that supports the model’s learned behavior.

Common Mistakes to Avoid

• Ignoring “Proxy” Interactions: Many teams look at features in isolation and miss how they create a biased interaction when combined. A feature might be benign on its own but create a discriminatory interaction effect.
• Over-relying on Global Feature Importance: Global importance rankings hide interactions. If you tell an auditor that “Feature A is important,” but you can’t explain how it interacts with “Feature B” to drive results, you have failed the compliance test.
• Lack of SME Involvement: Documentation written by data scientists without the input of business domain experts is often rejected. The “logic” must be understandable to a non-technical regulator.
• Poor Visualization Standards: Submitting raw, messy charts without annotations is a major red flag. Always clean your plots and annotate the key trends that you want the regulator to notice.

Advanced Tips for Compliance Success

Use Monotonicity Constraints: If you know for a fact that a feature should have a positive relationship with the outcome (e.g., higher savings balance should always lower risk), enforce this constraint in the model. This makes the model inherently more “reasonable” and easier to defend during audits.

Leverage Interaction Graphs: Beyond standard SHAP values, look into H-statistic measurements. The H-statistic allows you to quantify the strength of an interaction on a scale of 0 to 1. Using this metric allows you to objectively demonstrate to regulators, “We have tested for all interactions, and only these three variables show significant coupling.”

Maintain a Versioned “Model Registry”: Documentation is not a one-time event. Every time you update the model, you must update the interaction report. A versioned repository that links the model code, the data snapshot, and the reasonableness report is the gold standard for audit readiness.

Conclusion

Regulatory requirements for the “reasonableness” of feature interactions are not intended to stifle innovation; they are designed to ensure that the logic driving our automated systems is grounded in reality. By proactively documenting how your model’s features interact, you protect your organization from legal risk, improve the reliability of your predictions, and build trust with stakeholders.

Start by identifying your most sensitive interactions, use robust visualization tools to map those relationships, and always ground your documentation in the expertise of those who understand the business best. In an era of increasing AI scrutiny, the ability to explain why your model behaves the way it does is a competitive advantage that goes far beyond simple compliance.