The Critical Role of Baseline Values in Reproducible SHAP Audits

Introduction

In the high-stakes world of machine learning deployment, model interpretability is no longer a luxury—it is a regulatory and ethical requirement. SHAP (SHapley Additive exPlanations) has emerged as the industry standard for explaining complex model predictions. By assigning an importance value to each feature for a given outcome, it bridges the gap between opaque “black-box” algorithms and human-understandable logic.

However, many data science teams treat SHAP as a “black-box interpretability tool” without realizing that their explanations are inherently relative. SHAP values are not absolute truths; they are measurements of a feature’s contribution relative to a baseline (or reference) value. If you do not record and version-control your baseline strategy, your audit results become a moving target. This article explores why baseline consistency is the linchpin of reproducibility and how you can institutionalize this practice to satisfy auditors and stakeholders alike.

Key Concepts: The Mechanics of the Baseline

At its core, SHAP is built on game theory. It imagines each feature as a player in a game where the “payout” is the model’s prediction. The SHAP value represents the average marginal contribution of a feature across all possible combinations of features. To calculate this, the algorithm needs a reference point to compare against.

The baseline represents the model’s output in the absence of specific feature information. It is effectively the “neutral” or “default” state of your model. Without a baseline, the SHAP algorithm has no anchor, and the resulting feature importance values will fluctuate wildly based on which data points are chosen for the background dataset.

If you perform an audit on a credit scoring model today using a specific baseline, and your colleague performs the same audit next month using a different sample for the baseline, the two results will be mathematically inconsistent. Even if the model has not changed, the “explanation” will have shifted. This is why the baseline is not just a technical parameter—it is an audit artifact that must be saved alongside your model weights.

Step-by-Step Guide: Implementing Baseline Versioning

1. Define the Baseline Strategy: Decide whether your baseline will be the mean of the training set, the median, or a specific cluster of “neutral” observations. Avoid using the entire training set as a baseline for large models, as this is computationally expensive and introduces unnecessary noise.
2. Freeze the Background Dataset: Once you have defined your baseline, save the specific subset of data used as your background set. Treat this file with the same level of version control as your model artifacts. Store it in your model registry (e.g., MLflow, DVC, or SageMaker Model Registry).
3. Integrate into the Pipeline: Modify your inference pipeline so that it explicitly loads the saved baseline data object. Never derive the baseline on the fly during an audit.
4. Metadata Documentation: Create a manifest file that accompanies every model deployment. This file should contain the hash of the baseline dataset, the number of samples used, and the methodology (e.g., “K-means clustering, k=100”).
5. Verification Check: Before signing off on an audit report, run a “reproducibility check.” Re-run a sample explanation using the archived baseline. If the result deviates from your audit report, your documentation process has failed.

Real-World Applications

Consider a retail bank using a gradient-boosted tree model to determine loan eligibility. A regulator flags a decision as potentially discriminatory. The bank’s data science team uses SHAP to prove that “Income” was the primary driver of the decision, not “Ethnicity.”

If the bank did not save the baseline, a regulator could argue: “If you had chosen a different background dataset, would ‘Ethnicity’ have appeared more important?” By providing the specific, versioned baseline data used in the original calculation, the bank can re-run the exact same SHAP computation in front of the regulator, proving that the explanation is stable and reliable.

Similarly, in the healthcare sector, diagnostics models are audited for consistency. A model flagging a patient for a high risk of heart disease must be consistently explained to the attending physician. When the model is updated, the baseline must be reassessed and versioned to ensure that the “why” behind the diagnosis remains comparable across versions of the clinical software.

Common Mistakes to Avoid

• Using the Entire Training Set: While standard documentation often suggests using the full training set as a baseline, this is inefficient for production systems. It creates large, unmanageable artifacts and makes debugging difficult. Use a representative, small sample instead.
• Ignoring Data Drift: If your input data drifts significantly over time, your original baseline may become obsolete. Always track the relationship between your data distribution and your baseline, and update the baseline if the environment changes significantly—but ensure you keep the old baseline archived for previous model versions.
• Implicit Defaults: Relying on the default SHAP background (which some libraries set to zero) is dangerous. It assumes that a value of zero is inherently meaningful, which is rarely true in real-world tabular data.
• Lack of Versioning: Treating the baseline as a transient, temporary variable rather than a core model component. If the baseline isn’t in your Git repo or model registry, it doesn’t exist for the purpose of an audit.

Advanced Tips for Robust Audits

Use K-Means for Representative Baselines: If you want your baseline to reflect the distribution of your data accurately, use a clustering algorithm (like K-means) to select a set of “centroid” observations. This ensures that the baseline represents the range of “typical” model inputs without requiring you to carry the weight of the entire training dataset.

Sensitivity Analysis: Perform a sensitivity analysis on your baseline choice. Calculate SHAP values for the same prediction using three different baseline subsets. If the resulting feature importance rankings change drastically, your model’s explanations are unstable. This is a red flag that you should address by either simplifying your model or using a more robust baseline strategy.

Automation is Non-Negotiable: Use CI/CD pipelines to validate that every time a model is deployed, the baseline dataset is also validated and stored. If the pipeline detects a missing or mismatched baseline, the deployment should automatically halt.

Conclusion

The SHAP value is a powerful tool, but it is only as reliable as the reference point it is measured against. By failing to record and version your baseline values, you undermine the very goal of transparency that SHAP is intended to serve. An audit that cannot be reproduced is essentially a subjective observation, which is insufficient in any environment where fairness, accountability, and regulatory compliance are paramount.