The Silent Killer of Models: Why You Must Document Every Feature Engineering Assumption

Introduction

In the world of data science, we often obsess over hyperparameter tuning and model architecture. We spend hours agonizing over learning rates and loss functions. Yet, the most frequent cause of model failure isn’t a complex mathematical error; it is the “invisible” logic embedded within our feature engineering pipeline. When we transform raw data into model-ready inputs, we make dozens of small, often unconscious assumptions about the data’s distribution, causality, and stability. When these assumptions aren’t documented, they become technical debt that eventually bankrupts the project’s reliability.

Documentation in feature engineering is not mere administrative overhead; it is the audit trail of your intelligence. Without it, you are building a skyscraper on a shifting foundation. By documenting your assumptions, you ensure reproducibility, facilitate easier debugging, and allow stakeholders to understand the “why” behind the “what.”

Key Concepts: The Anatomy of an Assumption

An assumption in feature engineering is any decision where you inferred information that isn’t explicitly present in the data. These fall into three primary categories:

• Imputation Assumptions: Deciding that missing values mean “zero,” “the mean,” or “unknown” is a significant leap. If you replace nulls with a global mean, you are assuming the missingness is completely random (MCAR). If that assumption is wrong, you have just injected systematic bias into your model.
• Distributional Assumptions: Many models assume normality or stationarity. When you log-transform a skewed variable or remove outliers, you are assuming that the “extreme” values are noise rather than signal.
• Causal and Logical Assumptions: Creating a “Time-Since-Last-Purchase” feature assumes that the interval is a predictor of future behavior. This assumes that the user’s history is a relevant proxy for their future intent.

When these assumptions go undocumented, a new team member or a future version of yourself cannot distinguish between a deliberate feature design choice and a mistake caused by data quality issues.

Step-by-Step Guide to Documenting Assumptions

1. Create a “Feature Dictionary” with a Logic Column: Move beyond the standard CSV description. Add a column titled “Assumptions/Constraints.” For every feature, record why you chose the transformation and what you believe to be true about the data.
2. Record Data Source Limitations: If you are aggregating data, document the assumption regarding the grain. If you aggregate transactions by month, you are assuming that monthly trends are more predictive than daily or weekly trends. State why.
3. Define the Handling of Edge Cases: Explicitly document how your feature pipeline handles division by zero, empty strings, or values outside of an expected range (e.g., negative ages). Are these treated as errors or mapped to a sentinel value?
4. Version Control Your Rationale: Use Git or DVC (Data Version Control) not just for code, but for the reasoning behind the code. A commit message that says “updated feature engineering logic” is useless. Use: “Updated income_per_capita: changed imputation from median to zero to account for segment-specific missingness.”
5. Set Up “Assumption Alerts”: If you assume a feature will follow a specific distribution, write a small unit test or an assertion in your pipeline that triggers a warning if the data distribution shifts beyond a certain threshold.

Examples and Case Studies

The Churn Prediction Debacle

A telecommunications company built a churn model using a feature called “average_call_duration.” The feature engineering pipeline assumed that missing values meant the customer hadn’t made any calls that month, so they imputed these with zeros. Six months later, the model began failing. It turned out that a system upgrade caused a data pipeline failure where thousands of records were dropped, turning existing call durations into nulls. Because the assumption (null = zero usage) was never documented or tested, the model began predicting that these high-value customers were actually inactive, leading to a disastrous retention campaign.

The Retail Pricing Model

In a retail price elasticity model, an engineer assumed that any product with zero sales for 30 days was “discontinued.” They filtered these out of the training set. However, they didn’t document this threshold. When a new category of seasonal products was introduced, the model ignored them entirely, treating the slow start as “discontinued.” Because the “30-day” assumption wasn’t surfaced in documentation, the team spent weeks debugging the model’s coefficients rather than the underlying feature filter.

Documentation turns an “intuition-based” project into a “process-based” one. It transforms your data science team from a group of magicians into a team of engineers.

Common Mistakes

• The “Memory-Only” Fallacy: Assuming you will remember why you did something is the most dangerous error in engineering. Documentation must be written down, ideally inside the code repository.
• Vague Documentation: Writing “Cleaned the data” in your notes is useless. Instead, write, “Removed records with price < 0.01 under the assumption that these represent test transactions, not actual sales."
• Ignoring Data Drift: Assuming that the conditions present during training will hold true during production. If you assume feature X is a constant, document that you have performed no monitoring on it.
• Over-documenting the Obvious: Don’t document basic syntax (e.g., “calculating mean”). Document the intent and the belief behind the variable.

Advanced Tips

To take your documentation to a professional level, consider Data Contracts. A data contract is a formal agreement between the data producer and the data consumer (your model). It defines the expected schema, the expected distributions, and the assumptions you’ve made about the data flow.

Additionally, integrate your documentation into your testing suite. If you assume that “User_Age” will always be between 18 and 100, add a “Great Expectations” or “Pandera” validator to your pipeline. If the data violates your documented assumption, the pipeline should stop. This forces your assumptions to become active, enforced constraints rather than passive comments in a document.

Finally, always perform a “Stress Test” on your assumptions. During your EDA (Exploratory Data Analysis) phase, specifically try to prove yourself wrong. If you assume that a lack of web traffic indicates an inactive user, try to find a sample of users with no web traffic who were actually very active via mobile. Documenting that you performed this test—and why you stuck with your original assumption despite the edge cases—is the hallmark of a senior-level practitioner.

Conclusion

The quality of your model is ultimately capped by the quality of your assumptions. By treating your feature engineering assumptions as first-class citizens in your project, you move away from “black-box” data science and toward a robust, reliable engineering discipline. Documenting these choices prevents technical debt, improves collaboration, and safeguards your project against the inevitable changes in real-world data.

Start today: pick your current project, identify three core features, and write down the assumptions you’ve made about them. You will likely find at least one assumption that you aren’t sure about—and finding that uncertainty now is much cheaper than finding it after your model goes into production.