Data Cleaning for Machine Learning: Handling Missing Values, Outliers, and Noise

In academic tutorials and textbook problem sets, datasets arrive impeccably sanitized: perfectly formatted matrices with zero null values, clean bell curves, and reliable labels. In the real world, production data is messy, corrupted, and incomplete. Sensors lose battery power, web forms allow skipped fields, network timeouts drop packets, and human data entry introduces erratic typos.

Attempting to train an algorithm on raw, uncleaned data leads to catastrophic failures: distance-based models crash on NaN values, linear regression coefficients skew dramatically due to extreme outliers, and predictive performance degrades. In this guide, the second chapter of Module 3 in our AI Fundamentals series, we master the core techniques of professional data cleaning: diagnosing missing value mechanisms, applying statistical and machine learning imputation, and robustly detecting and treating outliers.

1. The Anatomy of Missing Data: Understanding the Mechanisms

Before blindly deleting rows or substituting numbers, an engineer must diagnose why the data is missing. Statistical theory identifies three distinct missingness mechanisms:

1. Missing Completely at Random (MCAR): The probability of a value being missing is entirely independent of both observed and unobserved data. Example: A laboratory test tube accidentally falls off a counter and shatters. Deleting MCAR rows does not introduce statistical bias.
2. Missing at Random (MAR): The probability of missingness depends on other observed variables in the dataset, but not on the missing value itself. Example: Younger users are statistically less likely to report their annual income than older users. If we control for age, missingness is random.
3. Missing Not at Random (MNAR): The probability of missingness depends directly on the unobserved value itself. Example: Individuals with extremely high debt or severe depression deliberately skip corresponding survey questions. Simply dropping MNAR rows introduces severe selection bias into the model.

2. Strategies for Handling Missing Values

Depending on the percentage of missing values and the underlying mechanism, machine learning practitioners deploy four primary remediation strategies:

1. Deletion (Listwise or Feature Dropping)

• Row Deletion (Complete Case Analysis): Drop rows containing null values. Only permissible when missing data constitutes less than 3% to 5% of an otherwise massive dataset under an MCAR assumption.
• Column Deletion: If a feature column is missing more than 60% to 70% of its entries and cannot be reliably imputed, dropping the entire column is often safer than introducing synthetic noise.

2. Statistical Imputation (Mean, Median, Mode)

• Mean Imputation: Replaces missing numerical cells with the feature average. Caution: Highly vulnerable to outliers and artificially compresses variance.
• Median Imputation: Replaces missing numerical cells with the 50th percentile value. Robust to skewness and extreme outliers; the gold standard for basic numerical imputation.
• Mode Imputation: Replaces missing categorical cells with the most frequent category.
• Missing Indicator Flags: Always create a companion binary column (e.g., income_was_missing = 1). This preserves the predictive signal of missingness for tree-based models.

3. Advanced Algorithmic Imputation (KNN & Iterative Imputer)

• K-Nearest Neighbors (KNN) Imputation: Identifies the $K$ most similar samples based on available features and averages their values to fill the missing cell. Preserves complex multi-variable relationships.
• Iterative Imputer (MICE): Models each feature with missing values as a function of other features via sequential round-robin regression models.

3. Outlier Detection: Finding the Anomalies

An outlier is an observation that deviates so significantly from other observations as to arouse suspicions that it was generated by a different mechanism. Outliers skew the mean, inflate variance, and distort linear regression lines. Two primary statistical tools detect univariate outliers:

1. The Z-Score Method (Parametric)

Measures how many standard deviations ($\sigma$) a data point lies away from the sample mean ($\mu$):

$$z = \frac{x – \mu}{\sigma}$$

Under a standard normal distribution, observations with $|z| > 3.0$ represent fewer than 0.27% of all data points and are flagged as statistical outliers. Limitation: Assumes the underlying feature follows a Gaussian bell curve.

2. The Interquartile Range (IQR) Method (Non-Parametric)

The IQR method requires zero distributional assumptions. It computes the difference between the 75th percentile ($Q_3$) and the 25th percentile ($Q_1$):

$$\text{IQR} = Q_3 – Q_1$$

Data points falling outside the Tukey fences are flagged as outliers:

• Lower Bound: $Q_1 – 1.5 \times \text{IQR}$
• Upper Bound: $Q_3 + 1.5 \times \text{IQR}$

4. Outlier Treatment: What to Do with Flagged Points

1. Trimming (Removal): Delete the outlier rows. Only advisable if the point is confirmed measurement error (e.g., human age recorded as 999).
2. Winsorization (Capping / Clipping): Cap extreme values at predefined threshold percentiles (e.g., capping all values above the 99th percentile at the 99th percentile value). Preserves sample size while neutralizing leverage.
3. Log Transformation: Compresses right-skewed power-law distributions (e.g., income or website views) using $\log(1 + x)$, naturally pulling extreme values inward toward the mean.

5. Summary Comparison: Data Cleaning Techniques

Technique	Target Issue	Key Assumption	Primary Advantage	Key Limitation
Mean Imputation	Missing Numerical	Normal distribution, MCAR	Instantaneous computation	Distorts variance, vulnerable to outliers
Median Imputation	Missing Numerical	Skewed distribution, MCAR/MAR	Robust to outliers, preserves median	Attenuates feature correlations
KNN Imputation	Missing Values	Feature correlations exist	Preserves complex multi-variable relationships	Computationally slow on large datasets
Z-Score Capping	Outliers	Normal (Gaussian) distribution	Standardized threshold ($|z| > 3$)	Fails on skewed or multi-modal data
IQR (Tukey) Fencing	Outliers	Non-parametric (any distribution)	Robust against extreme skewness	Can flag valid rare events in long-tailed data

6. Hands-On Python Implementation: End-to-End Data Cleaning Pipeline

import numpy as np
import pandas as pd

# Set random seed
np.random.seed(42)
n = 100

# 1. Synthesize a Corrupted Dataset with NaNs and Extreme Outliers
income = np.random.normal(55000, 12000, size=n)
# Introduce 3 severe outliers
income[10] = 450000.0
income[25] = -15000.0  # Impossible negative value
income[50] = 380000.0

age = np.random.randint(20, 65, size=n).astype(float)
# Introduce missing values (NaNs)
age[np.random.choice(n, size=10, replace=False)] = np.nan

df = pd.DataFrame({"age": age, "income": income})

print("=== 1. RAW CORRUPTED DATASET SUMMARY ===")
print(f"Missing Values in Age: {df['age'].isnull().sum()} / {n}")
print(f"Raw Income - Mean: ${df['income'].mean():,.2f} | Median: ${df['income'].median():,.2f}\\n")

# 2. Handling Missing Values with Median Imputation + Binary Indicator
df_clean = df.copy()

# Create binary missing indicator flag
df_clean["age_was_missing"] = df_clean["age"].isnull().astype(int)

# Impute age with median
median_age = df_clean["age"].median()
df_clean["age"] = df_clean["age"].fillna(median_age)

print("=== 2. MISSING DATA IMPUTATION ===")
print(f"Imputed Age using Median: {median_age:.1f}")
print(f"Remaining Missing Values in Age: {df_clean['age'].isnull().sum()}\\n")

# 3. Outlier Detection using IQR Method
Q1 = df_clean["income"].quantile(0.25)
Q3 = df_clean["income"].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

outliers = df_clean[(df_clean["income"] < lower_bound) | (df_clean["income"] > upper_bound)]

print("=== 3. IQR OUTLIER DETECTION ===")
print(f"Q1 (25th): ${Q1:,.2f} | Q3 (75th): ${Q3:,.2f} | IQR: ${IQR:,.2f}")
print(f"Acceptable Bounds: [${lower_bound:,.2f}, ${upper_bound:,.2f}]")
print(f"Detected Outliers ({len(outliers)} rows):")
print(outliers["income"].values)

# 4. Outlier Treatment via Winsorization (Clipping)
df_clean["income_clipped"] = np.clip(df_clean["income"], lower_bound, upper_bound)

print(f"\\n=== 4. POST-CLEANING VERIFICATION ===")
print(f"Cleaned Income - Mean: ${df_clean['income_clipped'].mean():,.2f}")
print(f"Cleaned Income - Max: ${df_clean['income_clipped'].max():,.2f}")
print(f"Cleaned Income - Min: ${df_clean['income_clipped'].min():,.2f}")

7. Practical Engineering Takeaways for AI Builders

1. Fit Imputers Strictly on Training Data: Never compute the mean or median across the entire dataset before splitting into train and test sets. Doing so causes Data Leakage, contaminating your test set with training signals. Always fit imputers on train data and transform test data.
2. Tree Models Handle Missing Values Differently: Algorithms like XGBoost and LightGBM possess native default split directions for NaN values, often making explicit imputation unnecessary for gradient boosting.
3. Never Blindly Drop Outliers: In fields like fraud detection, cyber-security, and rare disease diagnosis, the outliers are the entire target of interest. Confirm the operational context before trimming data points.

8. Frequently Asked Questions (FAQ)

What is the difference between MCAR, MAR, and MNAR?

MCAR (Completely at Random): Missingness is entirely accidental. MAR (At Random): Missingness relates to other observed features. MNAR (Not at Random): Missingness relates directly to the value of the missing feature itself, introducing severe statistical bias if ignored.

When should I use IQR instead of Z-scores for outlier detection?

Use the IQR method whenever data is skewed, multi-modal, or does not strictly follow a Gaussian normal distribution. The Z-score method relies on the mean and standard deviation, which are themselves distorted by the very outliers you are trying to detect.