The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Handling Missing Data

Next ►
►

Handling missing data is a critical step in the data preprocessing phase of any machine learning project. Missing data can lead to biased estimates, reduce the representativeness of the sample, and ultimately degrade the performance of machine learning models. This section will cover various techniques to handle missing data effectively.

Key Concepts

  1. Types of Missing Data:

    • Missing Completely at Random (MCAR): The missingness is independent of both observed and unobserved data.
    • Missing at Random (MAR): The missingness is related to the observed data but not the missing data.
    • Missing Not at Random (MNAR): The missingness is related to the unobserved data.

  2. Techniques for Handling Missing Data:

    • Deletion Methods:
      • Listwise Deletion
      • Pairwise Deletion
    • Imputation Methods:
      • Mean/Median/Mode Imputation
      • Regression Imputation
      • K-Nearest Neighbors (K-NN) Imputation
      • Multiple Imputation
    • Advanced Techniques:
      • Using Algorithms that Support Missing Values
      • Predictive Modeling

Deletion Methods

Listwise Deletion

Listwise deletion removes any row with at least one missing value.

import pandas as pd

# Sample DataFrame
data = {'A': [1, 2, None, 4], 'B': [5, None, 7, 8]}
df = pd.DataFrame(data)

# Listwise Deletion
df_listwise = df.dropna()
print(df_listwise)

Explanation:

  • The dropna() function removes any row that contains at least one missing value.

Pairwise Deletion

Pairwise deletion uses all available data to compute statistics, ignoring only the missing values.

import pandas as pd

# Sample DataFrame
data = {'A': [1, 2, None, 4], 'B': [5, None, 7, 8]}
df = pd.DataFrame(data)

# Pairwise Deletion Example
mean_A = df['A'].mean()
mean_B = df['B'].mean()
print(f"Mean of A: {mean_A}, Mean of B: {mean_B}")

Explanation:

  • The mean is calculated using all available data, ignoring the missing values.

Imputation Methods

Mean/Median/Mode Imputation

Impute missing values with the mean, median, or mode of the column.

import pandas as pd

# Sample DataFrame
data = {'A': [1, 2, None, 4], 'B': [5, None, 7, 8]}
df = pd.DataFrame(data)

# Mean Imputation
df_mean_imputed = df.fillna(df.mean())
print(df_mean_imputed)

Explanation:

  • The fillna() function replaces missing values with the mean of the column.

Regression Imputation

Use regression models to predict and impute missing values.

import pandas as pd
from sklearn.linear_model import LinearRegression

# Sample DataFrame
data = {'A': [1, 2, None, 4], 'B': [5, 6, 7, 8]}
df = pd.DataFrame(data)

# Regression Imputation
known = df[df['A'].notna()]
unknown = df[df['A'].isna()]

X_train = known[['B']]
y_train = known['A']
X_test = unknown[['B']]

model = LinearRegression()
model.fit(X_train, y_train)
predicted = model.predict(X_test)

df.loc[df['A'].isna(), 'A'] = predicted
print(df)

Explanation:

  • A linear regression model is trained on the known values and used to predict the missing values.

K-Nearest Neighbors (K-NN) Imputation

Use K-NN to impute missing values based on the nearest neighbors.

import pandas as pd
from sklearn.impute import KNNImputer

# Sample DataFrame
data = {'A': [1, 2, None, 4], 'B': [5, 6, 7, 8]}
df = pd.DataFrame(data)

# K-NN Imputation
imputer = KNNImputer(n_neighbors=2)
df_knn_imputed = pd.DataFrame(imputer.fit_transform(df), columns=df.columns)
print(df_knn_imputed)

Explanation:

  • The KNNImputer uses the nearest neighbors to impute missing values.

Advanced Techniques

Using Algorithms that Support Missing Values

Some machine learning algorithms can handle missing values internally, such as decision trees and certain ensemble methods.

Predictive Modeling

Use advanced predictive models to estimate and impute missing values.

Practical Exercise

Exercise: Given the following DataFrame, impute the missing values using mean imputation and K-NN imputation.

import pandas as pd

# Sample DataFrame
data = {'A': [1, 2, None, 4, 5], 'B': [5, None, 7, 8, 9]}
df = pd.DataFrame(data)

# Mean Imputation
df_mean_imputed = df.fillna(df.mean())

# K-NN Imputation
from sklearn.impute import KNNImputer
imputer = KNNImputer(n_neighbors=2)
df_knn_imputed = pd.DataFrame(imputer.fit_transform(df), columns=df.columns)

print("Mean Imputation:\n", df_mean_imputed)
print("K-NN Imputation:\n", df_knn_imputed)

Solution:

  • Mean Imputation: 
       A    B
0  1.0  5.0
1  2.0  7.25
2  3.0  7.0
3  4.0  8.0
4  5.0  9.0
  • K-NN Imputation: 
       A    B
0  1.0  5.0
1  2.0  7.0
2  3.0  7.0
3  4.0  8.0
4  5.0  9.0

Conclusion

Handling missing data is a crucial step in the data preprocessing pipeline. Various techniques, from simple deletion to advanced imputation methods, can be employed depending on the nature and extent of the missing data. Understanding these methods and their implications ensures that the data fed into machine learning models is as complete and accurate as possible, leading to better model performance and reliability.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2024. All rights reserved