The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Overfitting and Underfitting

Next ►
►

In this section, we will explore two common issues in machine learning: overfitting and underfitting. Understanding these concepts is crucial for building models that generalize well to new, unseen data.

What is Overfitting?

Overfitting occurs when a machine learning model learns the details and noise in the training data to such an extent that it negatively impacts the model's performance on new data. This means the model is too complex and captures the random fluctuations in the training data rather than the intended outputs.

Characteristics of Overfitting:

  • High accuracy on training data.
  • Poor performance on validation/test data.
  • Model complexity is too high.

Example:

Consider a polynomial regression model trying to fit a dataset:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Generate some data
np.random.seed(0)
X = np.sort(np.random.rand(20, 1) * 10, axis=0)
y = np.sin(X).ravel() + np.random.randn(20) * 0.5

# Fit polynomial regression model
poly = PolynomialFeatures(degree=15)
X_poly = poly.fit_transform(X)
model = LinearRegression()
model.fit(X_poly, y)

# Predict and plot
X_test = np.linspace(0, 10, 100).reshape(-1, 1)
X_test_poly = poly.transform(X_test)
y_pred = model.predict(X_test_poly)

plt.scatter(X, y, color='black')
plt.plot(X_test, y_pred, color='blue', linewidth=2)
plt.title('Overfitting Example')
plt.show()

In this example, the model fits the training data very well but is likely to perform poorly on new data due to its complexity.

What is Underfitting?

Underfitting occurs when a machine learning model is too simple to capture the underlying patterns in the data. This means the model performs poorly on both the training data and new data.

Characteristics of Underfitting:

  • Low accuracy on training data.
  • Poor performance on validation/test data.
  • Model complexity is too low.

Example:

Consider a linear regression model trying to fit a dataset:

# Generate some data
np.random.seed(0)
X = np.sort(np.random.rand(20, 1) * 10, axis=0)
y = np.sin(X).ravel() + np.random.randn(20) * 0.5

# Fit linear regression model
model = LinearRegression()
model.fit(X, y)

# Predict and plot
y_pred = model.predict(X_test)

plt.scatter(X, y, color='black')
plt.plot(X_test, y_pred, color='red', linewidth=2)
plt.title('Underfitting Example')
plt.show()

In this example, the linear model is too simple to capture the non-linear relationship in the data.

Balancing Model Complexity

To achieve a balance between overfitting and underfitting, we need to find the right model complexity. This can be done through techniques such as:

  • Cross-Validation: Splitting the data into training and validation sets multiple times to ensure the model generalizes well.
  • Regularization: Adding a penalty to the model for complexity (e.g., L1 or L2 regularization).
  • Pruning: Reducing the complexity of decision trees by removing branches that have little importance.
  • Early Stopping: Halting the training process when the model's performance on a validation set starts to degrade.

Example of Regularization:

Using Ridge Regression (L2 regularization):

from sklearn.linear_model import Ridge

# Fit Ridge regression model
ridge_model = Ridge(alpha=1.0)
ridge_model.fit(X_poly, y)

# Predict and plot
y_pred_ridge = ridge_model.predict(X_test_poly)

plt.scatter(X, y, color='black')
plt.plot(X_test, y_pred_ridge, color='green', linewidth=2)
plt.title('Regularization Example')
plt.show()

Practical Exercise

Exercise:

  1. Generate a synthetic dataset with a non-linear relationship.
  2. Split the dataset into training and test sets.
  3. Fit a linear regression model and a polynomial regression model to the training data.
  4. Evaluate the performance of both models on the test data.
  5. Apply Ridge regression to the polynomial model and observe the changes in performance.

Solution:

from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Generate synthetic data
np.random.seed(0)
X = np.sort(np.random.rand(100, 1) * 10, axis=0)
y = np.sin(X).ravel() + np.random.randn(100) * 0.5

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# Fit linear regression model
linear_model = LinearRegression()
linear_model.fit(X_train, y_train)
y_pred_linear = linear_model.predict(X_test)

# Fit polynomial regression model
poly = PolynomialFeatures(degree=15)
X_train_poly = poly.fit_transform(X_train)
X_test_poly = poly.transform(X_test)
poly_model = LinearRegression()
poly_model.fit(X_train_poly, y_train)
y_pred_poly = poly_model.predict(X_test_poly)

# Fit Ridge regression model
ridge_model = Ridge(alpha=1.0)
ridge_model.fit(X_train_poly, y_train)
y_pred_ridge = ridge_model.predict(X_test_poly)

# Evaluate models
mse_linear = mean_squared_error(y_test, y_pred_linear)
mse_poly = mean_squared_error(y_test, y_pred_poly)
mse_ridge = mean_squared_error(y_test, y_pred_ridge)

print(f"Linear Regression MSE: {mse_linear}")
print(f"Polynomial Regression MSE: {mse_poly}")
print(f"Ridge Regression MSE: {mse_ridge}")

# Plot results
plt.scatter(X_test, y_test, color='black', label='Test Data')
plt.plot(X_test, y_pred_linear, color='red', linewidth=2, label='Linear Model')
plt.plot(X_test, y_pred_poly, color='blue', linewidth=2, label='Polynomial Model')
plt.plot(X_test, y_pred_ridge, color='green', linewidth=2, label='Ridge Model')
plt.legend()
plt.title('Model Comparison')
plt.show()

Conclusion

In this section, we have learned about overfitting and underfitting, two critical issues in machine learning. We explored their characteristics, examples, and techniques to balance model complexity. By understanding these concepts, you can build models that generalize well to new data, ensuring better performance and reliability.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2024. All rights reserved