The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Basic Concepts of Neural Networks

Next ►
►

Introduction

Neural networks are the foundation of deep learning. They are computational models inspired by the human brain, designed to recognize patterns and solve complex problems. In this section, we will cover the basic concepts of neural networks, including their structure, components, and how they function.

Key Concepts

  1. Neurons and Layers

  • Neuron: The basic unit of a neural network, also known as a node or perceptron. It receives input, processes it, and produces an output.
  • Layers: Neural networks consist of multiple layers:
    • Input Layer: The first layer that receives the input data.
    • Hidden Layers: Intermediate layers that process the input data. There can be one or more hidden layers.
    • Output Layer: The final layer that produces the output.

  1. Weights and Biases

  • Weights: Parameters that determine the importance of each input. Each connection between neurons has an associated weight.
  • Biases: Additional parameters that allow the model to fit the data better by shifting the activation function.

  1. Activation Functions

  • Functions that determine the output of a neuron based on its input. Common activation functions include:
    • Sigmoid: \( \sigma(x) = \frac{1}{1 + e^{-x}} \)
    • ReLU (Rectified Linear Unit): \( \text{ReLU}(x) = \max(0, x) \)
    • Tanh: \( \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} \)

  1. Forward Propagation

  • The process of passing input data through the network to obtain an output. Each neuron's output is calculated and passed to the next layer.

  1. Loss Function

  • A function that measures the difference between the predicted output and the actual output. Common loss functions include:
    • Mean Squared Error (MSE): \( \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \)
    • Cross-Entropy Loss: Used for classification problems.

  1. Backward Propagation

  • The process of updating the weights and biases to minimize the loss function. It involves calculating the gradient of the loss function with respect to each weight and bias and adjusting them accordingly.

Example: Simple Neural Network

Let's create a simple neural network with one hidden layer to understand these concepts better.

Network Structure

  • Input Layer: 2 neurons (features)
  • Hidden Layer: 3 neurons
  • Output Layer: 1 neuron (binary classification)

Code Example

import numpy as np

# Activation function: Sigmoid
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Derivative of Sigmoid
def sigmoid_derivative(x):
    return x * (1 - x)

# Input data (4 samples, 2 features)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])

# Output data (4 samples, 1 output)
y = np.array([[0], [1], [1], [0]])

# Seed for reproducibility
np.random.seed(42)

# Initialize weights and biases
input_layer_neurons = X.shape[1]
hidden_layer_neurons = 3
output_layer_neurons = 1

# Weights
W1 = np.random.uniform(size=(input_layer_neurons, hidden_layer_neurons))
W2 = np.random.uniform(size=(hidden_layer_neurons, output_layer_neurons))

# Biases
b1 = np.random.uniform(size=(1, hidden_layer_neurons))
b2 = np.random.uniform(size=(1, output_layer_neurons))

# Training parameters
learning_rate = 0.1
epochs = 10000

# Training loop
for epoch in range(epochs):
    # Forward Propagation
    hidden_layer_input = np.dot(X, W1) + b1
    hidden_layer_output = sigmoid(hidden_layer_input)
    
    output_layer_input = np.dot(hidden_layer_output, W2) + b2
    predicted_output = sigmoid(output_layer_input)
    
    # Calculate error
    error = y - predicted_output
    
    # Backward Propagation
    d_predicted_output = error * sigmoid_derivative(predicted_output)
    error_hidden_layer = d_predicted_output.dot(W2.T)
    d_hidden_layer = error_hidden_layer * sigmoid_derivative(hidden_layer_output)
    
    # Update weights and biases
    W2 += hidden_layer_output.T.dot(d_predicted_output) * learning_rate
    b2 += np.sum(d_predicted_output, axis=0, keepdims=True) * learning_rate
    W1 += X.T.dot(d_hidden_layer) * learning_rate
    b1 += np.sum(d_hidden_layer, axis=0, keepdims=True) * learning_rate

# Output after training
print("Predicted Output: \n", predicted_output)

Explanation

  1. Initialization: We initialize the weights and biases randomly.
  2. Forward Propagation: We calculate the output of each layer using the sigmoid activation function.
  3. Error Calculation: We compute the error between the predicted output and the actual output.
  4. Backward Propagation: We calculate the gradients and update the weights and biases to minimize the error.
  5. Training Loop: We repeat the forward and backward propagation steps for a specified number of epochs.

Practical Exercise

Exercise: Implement a Neural Network from Scratch

Task: Implement a neural network with the following structure:

  • Input Layer: 3 neurons
  • Hidden Layer: 4 neurons
  • Output Layer: 2 neurons (multi-class classification)

Steps:

  1. Initialize the weights and biases.
  2. Implement the forward propagation.
  3. Calculate the loss using cross-entropy loss.
  4. Implement the backward propagation.
  5. Train the network for 5000 epochs.

Solution:

import numpy as np

# Activation function: Softmax
def softmax(x):
    exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=1, keepdims=True)

# Derivative of Softmax
def softmax_derivative(x):
    return x * (1 - x)

# Input data (4 samples, 3 features)
X = np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0], [1, 1, 1]])

# Output data (4 samples, 2 outputs)
y = np.array([[1, 0], [0, 1], [0, 1], [1, 0]])

# Seed for reproducibility
np.random.seed(42)

# Initialize weights and biases
input_layer_neurons = X.shape[1]
hidden_layer_neurons = 4
output_layer_neurons = 2

# Weights
W1 = np.random.uniform(size=(input_layer_neurons, hidden_layer_neurons))
W2 = np.random.uniform(size=(hidden_layer_neurons, output_layer_neurons))

# Biases
b1 = np.random.uniform(size=(1, hidden_layer_neurons))
b2 = np.random.uniform(size=(1, output_layer_neurons))

# Training parameters
learning_rate = 0.1
epochs = 5000

# Training loop
for epoch in range(epochs):
    # Forward Propagation
    hidden_layer_input = np.dot(X, W1) + b1
    hidden_layer_output = softmax(hidden_layer_input)
    
    output_layer_input = np.dot(hidden_layer_output, W2) + b2
    predicted_output = softmax(output_layer_input)
    
    # Calculate error
    error = y - predicted_output
    
    # Backward Propagation
    d_predicted_output = error * softmax_derivative(predicted_output)
    error_hidden_layer = d_predicted_output.dot(W2.T)
    d_hidden_layer = error_hidden_layer * softmax_derivative(hidden_layer_output)
    
    # Update weights and biases
    W2 += hidden_layer_output.T.dot(d_predicted_output) * learning_rate
    b2 += np.sum(d_predicted_output, axis=0, keepdims=True) * learning_rate
    W1 += X.T.dot(d_hidden_layer) * learning_rate
    b1 += np.sum(d_hidden_layer, axis=0, keepdims=True) * learning_rate

# Output after training
print("Predicted Output: \n", predicted_output)

Common Mistakes and Tips

  • Initialization: Ensure weights and biases are initialized properly to avoid vanishing or exploding gradients.
  • Learning Rate: Choose an appropriate learning rate. Too high can cause divergence, too low can slow down the training.
  • Activation Functions: Use appropriate activation functions for different layers. For example, ReLU for hidden layers and softmax for output layers in classification problems.

Conclusion

In this section, we covered the basic concepts of neural networks, including neurons, layers, weights, biases, activation functions, forward and backward propagation, and loss functions. We also implemented a simple neural network from scratch to solidify these concepts. Understanding these basics is crucial as we delve deeper into more complex neural network architectures in the upcoming modules.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2024. All rights reserved