The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Neural Network Architecture

Next ►
►

In this section, we will delve into the architecture of neural networks, which are the backbone of many modern AI applications. Understanding the structure and components of neural networks is crucial for designing and implementing effective AI models.

Key Concepts

  1. Neurons and Layers

    • Neurons: The basic units of a neural network, analogous to biological neurons. Each neuron receives input, processes it, and passes the output to the next layer.
    • Layers: Neural networks are composed of multiple layers of neurons. The three main types of layers are:
      • Input Layer: The first layer that receives the initial data.
      • Hidden Layers: Intermediate layers that process inputs from the input layer.
      • Output Layer: The final layer that produces the network's output.

  2. Activation Functions

    • Functions that determine the output of a neuron given an input or set of inputs. Common activation functions include:
      • Sigmoid: \( \sigma(x) = \frac{1}{1 + e^{-x}} \)
      • ReLU (Rectified Linear Unit): \( \text{ReLU}(x) = \max(0, x) \)
      • Tanh: \( \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} \)

  3. Weights and Biases

    • Weights: Parameters that transform input data within the network. Each connection between neurons has an associated weight.
    • Biases: Additional parameters that allow the activation function to be shifted to the left or right, improving the model's flexibility.

  4. Forward Propagation

    • The process by which input data is passed through the network, layer by layer, to generate an output.

  5. Backpropagation

    • A method used to train neural networks by adjusting weights and biases based on the error of the output. It involves:
      • Calculating the error at the output.
      • Propagating this error backward through the network.
      • Updating the weights and biases to minimize the error.

Example: Simple Neural Network

Let's consider a simple neural network with one input layer, one hidden layer, and one output layer.

Structure

  • Input Layer: 3 neurons (features)
  • Hidden Layer: 4 neurons
  • Output Layer: 1 neuron (binary classification)

Diagram

Input Layer       Hidden Layer       Output Layer
  (3 neurons)       (4 neurons)        (1 neuron)
     o                  o                  o
     o                  o
     o                  o
                        o

Code Example

Here's a basic implementation of a neural network using Python and NumPy:

import numpy as np

# Activation function and its derivative
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    return x * (1 - x)

# Input data (3 features)
inputs = np.array([[0, 0, 1],
                   [1, 1, 1],
                   [1, 0, 1],
                   [0, 1, 1]])

# Expected output (binary classification)
outputs = np.array([[0], [1], [1], [0]])

# Seed for reproducibility
np.random.seed(1)

# Initialize weights randomly with mean 0
weights_input_hidden = 2 * np.random.random((3, 4)) - 1
weights_hidden_output = 2 * np.random.random((4, 1)) - 1

# Training process
for epoch in range(10000):
    # Forward propagation
    input_layer = inputs
    hidden_layer = sigmoid(np.dot(input_layer, weights_input_hidden))
    output_layer = sigmoid(np.dot(hidden_layer, weights_hidden_output))
    
    # Calculate error
    output_error = outputs - output_layer
    
    # Backpropagation
    output_delta = output_error * sigmoid_derivative(output_layer)
    hidden_error = output_delta.dot(weights_hidden_output.T)
    hidden_delta = hidden_error * sigmoid_derivative(hidden_layer)
    
    # Update weights
    weights_hidden_output += hidden_layer.T.dot(output_delta)
    weights_input_hidden += input_layer.T.dot(hidden_delta)

# Output after training
print("Output after training:")
print(output_layer)

Explanation

  • Initialization: We initialize the weights for the input-to-hidden and hidden-to-output layers randomly.
  • Forward Propagation: We calculate the activations for the hidden and output layers using the sigmoid function.
  • Error Calculation: We compute the error at the output layer.
  • Backpropagation: We propagate the error backward, calculating the deltas for the hidden and output layers.
  • Weight Update: We update the weights to minimize the error.

Practical Exercise

Exercise

Implement a neural network with the following specifications:

  • Input Layer: 2 neurons
  • Hidden Layer: 3 neurons
  • Output Layer: 1 neuron

Use the following input data and expected output:

inputs = np.array([[0, 0],
                   [0, 1],
                   [1, 0],
                   [1, 1]])

outputs = np.array([[0], [1], [1], [0]])

Solution

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    return x * (1 - x)

inputs = np.array([[0, 0],
                   [0, 1],
                   [1, 0],
                   [1, 1]])

outputs = np.array([[0], [1], [1], [0]])

np.random.seed(1)

weights_input_hidden = 2 * np.random.random((2, 3)) - 1
weights_hidden_output = 2 * np.random.random((3, 1)) - 1

for epoch in range(10000):
    input_layer = inputs
    hidden_layer = sigmoid(np.dot(input_layer, weights_input_hidden))
    output_layer = sigmoid(np.dot(hidden_layer, weights_hidden_output))
    
    output_error = outputs - output_layer
    
    output_delta = output_error * sigmoid_derivative(output_layer)
    hidden_error = output_delta.dot(weights_hidden_output.T)
    hidden_delta = hidden_error * sigmoid_derivative(hidden_layer)
    
    weights_hidden_output += hidden_layer.T.dot(output_delta)
    weights_input_hidden += input_layer.T.dot(hidden_delta)

print("Output after training:")
print(output_layer)

Common Mistakes and Tips

  • Incorrect Weight Initialization: Ensure weights are initialized randomly but within a small range.
  • Learning Rate: Adjust the learning rate if the network is not converging.
  • Overfitting: Use techniques like regularization if the network performs well on training data but poorly on test data.

Conclusion

In this section, we explored the architecture of neural networks, including neurons, layers, activation functions, weights, biases, forward propagation, and backpropagation. We also implemented a simple neural network and provided an exercise to reinforce the concepts. Understanding these fundamentals is crucial for building more complex AI models and diving deeper into advanced topics like deep learning.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2024. All rights reserved