Backpropagation in Neural Network: Understanding the Process

What is Backward Propagation or Backpropagation?

Neural networks comprise interconnected nodes, or neurons, that help decode the complex patterns or relationships between data to develop an output.

There are two fundamental components to neural networks: weights and biases that help in making predictions.

• Weights are the numerical values that indicate the strength of influence of an input on a neuron's output
• Biases allow neurons to activate even when the weighted sum of inputs is insufficient

When input passes through a neural network, the generated output can differ from the desired or target output. This difference is called a loss of error. One can reduce the difference by training the neural networks.

One of the methods for training is back or backward propagation, where the weights and biases are adjusted by propagating the errors backward from the output to the input layer. The technique utilizes optimization algorithms, such as gradient descent or stochastic gradient descent, to update parameters based on the calculated gradients.

How Back Propagation Algorithm Works?

To understand the backward propagation algorithms, let's proceed step by step from the very first step.

Step 1: Neural Network Function

The input layer receives the input and multiplies each value by its corresponding weight. The product enters the hidden layer, where it is processed by applying the activation function and then passed on to the next layer. The process continues until the output is generated. It is called a forward pass as the input travels from the input layer to the output layer.

Step 2: Error Calculation

The output received from the neural network is compared with the desired output. If there is a difference between the two, it is calculated as an error value.

Step 3: Gradient Calculation

The gradient of the loss function is calculated concerning each weight. It is computed using the chain rule and indicates the degree to which each weight contributes to the loss.

Step 4: Backward Propagation

The error gradient is propagated backward through the network, starting from the output layer and proceeding through the hidden layers. Weights are updated in the direct opposite direction to the gradient (gradient descent) to minimize the loss. The learning rate controls the step size of each update.

Step 5: Repetition of the Cycle

The cycle from step 1 to step 4, i.e., forward pass, error calculation, backward pass, and weight update, runs repeatedly (epoch). It occurs until the cycle reaches a target value or loss is stabilized.

Did You Know? 🔍

The AI market is projected to reach a staggering $1,339 billion by 2030.

(Source: Forbes)

Key Concepts of Backward Propagation

Some of the essential concepts of backpropagation in neural networks are as follows:

• Loss Functions: It is the difference between the actual and desired output
• Gradient Calculation: It computes the loss derivative for each weight, layer by layer, using the chain rule
• Epoch: It involves repeated cycles of forward propagation and backward propagation, and weight updates to improve model performance iteratively
• Learning Rate: It is a hyperparameter that controls the magnitude of weight updates

Forward vs. Backward Propagation

The comparison between forward propagation and backward propagation in neural network is as follows:

Why Backpropagation Matters in Deep Learning?

Back propagation is a vital optimization algorithm used in deep learning. Here is why: 

• It helps train the deep learning model, enhancing its efficiency and reducing the prediction errors
• It has helped handle the vanishing gradient problem, which earlier posed a significant challenge in training multi-layered networks
• It is applicable across multiple AIs
• It also supports complex, multi-layered networks and exhibits scalability
• Offers an automated learning process and performance optimization

Backpropagation Formula and Examples

The backpropagation formula and example in machine learning are explained below.

We will be taking the following data:

• Input x = 0.5
• Weight w = 0.4
• Bias b = 0.1
• Target output y = 0.7
• Learning rate n = 0.1

The backpropagation examples will involve a single neuron that uses sigmoid activation for both forward and backward propagation.

Step 1: Forward Pass

Weighted sum (z): w*x + b

z = (0.4 * 0.5) + 0.1 
z = 0.3 
Sigmoid activation (output y’): y’ = 1/(1+(e^-z))
y’ = 1/(1+(e^-0.3))
y’ = 0.574 
Loss (Mean Squared Error): L = ½ (y’-y)^2
L = ½ (0.574-0.7)^2
L = 0.00794

Step 2: Backward Pass (Gradient Calculation) 

Updating weight w using gradient descent:

W’ = w - n*(dL/dw)

Gradient of loss for output:

dL/dy’ = y’ - y
dL/dy’ = 0.574 - 0.7
dL/dy’ = -0.126

Gradient of output for z (Sigmoid derivative):

dy’/dz = y’(1 - y’)
dy’/dz = 0.574(1 - 0574)
dy’/dz = 0.244

Gradient of z for weight w

dz/dw = x = 0.5

Chain Rule (total gradient):

dL/dw = dL/dy’ * dy’/dz * dz/dw
dL/dw = (-0.126) * 0.244 * 0.5 
dL/dw = -0.01537

Step 3: Weighted Update

w’ = 0.4 − 0.1 * (−0.01537)
w’ = 0.40154

Step 4: Bias Update

dz/db = 1dL/db = 
dL/dy’ * dy’/dz * 1
dL/db = -0.0307
b (new) = 0.1 − 0.1 * (−0.0307)
b (new) = 0.10307

Back Propagation Implementation in Python for XOR Problem

The Python implementation of a simple neural network solving the XOR problem using backpropagation is as follows:

1. Defining the Neural Network Structure

The network has two input neurons, four hidden neurons, and one output neuron.

• Weights_input_hidden: Connects input to the hidden layer
• Weights_hidden_output: Connects hidden to output layer
• Biases are initialized to zero to shift the activation when needed

import numpy as np
class XORNeuralNetwork:
def __init__(self, input_size, hidden_size, output_size):
self.input_size = input_size
self.hidden_size = hidden_size
self.output_size = output_size
self.weights_input_hidden = np.random.randn(self.input_size, self.hidden_size)
self.bias_hidden = np.zeros((1, self.hidden_size))
self.weights_hidden_output = np.random.randn(self.hidden_size, self.output_size)
self.bias_output = np.zeros((1, self.output_size))​

2. Activation Function and Its Derivative 

The sigmoid function compresses the input values into a range between 0 and 1, which is suitable for binary output, such as the XOR operation. The derivative of the sigmoid is used for gradient calculation during backpropagation.

def sigmoid(self, x):
return 1 / (1 + np.exp(-x))
def sigmoid_derivative(self, x):
return x * (1 - x)​

3. Feedforward Pass

In the input to the hidden layer, the dot product of the inputs, weights, and bias is passed through a sigmoid function. In the hidden layer, the dot product of the hidden output, weight, and bias is passed through the sigmoid. It stores outputs for use in backpropagation.

def feedforward(self, X):
self.hidden_input = np.dot(X, self.weights_input_hidden) + self.bias_hidden
self.hidden_output = self.sigmoid(self.hidden_input)
self.final_input = np.dot(self.hidden_output, self.weights_hidden_output) + self.bias_output
self.final_output = self.sigmoid(self.final_input)
return self.final_output​

4. Backward Propagation 

def backward(self, X, y, learning_rate):
error = y - self.final_output
d_output = error * self.sigmoid_derivative(self.final_output)
error_hidden = d_output.dot(self.weights_hidden_output.T)
d_hidden = error_hidden * self.sigmoid_derivative(self.hidden_output)
self.weights_hidden_output += self.hidden_output.T.dot(d_output) * learning_rate
self.bias_output += np.sum(d_output, axis=0, keepdims=True) * learning_rate
self.weights_input_hidden += X.T.dot(d_hidden) * learning_rate
self.bias_hidden += np.sum(d_hidden, axis=0, keepdims=True) * learning_rate

5. Training the Neural Network 

It performs forward and backward propagation in neural network for the specified epochs. It prints loss every 1000 epochs to track training progress and uses mean squared error as a loss metric. 

def train(self, X, y, epochs, learning_rate):
for epoch in range(epochs):
self.feedforward(X)
self.backward(X, y, learning_rate)
if epoch % 1000 == 0:
loss = np.mean(np.square(y - self.final_output))
print(f"Epoch {epoch}, Loss: {loss:.4f}")​

6. Training and Testing 

The training is for 10,000 epochs with a learning rate of 0.1. It prints predictions after training, where values should be near 0 or 1.

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])
nn = XORNeuralNetwork(input_size=2, hidden_size=4, output_size=1)
nn.train(X, y, epochs=10000, learning_rate=0.1)
print("Final Predictions:")
print(nn.feedforward(X))

Complexity: Time and Space

Time Complexity: Training the neural network is a time-consuming process. Time complexity analysis is necessary to understand the scope of scalability in training. It is described using Big O notation. Several factors, including the number of layers, the number of neurons per layer, the number of training samples, the number of epochs, and the complexity of the activation function, influence the time complexity of the model.

The time complexity of both the forward and backwards propagation is given by the following formula:

T(total) = E * S * T(sample) = O(E * S* L* N^2)

Here,

• E is the number of epochs
• S is the number of training samples 
• N is the number of compilations 
• L is the number of layers 

The formula indicates a linear increase in the training time for the number of epochs and samples, but a quadratic increase with the number of neurons per layer. 

Space Complexity: The space complexity in backpropagation in neural networks is based on the values required to be stored during the forward pass, gradients during the backward pass, and the network’s parameters. The following formula gives the space complexity:

Total space = O(Number of parameters + Number of activations during forward pass)

The space complexity can be optimized through methods such as active recomputation, gradient checkpointing, and parameter quantization when dealing with large parameter sets.

Activation Functions in Backward Propagation

The activation function in backward propagation is a mathematical function used to introduce non-linearity into the model, enabling complex decision-making by the neuron. It is responsible for deciding whether a neuron should be activated based on the weighted sum of inputs and a bias term. These calculations occur before the activation function is applied. The derivative of the activation function is used to compute gradients for updating weights. The activation function is of multiple types, such as:

• Linear activation function
• Non-linear activation function

1. Sigmoid function
2. Tanh activation function
3. ReLU function

• Exponential linear units

1. Softmax function 
2. Softplus function 

The activation functions play a crucial role in enhancing the model's training speed, ensuring adequate gradient flow and handling complex multi-class problems.

Join our 4.7 ⭐ rated program, trusted by over 3,800 learners who have successfully launched their careers as AI professionals. Start your learning journey with us today! 🎯

Advantages of Back Propagation for Neural Network Training

Back propagation offers the following advantages:

• Implementation of back propagation does not require prior or in-depth knowledge of neural networks
• Does not require much learning of features, input parameters, and other aspects 
• Backpropagation helps to generalize the models to new data, enhancing their accuracy
• Applicable across a variety of scenarios
• Offers scalability with large datasets and complex networks

Challenges With Backward Propagation

Backward propagation in neural networks is a fundamental technique that offers multiple benefits, but it also presents specific challenges. The hidden layers are required for calculating gradients and updating weights for backpropagation. The challenge can be posed with an imbalance in handling gradients, increased computational tasks, overfitting risk, and training instability. Insights into the same are necessary for better analysis:

1. Vanishing and Exploding Gradients

One of the challenges with gradients is their extreme values. They can be vanishingly small, which is referred to as the vanishing gradient problem, or explosively large, where the condition is referred to as the exploding gradient problem. It poses a problem in optimization. The challenge can be overcome with techniques such as architectural innovations, including skip connections and normalized activation functions (ReLU), as well as careful weight initialization.

2. Dead ReLUs

Rectified Linear Unit (ReLU) is also referred to as the rectifier activation function. It solves the vanishing gradient problem, but may also result in some neurons producing an output of zero for all inputs. It occurs when the weights consistently produce negative inputs, rendering them inactive (or 'dead') and halting learning. Lowering the learning rate and using a positive bias can help in handling this problem.

3. Regularization and Best Practices

Overfitting is another problem in backpropagation. It occurs when a neural network learns to ignore noise and outliers, resulting in poor generalization on unseen data. Regularization techniques, such as L1/L2 regularization and dropout regularization, help reduce overfitting by adding penalty terms to the loss function. It prevents noise learning.

Alternatives and the Future of Backpropagation

The computational cost and other limitations leave scope for exploring alternative options to backward propagation. There are different choices available as well, such as:

1. Equilibrium Propagation: It is a learning algorithm that uses the natural settling or equilibrium of neural network activations in response to inputs. It adjusts connections by nudging the network toward the target output using gradient ascent on this stable state.

2. Direct Feedback Alignment: It is a learning method where random fixed feedback weights guide weight updates instead of precise gradients. It relies on approximate error alignments for learning.

3. Different Target Propagation (DTP): It can handle deep or highly non-linear networks where gradient-based updates become inefficient. DTP computes gradients rather than target values for the layer to guide learning.

4. HSIC Bottleneck: It trains neural networks using an approximation of the information bottleneck rather than backpropagation. It increases dependency between hidden representations and outputs while minimising their dependency on inputs using the Hilbert-Schmidt Independence Criterion (HSIC).

5. Decoupled Neural Interfaces using Synthetic Gradients: It enables neural networks to learn and communicate in a decoupled, scalable way by generating synthetic gradients. It helps handle the long-term dependencies in recurrent neural networks without waiting for accurate gradient signals.

Not confident about your AI and ML skills? Join the Professional Certificate in AI and Machine Learning and master LLM, NLP, prompt engineering, generative AI, and machine learning algorithms in just 6 months! 🎯

Conclusion

Backward propagation is an efficient and widely used technique for training neural networks. They help learn complex patterns by iteratively refining their internal weights and biases. While offering scalability, flexibility, and other benefits, backpropagation still presents specific challenges. These involve vanishing and exploding gradients, dead neurons, and others.

However, insights into the technique help understand that these challenges can be overcome with suitable optimization methods. Additionally, alternative training methods exist for training neural networks. Gain overall information before selecting the optimal training method for the neural network.

Advance Your Career in AI and ML With Simplilearn

AI has entered almost every field, influencing the job duties and responsibilities of professionals. From easing tasks to speeding them up, even developers benefit from AI. However, the effective use of AI by professionals requires fundamental clarity and hands-on experience. It contributes to better decision-making and a deeper understanding of the processes being carried out.

Considering this, Simplilearn offers popular AI and ML courses from top universities and institutes, including Purdue University and IITs, among others. Besides the basic benefits, gain universally recognized certifications, Ask Me Anything sessions with experts, and much more. Explore the following courses:

• Professional Certificate in AI and Machine Learning
• Professional Certificate Program in Generative AI and Machine Learning
• Artificial Intelligence Engineer

FAQs

1. What is the main purpose of backpropagation?

Backpropagation is a technique used to train the neural network and enhance its performance. It helps neural networks improve decision-making. It does so by adjusting the weights and biases that influence the output.

2. Is backpropagation the same as gradient descent?

Backpropagation is the algorithm to determine the gradients of the cost function, while gradient descent is the optimization algorithm. The latter helps identify the weights capable of minimizing the cost function. The backpropagation requires differentiation through the chain rule, while gradient descent requires the gradient via backpropagation and a learning rate.

3. What is backpropagation through time?

Backpropagation Through Time (BPTT) is the variation of backpropagation. It is the extension of backpropagation used for training Recurrent Neural Networks (RNNs). It calculates gradients by backward error propagation through time steps, allowing the model to learn temporal dependencies.

4. What debugging techniques help trace gradient flow?

Some of the debugging techniques that help trace gradient flow are gradient checking, gradient histogram and visualizations, gradient clipping, gradient norm monitoring, and others.

5. Are there visualizations to explain error propagation?

Gradient flow charts, error surface visualization, computation graphs, and other visualizations aid in explaining error propagation.

6. How do you optimize memory during training?

Memory pinning, discarding, and recomputing activations, as well as using smaller data types, are among the techniques that help optimize memory during neural network training.