Performing backpropagation manually

Performing backpropagation manually in PyTorch involves computing gradients of a loss function with respect to model parameters using the chain rule. Let’s go through the steps to perform backpropagation manually:

1. Define the Model

First, we define a simple neural network model. Here’s an example:

import torch
import torch.nn as nn

class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(10, 5)
        self.fc2 = nn.Linear(5, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# Instantiate the model
model = SimpleNN()

2. Define the Loss Function

Next, we define a loss function. We’ll use Mean Squared Error (MSE) loss in this example:

criterion = nn.MSELoss()

3. Perform Forward Pass

Now, we perform a forward pass through the model to compute predictions and the corresponding loss:

# Example input and target
input_data = torch.randn(1, 10)
target = torch.randn(1, 1)

# Forward pass
output = model(input_data)
loss = criterion(output, target)

4. Perform Backward Pass Manually

After computing the loss, we manually perform the backward pass to compute gradients of the loss with respect to model parameters:

# Zero the gradients
model.zero_grad()

# Backward pass
loss.backward()

# Access gradients
for param in model.parameters():
    print(param.grad)

Conclusion

Performing backpropagation manually in PyTorch involves computing gradients of the loss function with respect to model parameters using the chain rule. By understanding the steps involved in manual backpropagation, we gain insight into how PyTorch handles gradient computation automatically during training. Manual backpropagation can be useful for debugging and understanding the inner workings of neural network training algorithms.