Introduction to Tensors

Tensors are the fundamental data structures used in PyTorch for representing and performing operations on multidimensional arrays. Understanding tensors is crucial for working effectively with PyTorch and building neural networks. In this section, we’ll delve into the basics of tensors, including their creation, manipulation, and common operations.

What are Tensors?

A tensor is a generalization of scalars, vectors, and matrices to higher dimensions. In PyTorch, tensors can have arbitrary shapes and sizes, making them versatile for representing various types of data, including images, text, and numerical data.

Creating Tensors

Let’s start by creating tensors in PyTorch. There are several ways to create tensors:

1. Directly from Data

import torch

# Create a tensor from a Python list
tensor_list = torch.tensor([1, 2, 3, 4, 5])
print("Tensor from a Python list:")
print(tensor_list)

# Create a tensor from a NumPy array
import numpy as np
numpy_array = np.array([[1, 2, 3], [4, 5, 6]])
tensor_numpy = torch.tensor(numpy_array)
print("\nTensor from a NumPy array:")
print(tensor_numpy)

Tensor from a Python list:
tensor([1, 2, 3, 4, 5])

Tensor from a NumPy array:
tensor([[1, 2, 3],
        [4, 5, 6]], dtype=torch.int32)

2. Using Built-in Functions

# Create a tensor of zeros with a specific shape
tensor_zeros = torch.zeros(2, 3)
print("\nTensor of zeros:")
print(tensor_zeros)

# Create a tensor of ones with a specific shape
tensor_ones = torch.ones(3, 2)
print("\nTensor of ones:")
print(tensor_ones)

# Create a tensor with random values from a uniform distribution
tensor_uniform = torch.rand(3, 3)
print("\nTensor with random values (uniform distribution):")
print(tensor_uniform)

# Create a tensor with random values from a normal distribution
tensor_normal = torch.randn(2, 2)
print("\nTensor with random values (normal distribution):")
print(tensor_normal)

Tensor of zeros:
tensor([[0., 0., 0.],
        [0., 0., 0.]])

Tensor of ones:
tensor([[1., 1.],
        [1., 1.],
        [1., 1.]])

Tensor with random values (uniform distribution):
tensor([[0.2279, 0.9568, 0.7859],
        [0.2056, 0.5483, 0.7812],
        [0.6688, 0.4346, 0.4980]])

Tensor with random values (normal distribution):
tensor([[-0.8922,  0.2279],
        [-1.2808,  0.7052]])

3. Using Specific Data Types

# Create a tensor of ones with a specific data type
tensor_float = torch.ones(2, 2, dtype=torch.float32)
print("\nTensor of ones with float32 data type:")
print(tensor_float)

# Create a tensor of zeros with a specific data type
tensor_long = torch.zeros(3, 3, dtype=torch.int64)
print("\nTensor of zeros with int64 data type:")
print(tensor_long)

Tensor of ones with float32 data type:
tensor([[1., 1.],
        [1., 1.]])

Tensor of zeros with int64 data type:
tensor([[0, 0, 0],
        [0, 0, 0],
        [0, 0, 0]])

Tensor Operations

PyTorch provides a wide range of operations for manipulating tensors, similar to NumPy. Here are some common tensor operations:

1. Element-wise Operations

# Element-wise addition
tensor_a = torch.tensor([[1, 2], [3, 4]])
tensor_b = torch.tensor([[5, 6], [7, 8]])
tensor_sum = tensor_a + tensor_b
print("\nElement-wise addition:")
print(tensor_sum)

# Element-wise multiplication
tensor_product = tensor_a * tensor_b
print("\nElement-wise multiplication:")
print(tensor_product)

Element-wise addition:
tensor([[ 6,  8],
        [10, 12]])

Element-wise multiplication:
tensor([[ 5, 12],
        [21, 32]])

2. Reshaping Tensors

# Reshape a tensor
tensor_original = torch.tensor([[1, 2, 3], [4, 5, 6]])
tensor_reshaped = tensor_original.view(3, 2)
print("\nOriginal tensor:")
print(tensor_original)
print("\nReshaped tensor:")
print(tensor_reshaped)

Original tensor:
tensor([[1, 2, 3],
        [4, 5, 6]])

Reshaped tensor:
tensor([[1, 2],
        [3, 4],
        [5, 6]])

3. Reduction Operations

tensor_original = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.float)
# Compute the sum of all elements in a tensor
tensor_sum_all = torch.sum(tensor_original)
print("\nSum of all elements:")
print(tensor_sum_all)

# Compute the mean of all elements in a tensor
tensor_mean = torch.mean(tensor_original)
print("\nMean of all elements:")
print(tensor_mean)

Sum of all elements:
tensor(21.)

Mean of all elements:
tensor(3.5000)

4. Matrix Operations

# Matrix multiplication
tensor_mat1 = torch.tensor([[1, 2], [3, 4]])
tensor_mat2 = torch.tensor([[5, 6], [7, 8]])
tensor_mat_product = torch.matmul(tensor_mat1, tensor_mat2)
print("\nMatrix multiplication:")
print(tensor_mat_product)

# Transpose a matrix
tensor_transpose = torch.transpose(tensor_original, 0, 1)
print("\nTransposed matrix:")
print(tensor_transpose)

Matrix multiplication:
tensor([[19, 22],
        [43, 50]])

Transposed matrix:
tensor([[1., 4.],
        [2., 5.],
        [3., 6.]])

Broadcasting

Broadcasting is a powerful feature in PyTorch that allows tensors with different shapes to be combined in element-wise operations. It automatically expands the dimensions of smaller tensors to match the shape of larger tensors.

# Broadcasting example
tensor_a = torch.tensor([[1, 2], [3, 4]])
scalar = 2
tensor_broadcasted = tensor_a + scalar
print("\nBroadcasting a scalar to a tensor:")
print(tensor_broadcasted)

Broadcasting a scalar to a tensor:
tensor([[3, 4],
        [5, 6]])

Indexing and Slicing

Similar to NumPy arrays, tensors support indexing and slicing operations to access specific elements or sub-tensors.

# Indexing and slicing example
tensor_a = torch.tensor([[1, 2, 3], [4, 5, 6]])
print("\nElement at row 1, column 2:")
print(tensor_a[0, 1])

print("\nFirst row:")
print(tensor_a[0, :])

print("\nLast column:")
print(tensor_a[:, -1])

Element at row 1, column 2:
tensor(2)

First row:
tensor([1, 2, 3])

Last column:
tensor([3, 6])

Moving Tensors to GPU

PyTorch allows tensors to be moved between CPU and GPU for acceleration using CUDA.

# Moving tensors to GPU
if torch.cuda.is_available():
    device = torch.device("cuda")
    tensor_gpu = tensor_a.to(device)
    print("\nTensor moved to GPU:")
    print(tensor_gpu)
else:
    print("\nCUDA is not available. Cannot move tensor to GPU.")

Tensor moved to GPU:
tensor([[1, 2, 3],
        [4, 5, 6]], device='cuda:0')

Tensor Concatenation

Concatenation allows tensors to be combined along specified dimensions.

# Tensor concatenation example
tensor_a = torch.tensor([[1, 2], [3, 4]])
tensor_b = torch.tensor([[5, 6], [7, 8]])
tensor_concatenated = torch.cat((tensor_a, tensor_b), dim=1)
print("\nConcatenated tensor along columns:")
print(tensor_concatenated)

Concatenated tensor along columns:
tensor([[1, 2, 5, 6],
        [3, 4, 7, 8]])

Tensor Comparison

PyTorch supports element-wise comparison operations between tensors.

# Tensor comparison example
tensor_a = torch.tensor([[1, 2], [3, 4]])
tensor_b = torch.tensor([[1, 3], [3, 5]])
tensor_comparison = tensor_a > tensor_b
print("\nComparison result (element-wise):")
print(tensor_comparison)

Comparison result (element-wise):
tensor([[False, False],
        [False, False]])

Conclusion

Tensors are the backbone of PyTorch and are used extensively in deep learning applications. In this section, we covered the basics of tensors, including their creation, manipulation, and common operations, including broadcasting, indexing, moving tensors to GPU, concatenation, and comparison operations. Understanding tensors is essential for effectively working with PyTorch and building complex neural networks. Experiment with different tensor operations and explore their applications in your deep learning projects.