In this tutorial, you will learn how to efficiently use PyTorch, one of the most well-known packages for deep learning in Python.
Note that this library has been heavily inspired by NumPy, so be sure to study the related lesson before starting this one.
tensor
PyTorch’s core data structure is the tensor, that is, a multidimensional
array, corresponding to a mathematical tensor:
tensor1D ➡️ vector$\begin{pmatrix} 1 & 5 & 7 & 2 \end{pmatrix}$
tensor2D ➡️ matrix$\begin{pmatrix} 1 & 5 & 7 & 2 \ 4 & 3 & 2 & 0 \ 3 & 6 & 1 & 8 \end{pmatrix}$
tensor3D → tensor 3D …tensor4D → tensor 4D ……
A tensor is almost equivalent to a NumPy’s ndarray.
Let’s build our first tensor, containing three values. In order to do this I
have to pass a list of three numbers to the torch.tensor function.
import torch
t = torch.tensor([0, 1, 2])
t
tensor([0, 1, 2])
A tensor is characterized by three important metadata
shape➡️ the dimensions of thetensordtype➡️ the type of elements stored in thetensor(decimal, integer, etc.)device➡️ the memory device in which thetensorhas been allocated and will be used ("cpu"= RAM/CPU,"cuda"= VRAM/GPU). More on this later
print(f't shape: {t.shape}')
print(f't dtype: {t.dtype}')
print(f't device: {t.device}')
t shape: torch.Size([3])
t dtype: torch.int64
t device: cpu
(3,) means that the array has only one dimension, with three values.
int64 means that the data contain integer values, and each value is stored in
the memory in a binary format that requires 64 bits.
Let’s instantiate now a 2D tensor of size $3 \times 3$ containing integer
numbers. For two dimensions, the trick consists of using a list of lists.
b = torch.tensor([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
b
tensor([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
print(f'b shape: {b.shape}')
print(f'b dtype: {b.dtype}')
b shape: torch.Size([3, 3])
b dtype: torch.int64
let’s build a tensor with shape (3, 2, 2)
t = torch.tensor([[[1,2], [3,4]], [[5,6], [7,8]], [[9,10], [11,12]]])
print(t)
print(t.shape)
tensor([[[ 1, 2],
[ 3, 4]],
[[ 5, 6],
[ 7, 8]],
[[ 9, 10],
[11, 12]]])
torch.Size([3, 2, 2])
A tensor with shape (n,) behaves like one with shape (1, n) but it is
different from one with shape (n,1). In mathematical term they correspond to
vectors $u \in \mathbb{R}^n$, $v \in \mathbb{R}^{1 \times n}$ e
$w \in \mathbb{R}^{n \times 1}$
u = torch.rand(3)
v = torch.rand(1, 3)
w = torch.rand(3, 1)
print(f'u: {u}, of shape {u.shape}')
print(f'v: {v}, of shape {v.shape}')
print(f'w: {w}, of shape {w.shape}')
u: tensor([0.0775, 0.6302, 0.1858]), of shape torch.Size([3])
v: tensor([[0.0140, 0.9222, 0.9028]]), of shape torch.Size([1, 3])
w: tensor([[0.7973],
[0.4812],
[0.7615]]), of shape torch.Size([3, 1])
Create a tensor
In the previous example we have used a new function, torch.rand, that has
generated values that have not been explicitly written.
Writing by hand all values of a tensor might be a very long task, if not
impossible. For this reason, there exist some useful functions to create
standard tensors.
torch.zeros(shape): tensor with only 0
z = torch.zeros(3, 3, 3)
z
tensor([[[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]],
[[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]],
[[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]]])
torch.ones(shape): tensor with only 1
o = torch.ones(5)
o
tensor([1., 1., 1., 1., 1.])
torch.full(shape): tensor initialized with the same values specified by fill_value
f = torch.full((4, 3), fill_value=3.14)
f
tensor([[3.1400, 3.1400, 3.1400],
[3.1400, 3.1400, 3.1400],
[3.1400, 3.1400, 3.1400],
[3.1400, 3.1400, 3.1400]])
torch.empty(shape): tensor with uninitialized values
e = torch.empty(3, 4)
e
tensor([[-4.1769e+30, 3.0754e-41, 2.3694e-38, 1.8750e+00],
[ 0.0000e+00, 1.8750e+00, 0.0000e+00, 1.8750e+00],
[ 0.0000e+00, 1.8750e+00, 1.4838e-41, 0.0000e+00]])
torch.rand(shape): tensor with random values between 0 (included) and 1 (excluded)
rnd = torch.rand(2, 3)
rnd
tensor([[0.2704, 0.6800, 0.2617],
[0.5777, 0.1751, 0.1285]])
torch.arange(low, high): tensor with a sequence of integer between low
(included) and high (excluded)
rng = torch.arange(9) # dtype = torch.int
print(rng)
frng = torch.arange(1., 10.) # dtype = torch.float
print(frng)
tensor([0, 1, 2, 3, 4, 5, 6, 7, 8])
tensor([1., 2., 3., 4., 5., 6., 7., 8., 9.])
3. Operations on tensors
Change shape
the shape of a tensor can be modified through torch.reshape(shape) or
torch.view(shape). Their behavior is similar and they usually share the
memory with the original tensor. When this is not possible a new tensor is
created
rng = rng.reshape(3, 3)
print(rng.shape)
print(rng)
torch.Size([3, 3])
tensor([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
f = f.reshape(2, 2, -1) # automatically infer the second dimension
print(f.shape)
print(f)
torch.Size([2, 2, 3])
tensor([[[3.1400, 3.1400, 3.1400],
[3.1400, 3.1400, 3.1400]],
[[3.1400, 3.1400, 3.1400],
[3.1400, 3.1400, 3.1400]]])
To “stretch out” a tensor we can use the functions tensor.flatten,
tensor.ravel or tensor.view(-1), obtaining a 1D tensor with shape
(n,). flatten and ravel create a new tensor only if it is needed, while
view always shares the memory with the original tensor
Let’s try now some simple operation with tensor.
Sum
$$ \begin{pmatrix} 3 & 2 & 4 \\ 5 & 1 & 10 \\ 7 & 2 & 9 \end{pmatrix} + \begin{pmatrix} 1 & 4 & 7 \\ 12 & 5 & 0 \\ 3 & 2 & 1 \\ \end{pmatrix} = \begin{pmatrix} 4 & 6 & 11 \\ 17 & 6 & 10 \\ 10 & 4 & 10 \\ \end{pmatrix} $$
x = torch.tensor([[3., 2., 4.], [5., 1., 10.], [7., 2., 9.]])
y = torch.tensor([[1., 4., 7.], [12., 5., 0.], [3., 2., 1.]])
x + y
tensor([[ 4., 6., 11.],
[17., 6., 10.],
[10., 4., 10.]])
Hadamard Product
$$ \begin{pmatrix} 3 & 2 & 4 \\ 5 & 1 & 10 \\ 7 & 2 & 9 \end{pmatrix} \odot \begin{pmatrix} 1 & 4 & 7 \\ 12 & 5 & 0 \\ 3 & 2 & 1 \end{pmatrix} = \begin{pmatrix} 3 & 8 & 28 \\ 60 & 5 & 0 \\ 21 & 4 & 9 \end{pmatrix} $$
x * y
tensor([[ 3., 8., 28.],
[60., 5., 0.],
[21., 4., 9.]])
ar1 = torch.arange(3)
ar2 = torch.arange(4, 7)
print(ar1, ar2)
torch.dot(ar1, ar2)
tensor([0, 1, 2]) tensor([4, 5, 6])
tensor(17)
Scalar-vector product
$$ 3 \cdot \begin{pmatrix} 3 & 2 & 4 \\ 5 & 1 & 10 \\ 7 & 2 & 9 \end{pmatrix} = \begin{pmatrix} 9 & 6 & 12 \\ 15 & 3 & 30 \\ 21 & 6 & 27 \end{pmatrix} $$
k = 3
print(k * x)
tensor([[ 9., 6., 12.],
[15., 3., 30.],
[21., 6., 27.]])
Matrix Product
$$ \begin{pmatrix} 3 & 2 & 4 \\ 5 & 1 & 10 \\ 7 & 2 & 9 \end{pmatrix} \times \begin{pmatrix} 1 & 4 & 7 \\ 12 & 5 & 0 \\ 3 & 2 & 1 \end{pmatrix} = \begin{pmatrix} 39 & 30 & 25 \\ 47 & 45 & 45 \\ 58 & 56 & 58 \end{pmatrix} $$
x @ y # o torch.matmul(x, y) o torch.mm(x, y)
tensor([[39., 30., 25.],
[47., 45., 45.],
[58., 56., 58.]])
powers, exp, sin, cos, log, etc.
print(x ** 2) # squares of x
print(torch.exp(x))
print(torch.sin(y))
tensor([[ 9., 4., 16.],
[ 25., 1., 100.],
[ 49., 4., 81.]])
tensor([[2.0086e+01, 7.3891e+00, 5.4598e+01],
[1.4841e+02, 2.7183e+00, 2.2026e+04],
[1.0966e+03, 7.3891e+00, 8.1031e+03]])
tensor([[ 0.8415, -0.7568, 0.6570],
[-0.5366, -0.9589, 0.0000],
[ 0.1411, 0.9093, 0.8415]])
Scalar + matrix
z = torch.zeros((3,3))
z + 2
tensor([[2., 2., 2.],
[2., 2., 2.],
[2., 2., 2.]])
Scalar $\times$ matrix
z = torch.ones((3,3))
z * 2
tensor([[2., 2., 2.],
[2., 2., 2.],
[2., 2., 2.]])
Aggregation functions
We can also perform some aggregation function on an ndarray, such as the sum
or the mean
print(x.sum()) # sum all values
print(y.mean()) # compute the mean of all values
print(x.min(), x.max())
tensor(43.)
tensor(3.8889)
tensor(1.) tensor(10.)
And we can specify that the aggregations be performed only over a specified
dimension, using the dim parameter.
print(x.sum(dim=0)) # compute the sum of each column
print(y.mean(dim=1)) # compute the mean of each row
tensor([15., 5., 23.])
tensor([4.0000, 5.6667, 2.0000])
4. tensor Indexing
Sometimes it is necessary to access a subset of the values in a tensor. This
operation is called indexing. Following, some examples. Note that rows and
columns start from the index $0$:
$$ m = \begin{pmatrix} 0 & 1 & 2 \\ 3 & 4 & 5 \\ 6 & 7 & 8 \\ 9 & 10 & 11 \\ 12 & 13 & 14 \end{pmatrix} $$
m = torch.arange(15.).reshape(5, 3)
m
tensor([[ 0., 1., 2.],
[ 3., 4., 5.],
[ 6., 7., 8.],
[ 9., 10., 11.],
[12., 13., 14.]])
$$ m_{12} = \begin{pmatrix} 5 \end{pmatrix} $$
m[1, 2]
tensor(5.)
$$ m_{1*} = \begin{pmatrix} 3 & 4 & 5 \end{pmatrix} $$
m[1, :]
tensor([3., 4., 5.])
$$
m_{1:5, *} = \begin{pmatrix}
3 & 4 & 5 \\
6 & 7 & 8 \\
9 & 10 & 11 \\
12 & 13 & 14
\end{pmatrix}
$$
N.B. If the interval exceeds the number of rows/columns of the matrix, NumPy
returns still the maximum possible number of rows/columns. For example, the
row indices in $m$ are from $0$ to $4$. Using 1:6, the rows from $1$ to $4$
are selected
m[1:6, :]
tensor([[ 3., 4., 5.],
[ 6., 7., 8.],
[ 9., 10., 11.],
[12., 13., 14.]])
$$ m_{1:5, 2:} = \begin{pmatrix} 5 \\ 8 \\ 11 \\ 14 \end{pmatrix} $$
m[1:5, 2:]
tensor([[ 5.],
[ 8.],
[11.],
[14.]])
Fancy Indexing
$$ \begin{pmatrix} m_{10} & m_{12} & m_{31} \end{pmatrix} = \begin{pmatrix} 3 & 5 & 10 \end{pmatrix} $$
N.B In this case the returned tensor has always shape (n,)
m[[1, 1, 3], [0, 2, 1]]
tensor([ 3., 5., 10.])
With the boolean masks I can select the elements that satisfy a condition,
evaluating to True
$$ \begin{pmatrix} m_{00} & m_{12} & m_{20} & m_{21} & m_{30} & m_{31} & m_{32} \end{pmatrix} = \begin{pmatrix} 0 & 5 & 6 & 7 & 9 & 10 & 11 \end{pmatrix} $$
N.B. With PyTorch <= 1.2.0 the boolean type does not exist, torch.byte is
used instead
N.B. In this case too, the returned tensor has always shape (n,)
mask = torch.tensor([[True, False, False],
[False, False, True],
[True, True, False],
[True, True, True],
[False, False, False]])
m[mask]
m[m > 5]
tensor([ 6., 7., 8., 9., 10., 11., 12., 13., 14.])
In general, PyTorch follows NumPy’s indexing conventions
5. device
Differently from the NumPy ndarrays, in PyTorch the tensor can be also
allocated to the VRAM of the GPU to accelerate the execution. To use this
feature you must need an Nvidia GPU that is CUDA enabled
and must moreover install specific versions of the libcuda and libcudnn
libraries required by the version of PyTorch that you are using.
Let’s see the difference in the execution time between CPU and GPU
import time
r1 = torch.rand(1000, 1000) # 100 * 100 = 10000 random numbers in [0, 1)
r2 = torch.rand(1000, 4000)
print(f'r1 {r1.device}')
print(f'r2 {r2.device}')
r1 cpu
r2 cpu
start = time.time()
r1 @ r2
print(f'r1 @ r2 total time (CPU): {time.time() - start} sec')
r1 @ r2 total time (CPU): 0.1771993637084961 sec
Now, let’s see the difference with the GPU. First of all, let’s move the
tensors in the correct device.
N.B. At the first computation in GPU, PyTorch has a computational overhead due to the initialization of the CUDA runtime API
r1 = r1.to('cuda') # or cuda:n to indicate the device to use (default 0)
r2 = r2.to('cuda')
print(f'r1 {r1.device}')
print(f'r2 {r2.device}')
r1 cuda:0
r2 cuda:0
start = time.time()
r1 @ r2
print(f'r1 @ r2 total time (GPU): {time.time() - start} sec')
r1 @ r2 total time (GPU): 0.06495833396911621 sec
We can also allocate the tensor directly to the GPU, by specifying the
device parameter during the initialization
r3 = torch.rand(1000, 1000, device='cuda')
print(f'r3 {r3.device}')
r3 cuda:0
6. Gradient
PyTorch is a library for Deep Learning, therefore, it is important that it
computes efficiently the derivatives of the operations for learning by
optimization, without the need of manually writing them.
To this end, PyTorch uses the autograd differentiation engine that computes
the derivative using the Automatic Differentiation
Let’s see how to compute the derivative of $y = x_1^2$ with $x_1 = 3$ that is:
$$ \left.\frac{\partial y}{\partial x_1}\right|_{x_1 = 3} = 2 \cdot x_1|_{x_1 = 3} = 2 \cdot 3 = 6 $$
x1 = torch.tensor(3., requires_grad=True)
print(x1)
print(f'grad before backward: {x1.grad}')
y = x1 ** 2.
print(y)
y.backward()
print(f'grad after backward: {x1.grad}')
tensor(3., requires_grad=True)
grad before backward: None
tensor(9., grad_fn=<PowBackward0>)
grad after backward: 6.0
The previous example shows three things:
- A
tensortakes care of saving the output of its derivative - At the moment of its creation, I set the parameter
requires_gradtoTrue3. The computation of the derivatives is not automatic and must be started by the user, by calling thebackward()function.
To check if a tensor requires the gradient, I can control the attribute
tensor.requires_grad.
To modify this parameter, I can call the tensor.requires_grad_() function
(with the trailing underscore) that sets to True the requires_grad
parameter (but never sets it to False)
N.B. Be careful to not confuse them!
x2 = torch.tensor(4.) # requires_grad è a False
print(f'1. x2 requires the gradient? {x2.requires_grad}')
x2.requires_grad_() # setta a True
print(f'2. x2 requires the gradient? {x2.requires_grad}')
x2.requires_grad_() # non cambia nulla
print(f'3. x2 requires the gradient? {x2.requires_grad}')
1. x2 requires the gradient? False
2. x2 requires the gradient? True
3. x2 requires the gradient? True
There are other methods to block the computation of the gradient, for example
by manually setting requires_grad, or by using the torch.no_grad() context
manager
x3 = torch.tensor(5., requires_grad=True)
with torch.no_grad():
print(f'x3 requires grad? {x3.requires_grad}')
y = 4. * x3
y.backward() # Error!
x3 requires grad? False
<Error message>
In general, managing the graident of tensors might be complex. There exist
some debugging functions that help in checking the gradients!