Chapter 2

Linear Algebra

Linear algebra is a branch of mathematics that is widely used throughout science

and engineering. However, because linear algebra is a form of continuous rather

than discrete mathematics, many computer scientists have little experience with it.

A good understanding of linear algebra is essential for understanding and working

with many machine learning algorithms, especially deep learning algorithms. We

therefore precede our introduction to deep learning with a focused presentation of

the key linear algebra prerequisites.

If you are already familiar with linear algebra, feel free to skip this chapter. If

you have previous experience with these concepts but need a detailed reference

sheet to review key formulas, we recommend The Matrix Cookbook (Petersen and

Pedersen, 2006). If you have no exposure at all to linear algebra, this chapter

will teach you enough to read this book, but we highly recommend that you also

consult another resource focused exclusively on teaching linear algebra, such as

Shilov (1977). This chapter will completely omit many important linear algebra

topics that are not essential for understanding deep learning.

2.1 Scalars, Vectors, Matrices and Tensors

The study of linear algebra involves several types of mathematical objects:

• Scalars

: A scalar is just a single number, in contrast to most of the other

objects studied in linear algebra, which are usually arrays of multiple numbers.

We write scalars in italics. We usually give scalars lower-case variable names.

When we introduce them, we specify what kind of number they are. For

31

CHAPTER 2. LINEAR ALGEBRA

example, we might say “Let

s ∈ R

be the slope of the line,” while deﬁning a

real-valued scalar, or “Let

n ∈ N

be the number of units,” while deﬁning a

natural number scalar.

• Vectors

: A vector is an array of numbers. The numbers are arranged in

order. We can identify each individual number by its index in that ordering.

Typically we give vectors lower case names written in bold typeface, such

as

x

. The elements of the vector are identiﬁed by writing its name in italic

typeface, with a subscript. The ﬁrst element of

x

is

x

1

, the second element

is

x

2

and so on. We also need to say what kind of numbers are stored in

the vector. If each element is in

R

, and the vector has

n

elements, then the

vector lies in the set formed by taking the Cartesian product of

R n

times,

denoted as

R

n

. When we need to explicitly identify the elements of a vector,

we write them as a column enclosed in square brackets:

x =

x

1

x

2

.

.

.

x

n

. (2.1)

We can think of vectors as identifying points in space, with each element

giving the coordinate along a diﬀerent axis.

Sometimes we need to index a set of elements of a vector. In this case, we

deﬁne a set containing the indices and write the set as a subscript. For

example, to access

x

1

,

x

3

and

x

6

, we deﬁne the set

S

=

{

1

,

3

,

6

}

and write

x

S

. We use the

−

sign to index the complement of a set. For example

x

−1

is

the vector containing all elements of

x

except for

x

1

, and

x

−S

is the vector

containing all of the elements of x except for x

1

, x

3

and x

6

.

• Matrices

: A matrix is a 2-D array of numbers, so each element is identiﬁed

by two indices instead of just one. We usually give matrices upper-case

variable names with bold typeface, such as

A

. If a real-valued matrix

A

has

a height of

m

and a width of

n

, then we say that

A ∈ R

m×n

. We usually

identify the elements of a matrix using its name in italic but not bold font,

and the indices are listed with separating commas. For example,

A

1,1

is the

upper left entry of

A

and

A

m,n

is the bottom right entry. We can identify all

of the numbers with vertical coordinate

i

by writing a “:” for the horizontal

coordinate. For example,

A

i,:

denotes the horizontal cross section of

A

with

vertical coordinate

i

. This is known as the

i

-th

row

of

A

. Likewise,

A

:,i

is

32

CHAPTER 2. LINEAR ALGEBRA

A =

A

1,1

A

1,2

A

2,1

A

2,2

A

3,1

A

3,2

⇒ A

=

A

1,1

A

2,1

A

3,1

A

1,2

A

2,2

A

3,2

Figure 2.1: The transpose of the matrix can be thought of as a mirror image across the

main diagonal.

the i-th column of A. When we need to explicitly identify the elements of

a matrix, we write them as an array enclosed in square brackets:

A

1,1

A

1,2

A

2,1

A

2,2

. (2.2)

Sometimes we may need to index matrix-valued expressions that are not just

a single letter. In this case, we use subscripts after the expression, but do

not convert anything to lower case. For example,

f

(

A

)

i,j

gives element (

i, j

)

of the matrix computed by applying the function f to A.

• Tensors

: In some cases we will need an array with more than two axes.

In the general case, an array of numbers arranged on a regular grid with a

variable number of axes is known as a tensor. We denote a tensor named “A”

with this typeface:

A

. We identify the element of

A

at coordinates (

i, j, k

)

by writing A

i,j,k

.

One important operation on matrices is the

transpose

. The transpose of a

matrix is the mirror image of the matrix across a diagonal line, called the

main

diagonal

, running down and to the right, starting from its upper left corner. See

ﬁgure 2.1 for a graphical depiction of this operation. We denote the transpose of a

matrix A as A

, and it is deﬁned such that

(A

)

i,j

= A

j,i

. (2.3)

Vectors can be thought of as matrices that contain only one column. The

transpose of a vector is therefore a matrix with only one row. Sometimes we

33

CHAPTER 2. LINEAR ALGEBRA

deﬁne a vector by writing out its elements in the text inline as a row matrix,

then using the transpose operator to turn it into a standard column vector, e.g.,

x = [x

1

, x

2

, x

3

]

.

A scalar can be thought of as a matrix with only a single entry. From this, we

can see that a scalar is its own transpose: a = a

.

We can add matrices to each other, as long as they have the same shape, just

by adding their corresponding elements: C = A + B where C

i,j

= A

i,j

+ B

i,j

.

We can also add a scalar to a matrix or multiply a matrix by a scalar, just

by performing that operation on each element of a matrix:

D

=

a · B

+

c

where

D

i,j

= a · B

i,j

+ c.

In the context of deep learning, we also use some less conventional notation.

We allow the addition of matrix and a vector, yielding another matrix:

C

=

A

+

b

,

where

C

i,j

=

A

i,j

+

b

j

. In other words, the vector

b

is added to each row of the

matrix. This shorthand eliminates the need to deﬁne a matrix with

b

copied into

each row before doing the addition. This implicit copying of

b

to many locations

is called broadcasting.

2.2 Multiplying Matrices and Vectors

One of the most important operations involving matrices is multiplication of two

matrices. The

matrix product

of matrices

A

and

B

is a third matrix

C

. In

order for this product to be deﬁned,

A

must have the same number of columns as

B

has rows. If

A

is of shape

m × n

and

B

is of shape

n × p

, then

C

is of shape

m × p

. We can write the matrix product just by placing two or more matrices

together, e.g.

C = AB. (2.4)

The product operation is deﬁned by

C

i,j

=

k

A

i,k

B

k,j

. (2.5)

Note that the standard product of two matrices is not just a matrix containing

the product of the individual elements. Such an operation exists and is called the

element-wise product or Hadamard product, and is denoted as A B.

The

dot product

between two vectors

x

and

y

of the same dimensionality

is the matrix product

x

y

. We can think of the matrix product

C

=

AB

as

computing C

i,j

as the dot product between row i of A and column j of B.

34

CHAPTER 2. LINEAR ALGEBRA

Matrix product operations have many useful properties that make mathematical

analysis of matrices more convenient. For example, matrix multiplication is

distributive:

A(B + C) = AB + AC. (2.6)

It is also associative:

A(BC) = (AB)C. (2.7)

Matrix multiplicat