Zac Blanco     Blog     Education     Projects     About

Intro to Linear Algebra


Chapter 1. Matrices, Vectors, and Systems of Linear Equations

Matrix: is a rectangular array of scalar quantities

We can see here that rows are numbered from 1 to . and that we have columns from 1 to .

Normally we say that a matric will be in size where m denotes rows and n denotes colummns.

That is:

  • m is rows
  • n is columns

Matrices provide “tools” for calculations

There is a short notation for a matrix which looks like the following

but what if we have the expression:

This means that the matrices are equal at values in each matrix

Matrix Operations

Addition

Given and that

Then when adding two matrices, they must be the same size. As long as they are the same size. You simple add the elements which are in the same position of each matrix,to form the new value in that location in the resultant matrix

So for addition in short notation

Scalar Multiplication

In scalar multiplication of a matrix you simply just multiply a scalar quantity by every entry of a matrix. In short notation this would be:

Example:

and

Then is equal to

Properties of Matrix Addition and Scalar Multiplication

Vectors

Vectors are simply one-dimensional matrices. All of the properties of matrices apply to vectors as well.

Example:

or

Geometry of Vectors

Given we can say that we can graph this vector on a 2D coordinate system.

But what if there are 3? or more? So our vector might be well then we can graph on an larger coordinate system with more dimensions. So 3 rows in a matrix might lead us to 3-dimensional coordinates and a 4th to 4, and so on…

Linear Combinations - Matrix-Vector Products and Special Matrices

Let be vectors. Let be scalars

We call . This is called a linear combination

In other words, it is the sum of all of our vectors , each with their own respective coefficient.

Matrix Multiplication

Matrices can be made of of m rows and n columns.

That is to generalize for a Matrix A and B with sizes and we have that product matrix of size where to be able to multiply

But how do we solve a system of equations using these matrices and the knowledge of linear combinations?

The answer is to utilize the Reduced Row Echelon Form or (RREF) for short.

Generally for a linear system we say that given a Matrix of coefficients A, a matrix of uknowns, , and what the product of those two are equal to, we get the equation

Transpose of a Matrix

A matrix transpose is a special operation that we can perform on a matrix. It will turn all of the rows of a matrix into columns

Example: Given the matrix

Then

Transposing also has a few special properties

Solving Systems of Linear Equations

Now that we’ve gone over the basics of Matrices we can actually begin to utilize them to solve systems of linear equations.

The basics of a system of linear equations:

In a system there are three possible different outcomes:

  1. No Solution
  2. Infinite Solutions
  3. A Single Solution

These outcomes depend on the system and what we are given.

If we recall the equation of A system for matrixes

We can represent our system with something called an Augmented Matrix where we display the right hand side of each equation in a matrix with the coefficients.

Example:

Augmented Matrix:

Once we have an augmented matrix we can turn it into the Reduced Row Echelon Form (RREF)

We do this by multiplying the first row by a constant , which when added to the 2nd row, will allow us to cancel out the value (sum to 0) the number immediately below the first number in the first row.

i.e. our first would be for the 2nd and 3rd rows

So then our matrix becomes:

This is reduced row echelon form because we see that as we go down rows, all numbers below the first number in each row are zero.

From that matrix we can translate it back into equations which will allow us to solve the system. From the RREF matrix we find that:

From here we then solve our final equation giving us:

In order to obtain these RREF matrices we can perform what are called Elementary Operations on each matrix.

Elementary Operations on a Matrix

  1. Exchange the placement of two rows
  2. Multiply a whole row with a non-zero scalar value
  3. Add a multiple of one row to another.

All three of these operations can help simplify a matrix, but will not change the outcome or solution for a system.

Conditions for an RREF Matrix

There are actually two conditions which need to be met for a matrix to be considered to be in RREF form. They are:

  1. The first non-zero number in each row must be equal to 1.
  2. All numbers which are below and to the left of the first number in each row must be equal to zero.

Gaussian Elimination

Gaussian elimination is the name method which we can use to reduce a matrix into RREF (Row-Reduced-Echelon-Form)

This basically consists of using the elementary row operations (noted above) to bring the matrix to a form which falls under the conditions for RREF stated above.

Note on solving linear systems with augmented matrices and RREF

Whenever an augmented matric contains a row in which the only nonzero entry lies in the last column, the corresponding system of linear equations has no solution.

Procedure for solving a System of Linear Equations

  1. Write the augmented matric of the system
  2. Find the reduced row echelon form
  3. if contains a row in which the only nonzero entry lies in the last column, the has no solution.
    • Otherwise, the system has at least one solution. Write the system of linear equations corresponding to the matrix and solve this system for the basic variables in terms of the free variables to obtain a general solution of

Rank and Nullity of a Matrix

Definition: Rank: the rank of an matrix A, denoted by is defined to be the number of nonzero rows in the reduced row echelon form of A

Definition: Nullity: The nullity of A, denoted by is defined to be , or we can say that , where is the number of columns of the matrix A.

If Ax = b is the matrix form of a consistent system of linear equations, then

  1. The number of basic variables in a general solution of the system equals the rank of A
  2. The number of free variables in a general solution of the system equals the nullity of A

Thus a consistent system of linear equations has a unique solution only if the nullity of its coefficient matrix equals 0.

Equivalently we can say that a consistent system has infinite solution if the nullity of its coefficient matrix is positive.

In shorter terms:

Nullity(A) Number of solutions of the System with matrix A
1

Testing for Consistency of a Matrix

The following statements are all equivalent

  1. The matrix equation is consistent
  2. The vector is a linear combination of the columns of A.
  3. The reduced row echelon form of the augmented matrix has no row of the form where

The Span of a Set of Vectors

Definition: Span: For a nonempty set of vectors, in the space of , we define the span of S to be the set of all linear combinations of in . This set is denoted by Span S or Span {u_1, u_2,\dots u_k}

Generating Sets

Imagine instead of trying to find the Span of a set we have a set of vectors and we want to find the span which generates the set . We then call the generating set of or that generates

Theorem - Generating Sets

The following statements are all equivalent:

  1. The span of the columns of A is
  2. The equation
  3. The rank of A is , the number of rows of A
  4. The reduced row echelon form of A has no zero rows
  5. There is a pivot position in each row of A

Making Smaller Generating Sets

Let be a set of vectors from and let be a vector in . Then the Span if and only if v belongs to the span of

Linear Dependence and Independence

A set of vectors in is called linearly dependent if there exists a set of scalars such that

and that

In other words, if the only set of scalars which satisfy the equation are scalars which are all equal to 0, then the set is linearly independent

The set is linearly dependent if an only if there exists a nonzero solution of where

The following statements involving linear independence about a matrix are equivalent:

  1. The columns of A are linearly independent
  2. the equation has at most one solution for each in
  3. The nullity of A is zero
  4. The rank of A is , the number of columns of A
  5. The columns of the RREF of A are distinct standard vectors in
  6. The only solution of zero
  7. There is a pivot position in each column of A

Matrix Multiplication

Multiplication of two matrices is different than normal multiplication by a scalar.

Definition Let A be an matrix and B be an matrix. We dfine the matrix product of to be the matrix who’s jth column is . That is

Another way of thinking is that the product matrix is equivalent to doing the dot product for all combinations of rows of matrix with all the columns of

The product matrix will have a size of

Properties of Matrix Multiplication

  • The product of any matrix and a zero matrix is zero

Invertibility and Elementary Matrices

Definition an matrix A is called invertible if there exists an matrix B such that . In this case B is called the inverse of A.

If A is an invertible matrix, then for every in has the unique solution

Useful Properties of Matrix Inverses

  1. If A is invertible, then is invertible and
  2. If A and B are invertible, then AB is invertible and
  3. If A is invertible, then is invertible and

Let be invertible matrices. Then the product is invertible and

Chapter 3 - Determinants of a Matrix

Given the matrix and the inverse of A, .

Now if we multiply these two together we get

Thus if then we know that it is possible

So then the matrix which is

So then if we then know it is possible for matrix to have an inverse. The quantity is called the determinant.

So then what about an matrix?

First we need to define cofactor expansion which is the the basic method which can be used to find the determinant of a larger matrix.

Given

We can label the rows and columns with alternating pluses and minuses. So if we replace the first row and column with pluses and minuses we get

So to find the determinant via cofactor expansion we must use the first row of the matrix and the alternating pluses and minuses to find the determinant.

We can define the determinant of the matrix as

We can define the matrices , , etc.. as the matrix which appears when we remove the i-th row and j-th column

So the determinant in this case is actually recursively defined. It is actually composed of even more operations

We find the determinant of A is

We can see that once we reduce the matrix to a matrix we will eventually get the scalar value.

Obviously this cofactor expansion is very tedious even for a so we will explore an even easier way to calculate a larger determinant.

So what is an easier method for this? We can simply find the upper triangular or lower triangular matrix from the original using elementary row operations

Take the following matrix:

Say we want to find the determinant of this matrix, we could use cofactor expansion to find it, but that would take us a long amount of time.

We’re going to use the following elementary row operations

Which then gives the matrix

Then we can use these row operations to reduce further

Then we can swap the 3rd and 4th rows of the matrix to make it the upper triangular form.

So now we have the upper triangular matrix. We can simply multiply all of the diagonal entries of the matrix together to find the determinant

So

So our determinant is

Wait wait wait, hold up! there’s an extra in the front of that product!

Correct! we had to change the sign of the value we found because in one of our elementary row operations, we switched the two rows, instead of just adding. When calculating the determinant you have to multiply the product of all the diagonal entries by

So in this case we multiplied the determinant by which makes the determinant which is the correct answer.

Geometric Applications of The Determinant

Area of a Parallelogram

Given two vectors and in which create a parallelogram, the area of the parallelogram is the absolute value of the determinant of

So given a two vectors and the area created by the parallelogram defined by these vectors is equal to

Volume of a Parallelepiped

Given three vectors which create a parallelepiped we can find the volume of such a figure by taking the absolute value of the determinant of the matrix created by the three vectors which is

Example:

Given , and

Then the matrix becomes

So reducing with row operations we get

and then

So then if we calculate the determinant we get , and the volume is the absolute value of the determinant, so the volume of the figure is

Rules of Calculating the determinant of a Matrix

Let A be an matrix

  1. if B is a matrix obtained by interchanging two rows of A, then
  2. If B is a matrix obtained by multiplying each entry of some row of A by a scalar k, then
  3. If B is a matrix obtained by adding a multiple of some row of A to a different row, then
  4. For any matrix then

Below are a few properties of the determinant:

  • A is an invertible matrix is
  • If A is an invertible matrix, then

Chapter 4 - Subspaces and Their Properties

Suppose we have an sized matrix, A which has solutions to

Then we can say that

Subspaces

Definition: A set of vectors in is called a subspace of if it has the following three properties

  1. The zero vector belongs to W
  2. Whenever any two vectors and belong to W, then must also belong to W
  3. Whenever belongs to W and c is a scalar, then belongs to W

Subspaces Associated with Matrices

There are many different types of subspaces associated with a matrix.

Null Space

The null space of a matrix is the solution set of . This subspace is denoted by

from an matrix A, the null space of A is the set

Column Space

The column space of a matrix A is the span of its columns. It is denoted by

For example, given the matrix

Then the

Row Space

The row space of a matrix is defined to be the span of its rows. The row space of a matrix, A, is denoted by So then given the matrix

The row space is then

Basis and Dimension

Let be a nonzero subspace of . Then a basis for is a linearly independent generating set for .

Or in other words, none of the vectors in the set are allowed to be a linear combination of one another.

To find the basis for , we reduce the matrix A into its reduced echelon form. The columns which have pivots are the columns from the original matrix which we then put into the basis (or minimal generating set) for .

This method can then be generalized for any set of vectors, which span a set where we can find the basis.

Let S be a finite subset of . Then we have the following properties.

  • If S is a generating set for then S contains at least vectors
  • If S is linearly independent then S contains at most n vectors
  • If S is a basis for , then S contains exactly n vectors.

A basis is a linearly independent subset of a subspace that is as large as possible

Another way to describe a basis is that it is a minimal set of generators for a specific space, whether is be a matrix, , or

Chapter 5 - Eigenvalues, Eigenvectors, and Diagonalization

This section explores different representations of matrices which are diagonalizable. We first need to explore eigenvalues and eigenvectors before we can understand how to diagonalize a matrix.

Definition: Let T be a linear operator on . A nonzero vector, v, in is called an eigenvector of T is is a multiple of . That is, for some scalar . The scalar value is called the eigenvalue of which corresponds to

In other words, v is an eigenvector if it satisfies the equation . Where is a scalar value called an eigenvalue of the matrix which corresponds to

How to determine if a vector is an eigenvector of a matrix

Given and

How do we tell if is an eigenvector of A?

We can simply do the multiplication first, to find the resultant vector

We can see from this result that we can factor out a 4 from the resultant matrix

That is

We see the remaining matrix is actually , so then we determine that is indeed an eigenvector. Also note that this eigenvector corresponds to the eigenvalue, 4

Finding the Eigenvectors of a Matrix

We can find the eigenvalues for a given matrix by the following:

From (4) we can find the eigenvectors of a matrix by simply solving that equation for any eigenvalue, , and all of the solutions from that equation will be the result of that solution

Finding the Eigenvalues of a Matrix

So then how can we just find the eigenvalues of a matrix?

We can find the characteristic equation of the matrix to get the eigenvalues which correspond to it. The equation for the matrix is simply the solutions for of:

Sometimes the characteristic equation is also referred to as the characteristic polynomial

For example, let’s find the eigenvalues of

Given the equation

Now we learned how to find the determinant of this matrix before. It is equal to

So then the equation becomes

This then gives us the solution of

Thus our eigenvalues

Here’s another quick definition

Let be an eigenvalue of a matrix, A. The dimension of the eigenspace of A corresponding to is less than or equal to the multiplicity of

Diagonalization of Matrices

Now that we’ve learned how to find eigenvectors and eigenvalues we can now determine whether or not the matrices are diagonalizable.

Diagonalization means turning a matrix, A, into the form of

.

This has many unique applications and can make certain operations very easy

First review that for a matrix, such as which has all diagonal entries it is very easy to calculate the matrix to any power.

Given a matrix

Now recall

Obviously this is very easy to calculate.

But what if our matrix isn’t diagonal like that?

Well if we can transfer it to the from of it actually is possible (if the matrix is determined as diagonalizable)

Because if

Then

Definition: An matrix is diagonalizable if and only if there is a basis for consisting of eigenvectors of A. Furthermore, where is a diagonal matrix and P is an invertible matrix, if and only if the columns of P are a basis for consisting of eigenvectors of A and the diagonal entries of D are the eigenvalues of the respective columns of P

Theorem, Every matrix having distinct eigenvalues is diagonalizable

Testing to see if a matrix is Diagonalizable

For an matrix A:

  • There must be at least distinct eigenvalues
  • For each eigenvalue, of A, the dimension of the corresponding eigenspace which is equal to is equal to the multiplicity of

Chapter 6 - Orthogonality

The norm of a vector, is:

A vector with a norm of 1 is called a unit vector

The distance between two vectors is simply the norm of the difference of the two vectors.

The distance between and is

We also define the dot product of any two n-length vectors to be:

If the dot product is zero, then the vectors are said to be orthogonal

You might have heard pythagorean theorem before, with respect to finding the sides of a triangle. Well, it turns out that the formula holds as well for any vectors of size

The Cauchy-Schwarz Inequality

For any two vectors and , it holds true that:

In other words, the absolute value of the dot product, must always be equal to or less than the norm of both vectors multiplied together.

Orthogonal Sets of Vectors

Any set of non-zero vectors is always going to be linearly independent.

Gram-Schmidt Process - Given a set of vectors

Then to define an orthogonal set based on these vectors:

The general formula for the k-th vector is:

Orthogonal Complements

An orthogonal complement of set of vectors is a vector, in which for all in a set,

In other words, it is solution to , or the null space of the matrix obtained from all concatenating the column vectors together.

Similarly, if we were wanted the orthogonal basis for the transpose of the set of vectors, Then it is simply:

It is simply the Null space of the transpose of the matrix.

A quick note: The dimension of a subsapce and its orthogonal complement sum to equal .

Orthogonal Projection Matrix

An orthogonal projection matrix is denoted by .

is defined as:

a matrix with a set of columns whose span is the subspace

Using you can also get the orthogonal projection of any vector onto the subspace by simple using the formula:

Least squares Regression Line

Given data points and variables, , and , we can find the coefficients for the least-squared regression line by the formula:

Where the matrix for quadratic lines and for linear lines.

The vectors are:

  • always represents a constant
  • is the coefficient
  • is the coefficient