Linear Algebra Primer

Basic Definitions

In the context of linear algebra a scalar is a number (and typically denoted symbolically by lowercase letters).  The requirements on the scalar set is essentially you can add, subtract, multiply, and divide (at least by non-zero numbers) and these operations work as you expect them to from arithmetic (such a set of numbers is called a field).  The most common scalar sets are ℝ, ℂ, and ℚ.  If p is a prime number the set (Subscript[ℤ, p]) (integers from 0 to 1 under addition and multiplication modulo p) is also a field, although you have to interpret division as multiplication by the multiplicative inverse (so for example in (Subscript[ℤ, 11]) we have 2 6≡1, so when you would normally divide by 2 instead you have to multiply by 6.

A matrix is a rectangular array of scalars (and denoted symbolically by uppercase letters).  A m×n matrix is one with m rows and n columns (sometimes written (Subscript[A, m×n])). The entries in a matrix are typically denoted by the same letter as the matrix itself in lowercase - so A=(Subscript[[(Subscript[a, ij])], m×n]) is a matrix with m rows and n columns and whose entry in row i column j is (Subscript[a, ij]).  An m×n matrix is square if m=n.

Matrix Operations

Addition and subtraction

If A and B are both m×n matrices, then the matrices A+B and A-B are the m×n matrices you form by adding/subtracting entries in the same positions in A and B.

MatrixPlus[ {{1,2,3}, {4,5,6}}, {{3,1,4}, {2,7,1}}]

MatrixSubtract[ {{1,2,3}, {4,5,6}, {7,8,9}}, {{5,1,6}, {2,1,2}, {-7,-1,-8}}]

Matrix addition is commutative and associative.  If A is an m×n matrix and 0 is the m×n matrix of all 0’s (a “zero matrix”), then A+0=0+A=A.

Scalar Multiplication

If x is a scalar and A is a matrix, then x A is the matrix where you scale all of the entries of A by x.

MatrixScale[-8, {{1,7,-1}, {2,1,4}}]

MatrixScale[x, {{1,2}, {3,4}}]

Scalar multiplication distributes over both matrix addition and subtraction.

Matrix Multiplication

If A is an m×k matrix and B is a k×n matrix, then the matrix product A B is the m×n matrix whose i-j entry is the dot product of row i of A with row j of B (dot products having been defined in the Multivariate Calculus primer).

MatrixMultiply[ {{1,2},{-3,4}}, {{5,1,6}, {2,-1,2}}]

MatrixMultiply[ {{1,4,3}, {0,-1,2}, {1,6,1}},{{1,2,3},{2,1,3}, {-9,1,7}}]

MatrixMultiply[ {{1,2,1}, {3,1,8}, {2,1,9}}, {{0}, {1}, {0}}]

Note that when defined matrix multiplication is associative but not commutative (i.e. A B≠ B A for many matrices A and B).

The matrix (Subscript[I, n]), the “n×n identity matrix”, is the matrix with n rows and columns that has 1’s on the main diagonal and 0’s everywhere else.

({{({{(Subscript[I, 2])=(({{1, 0}, {0, 1}}))            (Subscript[I, 3])}})=(({{1, 0, 0}, {0, 1, 0}, {0, 0, 1}}))}})

If A is an m×n matrix then (Subscript[I, m])A=A and A (Subscript[I, n])=A.  In addition if A is an m×n matrix and J is the j-th column of (Subscript[I, n]) is just the j-th column of A (as in the last example above for matrix multiplication - multiplying by the second column of (Subscript[I, 3]) results in a copy of the second column of the original matrix).

If A is a n×n matrix then A is said to be invertible if there is an n×n matrix B with A B=B A=(Subscript[I, n]). Such a matrix B if it exists in unique and is called the inverse of A (and written (A^(-1))).

Matrix Powers

If A is a n×n matrix and n∈ℕ, then (A^n)=A....A (n times).  By definition (A^0)=(Subscript[I, n]).  Negative powers of A are only defined if A is invertible and are defined as (A^(-k))=((((A^(-1))))^k).  If all of the individual powers of A are well-defined then (A^k)(A^j)=(A^(j+k)) and ((((A^j)))^k)=(A^(j k)).  In many cases high powers of a matrix can be difficult to compute unless there is a pattern you can detect in the powers (for example (((({{1, 1}, {0, 1}})))^k)=(({{1, k}, {0, 1}})) ) or if A has a special property (such as all entries off the main diagonal being 0, the rows being a rearrangement of (Subscript[I, n]), being similar to a diagonal matrix, etc.)

Matrix Transpose

If A is an m×n matrix, then the transpose of A (written (A^T)) is the n×m matrix which is A with its rows and columns swapped. For example

({{({{(((({{1, 2, 3}, {4, 5, 6}})))^T)=(({{1, 4}, {2, 5}, {3, 6}}))}})}})

Properties of the transpose include (((A+B))^T)=(A^T)+(B^T), (((A-B))^T)=(A^T)-(B^T), (((c A))^T)=c (A^T), (((A B))^T)=(B^T)(A^T), and ((((A^T)))^T)=A.

Special Types of Matrices

A square matrix A is symmetric if and only if A=(A^T).

A square matrix A is skew-symmetric if and only if (A^T)=-A.

A square matrix is upper-triangular if and only if all entries below the main diagonal are 0.

A square matrix is lower-triangular if and only if all entries above the main diagonal are 0.

A square matrix is diagonal if and only if all entries off the main diagonal are 0.

A n×n matrix A is called nilpotent if (A^k)=(Subscript[0, n×n]) for some natural number k.

A square matrix A is called idempotent if (A^2)=A.

An elementary matrix is one that is obtained from an identity matrix (Subscript[I, n]) via a single row or column operation.

Matrix Computations

Determinant of a Square Matrix

If A is a square matrix then the determinant of A (written det(A), or |A| ) is a number computed from the entries of A. Although it is defined for any size n (typically by a “cofactor expansion” or the Leibniz formula), on a timed exam you are likely to encounter determinants of 2×2 and 3×3 matrices.

For a 2×2 matrix the determinant is det((({{a, b}, {c, d}})))=a d-b c.

For a 3×3 matrix the easiest way to compute the determinant is:

Add a copy of columns 1 and 2 at the end of the matrix (changing it to a 3×5).

Add the products along the three full “left-to-right” diagonals (starting at entries (Subscript[a, 11]), (Subscript[a, 12]), and (Subscript[a, 13])) and subtract the products along the 3 full “right-to-left” diagonals (starting at entries (Subscript[a, 13]), (Subscript[a, 14]), and (Subscript[a, 15]))

({{({{(({{a, b, c}, {d, e, f}, {g, h, i}}))⟶}})(({{a, b, c, a, b}, {d, e, f, d, e}, {g, h, i, g, h}}))⟶}, {a e i + b f g + c d h -c e g-a f h-b d i}})

If det(A)=0 then A is called a singular matrix

The determinant has many special properties:

det(A B)=det(A)det(B)

det((Subscript[I, n]))=1

A square matrix A is invertible if and only if det(A)≠0

A system of n equations in n variables has a unique solution for all “right-hand sides” if and only if the determinant of the coefficient matrix is not 0.

A square matrix A is the product of elementary matrices if and only if det(A)≠0.

If (A^(-1)) exists, det((A^(-1)))=(1/(det(A)))

det ((A^T)) = det (A)

If B is obtained by scaling a single row or column of A by a factor of c, det(B)=c det(A)

If B=c A and A is an n×n matrix, then det(B)=(c^n)det(A) (this follows from the previous property but is useful to have on tis own).

If B is obtained from A by a single swap or rows or a single swap of columns, det(B)=-det(A)

If B is obtained from A by adding a multiple of one row to another or adding a multiple of one column to another, det(B)=det(A)

If A has two identical rows, det(A)=0

Trace of a Matrix

The trace of a square matrix (written tr(A)) is the sum of the entries on the main diagonal.

Reduced Row Echelon Form and Gauss-Jordan Reduction

A matrix is said to be in reduced row echelon form if:

Any row that is not all 0’s has a 1 for its first non-zero entry (such an entry is called the leading 1 for that row).

Any entry above or below a leading 1 must be 0.

Rows which are all 0 must be at the bottom of the matrix.

Each leading one is to the right of all the leading ones above it.

Any square matrix A can be put in reduced row echelon form through the use of elementary row operations (scaling rows, swapping rows, and adding multiples of one row to another), and for any any matrix A the reduced row echelon form is unique.  The standard process for putting a matrix A in reduced row echelon form (which involves creating leading ones by row swaps and row scaling and then eliminating entries above and below that leading 1) is called Gauss-Jordan reduction.

A n×n matrix A is invertible if and only its reduced row echelon form is (Subscript[I, n]).

The number of leading ones in reduced row echelon form is the row rank/rank of the matrix (formally defined later).

A matrix whose form meets all the requirements for reduced row echelon form except that all entries above leading ones must be 0 is said to be in row echelon form.

Inverse of a Matrix

Let A be an n×n matrix.  Then (A^(-1)) (if it exists) can be found by:

Adjoining the matrix (Subscript[I, n]) to A (making the block matrix [A|(Subscript[I, n])])

Putting the adjoined matrix A in reduced row echelon form [J|B]

If J=(Subscript[I, n]), then B=(A^(-1)).  If J≠(Subscript[I, n]), then A is not invertible.

For example consider the matrix (({{3, 1}, {2, 8}})):

Text[Style[GaussJordan[ {{3,1,1,0}, {2,8,0,1}}, All], FontSize-> 14, FontFamily->"Times"]]

So (((({{3, 1}, {2, 8}})))^(-1))=(({{(4/11), -(1/22)}, {-(1/11), (3/22)}})).

For a 2×2 matrix A=(({{a, b}, {c, d}})), if det(A)≠0 then (A^(-1))=(1/(a d-b c))(({{d, -b}, {-c, a}})) (if the entries are from (Subscript[ℤ, p]), then (1/(a d-b c)) must be interpreted as (((a d-b c))^(-1))). If det(A)=0 then A is not invertible.

Note that (((A B))^(-1))=(B^(-1))(A^(-1)) (provided A and B are invertible) and if A is invertible and c≠0 then (((c A))^(-1))=(1/c)(A^(-1)) and ((((A^T)))^(-1))=((((A^(-1))))^T).  If A is an invertible matrix then the normal algebra rules for exponents (((((A^m)))^n)=(A^(m n)), (A^m)(A^n)=(A^(m+n))) apply for all integer exponents, not just the non-negative ones.

Eigenvalues and Eigenvectors of a Matrix

Let A be an n×n matrix.  A scalar λ is an eigenvalue for A if there is a non-zero solution to the equation A x=λ x (or (λ (Subscript[I, n])-A)x=0), where x is the column matrix of variables (({{(Subscript[x, 1])}, {(Subscript[x, 2])}, {⋮}, {(Subscript[x, n])}})) (such a solution is said to be an eigenvector for A corresponding to A and the set of all such solutions is the eigenspace for A corresponding to λ).  Equivalently λ is an eigenvalue for A iff det(λ (Subscript[I, n])-A)=0.  (Subscript[p, A])(λ)=det(λ (Subscript[I, n])-A) is a polynomial of degree n in λ called the characteristic polynomial of A.  If you interpret the constant term of p(A) as being a multiple of (Subscript[I, n]), then (Subscript[p, A])(A)=(Subscript[0, n×n]).

For example let A=(({{2, 1}, {3, 4}})).  Then the characteristic polynomial is given by p(λ)=det(λ (Subscript[I, 2])-A)=det((({{λ-2, -1}, {-3, λ-4}})))=(λ-2)(λ-4)-(-1)(-3)=(λ^2)-6λ+5.  Solving p(λ)=0 yields λ=1,5.  For the eigenvalue 1 the eigenvectors would be the solutions to (({{2, 1}, {3, 4}}))(({{x}, {y}}))=1(({{x}, {y}})).  This is the same as 2x+y=x and 3x+4y=y.  This becomes x+y=0 and 3x+3y=0.  So y=-x (for any x), and the vectors have the form (({{x}, {-x}}))=x(({{1}, {-1}})) (the set of all such vectors would be the eigenspace for 1, and individual vectors like (({{2}, {-2}})) and (({{-3}, {3}})) are eigenvectors).  Using the alternative definition  ((λ (Subscript[I, n])-A)x=0 to find the eigenspace for λ=5 we have ((({{5, 0}, {0, 5}}))-(({{2, 1}, {3, 4}})))(({{x}, {y}}))=(({{0}, {0}})), or (({{3, -1}, {-3, 1}}))(({{x}, {y}}))=(({{0}, {0}})), which is 3x-y=0 and -3x+y=0.  The solution to this is y=3x (for any x), so the corresponding eigenspace is (({{x}, {y}}))=(({{x}, {3x}}))=x(({{1}, {3}})).  Examples of individual eigenvectors for λ=5 would therefore include (({{1}, {3}})), (({{5}, {15}})), and (({{-2}, {-6}})).  Also note that

({{({{ p(A)=(((({{2, 1}, {3, 4}})))^2)-6(({{2, 1}, {3, 4}}))+5(Subscript[I, 2])=(({{7, 6}, {18, 19}}))-(({{12, 6}, {18, 24}}))+(({{5, 0}, {0, 5}}))=(({{0, 0}, {0, 0}}))}})}})

Some of the properties of eigenvalues are:

The number of independent eigenvectors in the eigenspace for an eigenvalue λ (i.e. the number of “free variables” in the solution for finding the eigenspace for λ, or the “dimension” of the eigenspace) is at least 1 but at most the multiplicity of λ as a root of the characteristic polynomial.

Eigenvalues are invariant under similarity (i.e. the eigenvalues of A and P A (P^(-1)) are the same for all invertible matrices P).

If A is an n×n matrix, then when finding the characteristic polynomial for A it has the form (λ^n)-tr(A)(λ^(n-1))....+(((-1))^n)det(A)

A matrix A is similar to a diagonal matrix (i.e. A=P D (P^(-1)) where D is diagonal and P is an invertible matrix) if and only if every eigenspace for A has “dimension” equal to the multiplicity of its eigenvalue λ as a root of the characteristic polynomial.  The form A=P D (P^(-1)) makes it easy to find powers of A (as (A^n)=(((P D (P^(-1))))^n)=P((D^n))(P^(-1)), and powers of a diagonal matrix are easy).

If A is upper or lower triangular, the eigenvalues are exactly the numbers that appear on the main diagonal (including multiplicity).

The eigenvalues of (A^T) are the same as those of A

If c≠0, the eigenvalues of c A are c times the eigenvalues of A

If n∈ℕ and λ is an eigenvalue of A, then (λ^n) is an eigenvalue of (A^n) (if A is invertible this is true for all integer powers n)

λ=0 is an eigenvalue of A if and only if A is singular (i..e det(A)=0)

If A has real entries and is a symmetric matrix (i.e. A=(A^T)), all of the eigenvalues of A are real.

Systems of Linear Equations (SOLE’s)

A system of m linear equations in n unknowns (Subscript[x, 1]),⋯,(Subscript[x, n]) can be represented in matrix form A x=b, where A is the matrix of coefficients, x= (({{(Subscript[x, 1])}, {(Subscript[x, 2])}, {⋮}, {(Subscript[x, n])}})), and b is the column vector of right-hand sides. If all the right-hand sides are 0 the system is called homogenous.

Solution Techniques for All SOLE’s

If you have a SOLE in the form A x=b the system can always be solved by creating the augmented matrix [A|b] and using row operations to put it in either reduced row echelon form (or row echelon form).  In either form the system will have no solution if and only if one of the rows is (({{0, ⋯, 0, a}})) where a≠0. If there is a solution, in reduced row echelon the solution can be found be rewriting the final matrix back as equations and solving for each variable that corresponds to a leading 1 (if there variables that do not correspond to a leading 1, they are “free” variables, and the number of them is dimension of the infinite set of solutions).  

In row echelon form you can solve the system (if it has a solution) by rewriting the final matrix as equations and solving for the variables corresponding to leading 1’s in order from bottom to top (so-called “back substitution”).

Solution Techniques for Square SOLE’s

If the SOLE A x=b is square (same number of equations as variables) there may be other (is some cases faster) ways to solve the system.  If det(A)=0 then the system has either no or infinitely many solutions, and by hand you would use the previous technique to see which possibility has.

If det(A)≠0, then the solution is unique and there are two other common ways to find the solution:

Inverse Method

If det(A)=0 then (A^(-1)) exists, and you can find the solution via taking A x=b and multiplying by (A^(-1)) on the left to get x=(A^(-1))b.  On a timed test this can be a good approach if the system is only 2 equations in 2 unknowns (so (A^(-1)) can be computed quickly).  In other settings it is also useful if you have to resolve the same system repeatedly and only change the right-hand side each time (so (A^(-1)) only has to be determined once).

Cramer’s Rule

If the SOLE is written as A x=b, det(A)≠0, and the variables are (Subscript[x, 1]),…,(Subscript[x, n]) , then the value of (Subscript[x, k]) can be directly computed as (Subscript[x, k])=((det((Subscript[A, k])))/(det(A))), where (Subscript[A, k]) is A with its (k^th) column replaced with b.  On a timed test this can be useful if the system is small enough that the determinants can be computed quickly or if the question is only to find the value of one of the variables in the solution.

Vector spaces

Vector Spaces and Subspaces

A vector space is a set of objects which have a notion of being added, subtracted, and scaled (either by real numbers or in some cases complex numbers).  Implied by the notion of vector subtraction is the idea of a unique “zero” vector (often written 0 or (OverVector[0])).  The archetype of real-scalar (or just “real”) vector spaces is (ℝ^n) in which n-tuples of real numbers are just added, subtracted, and scaled coordinate-by-coordinate (so (3,2)+(1,-3)=(4,-1), (4,5,1)-(8,1,0)=(-4,4,1), and 5(1,2,3,4)=(5,10,15,20)).  The archetype for vector spaces with complex scalars (“complex vector spaces”) is (ℂ^n), defined in the same way.  Vector spaces can also be defined with the scalar set (Subscript[ℤ, p]) (where p is a prime).  It is common to see either (ℝ^n) or (ℂ^n) represented either by row or column matrices. 

A subspace of a vector space is a subset of the vector space which is still a vector spacing using the same addition, and subtraction, and scaling.  A subset of a vector space will be a subspace if and only if it contains the zero vector and is closed under vector addition and scaling (i.e. if the subset is S, then 0∈S and whenever (Subscript[v, 1]),(Subscript[v, 2])∈S, then (Subscript[v, 1])+(Subscript[v, 2])∈S and c (Subscript[v, 1])∈S for all scalars c).  Equivalently a subspace is a subset which contains the zero vector and is closed under linear combinations (i.e. if the subset is S, then 0∈S and whenever (Subscript[v, 1]),(Subscript[v, 2])∈S then (Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])∈S for all scalars (Subscript[c, 1]) and (Subscript[c, 2])).  Note that the intersection of subspaces is always another subspace (however in general the union of subspace is not a subspace).

Examples of other real vector spaces would include (Subscript[P, n])(ℝ) (the set of polynomials of degree n or less with real coefficients under polynomial addition/subtraction and regular scalar multiplication) and (Subscript[M, m,n])(ℝ) (the set of m+×n matrices with real entries and matrix addition, subtraction, and scalar multiplication).  Replacing ℝ with ℂ gives examples of complex vector spaces (and the same can be done with (Subscript[ℤ, p]) where p is a prime).

Examples of subspaces include the line y=3x in (ℝ^2), polynomials f(x) in (Subscript[P, 4])(ℝ) with f(2)=0, and matrices in (Subscript[M, 3,2])(ℝ) whose first column is 0.  If A is a square matrix and λ an eigenvalue of A, the set of eigenvectors for A corresponding to λ (the eigenspace) is a subspace of (ℝ^n).

Linear Independence, Spanning Sets, Bases, and Dimension

Linear Independence

Let {(Subscript[v, 1]),...(Subscript[v, k])} be a set of vectors in a vector space V.  {(Subscript[v, 1]),...(Subscript[v, k])} is said to be linearly independent if and only if

({{({{(Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])+…+(Subscript[c, k])(Subscript[v, k])=0}}) ⟹ (Subscript[c, 1])=(Subscript[c, 2])=…=(Subscript[c, k])=0}})

.   Set of vectors which is not linearly independent is said to be linearly dependent.  Any vector of the form (Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])+…+(Subscript[c, k])(Subscript[v, k]) is called a linear combination of the vectors (Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k]) (so linear independence can be phrased as the only linear combination which results in the zero vector is the “zero combination”).  Any set of non-zero vectors will have a subset which is linearly independent.

Checking for linear independence typically involves starting with  ({{(Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])+…+(Subscript[c, n])(Subscript[v, k])=0}}), converting it to a homogenous system of linear equation, and showing that the system has a unique solution (the 0 solution).  Note that if the system is square checking to see if the determinant of the coefficient matrix is 0 or not is probably the simplest way to do this.

If the vectors (Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k]) are in (ℝ^n), linear independence can be checked by making a matrix A whose rows are the vectors and putting it in row echelon (or reduced row echelon form, although that is more work).  (Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k]) are linearly independent if and only if in a row echelon form for A there are exactly k leading 1’s (even better, the number of leading 1’s is the largest number of vectors from (Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k]) which can be linearly independent).

For example to check to see if { (1,2,4,3), (1,1,1,1), (3,1,2,1) } are linearly independent in (ℝ^4) we form the matrix (({{1, 2, 4, 3}, {1, 1, 1, 1}, {3, 1, 2, 1}})) and put it in row echelon form to get (({{1, 2, 4, 3}, {0, 1, 3, 2}, {0, 0, 1, (2/5)}})); as we started with 3 vectors and got 3 leading 1’s, the set is lienarly independent.

If A is a square matrix and (Subscript[λ, 1])≠(Subscript[λ, 2]) are distinct eigenvalues with non-zero eigenvectors (Subscript[v, 1]) and (Subscript[v, 2]), then the set {(Subscript[v, 1]),(Subscript[v, 2])} is linearly independent.

Spanning Sets

A set of vectors {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} in a vector space V is a spanning set for V (or “spans V”) if and only if every vector v∈V can be written as a linear combination of the vectors (Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k]).  To show a set {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} typically you set up the equation (Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])+…+(Subscript[c, k])(Subscript[v, k])=v ( where v is a “generic” vector of V), convert this to a system of linear equations, and show this system always has a solution (a “generic” vector in (ℝ^2) looks like (a,b), a “generic” vector in (Subscript[P, 3])(ℝ) looks like a+b x + c (x^2)+d (x^3), etc.).  For example to check if {1+x, (x^2)+x+1, 2x-(x^2)} is a spanning set we set up the linear combination (Subscript[c, 1])(1+x)+(Subscript[c, 2])((x^2)+x+1)+(Subscript[c, 3])(2x-(x^2))=a+b x+c (x^2).  Equating coefficients gives the system (Subscript[c, 1])+(Subscript[c, 2])=a, (Subscript[c, 1])+(Subscript[c, 2])+2(Subscript[c, 3])=b, (Subscript[c, 2])-(Subscript[c, 3])=c.  In matrix form this is (({{1, 1, 0, a}, {1, 1, 2, b}, {0, 1, -1, c}})), which in reduced row echelon form this is (({{1, 0, 0, (3/2)a-(1/2)b-c}, {0, 1, 0, -(1/2)a+(b/2)+c}, {0, 0, 1, -(a/2)+(b/2)}})), so the system always has a solution ((Subscript[c, 1])=(3/2)a-(1/2)b-c, (Subscript[c, 2])=-(1/2)a+(b/2)+c, (Subscript[c, 3])=-(a/2)+(b/2)) and that means the set {1+x, (x^2)+x+1, 2x-(x^2)} is a spanning set.

Note that a faster way to check the previous example is to take advantage of the fact that it is a square system.  As the determinant of the coefficient matrix (({{1, 1, 0}, {1, 1, 2}, {0, 1, -1}})) is -2≠0 it must have a unique solution for any “right hand sides”.  Since we don’t care so much what the solution is as the fact that there is a solution, the determinant will be a lot faster that the row reduction approach.

As a special case if V is (ℝ^n), spanning can be checked by making a matrix A whose rows are the vectors {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} and putting it in row echelon (or reduced row echelon form, although that is more work). {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} will span (ℝ^n) if and only if in row echelon form there are n leading 1’s.

An idea related to spanning sets is the idea of the span of a set of vectors.  If {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} are vectors in V, then span({(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])}) is the set of all possible linear combinations of the vectors (i.e. the set (Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])+…+(Subscript[c, k])(Subscript[v, k]) for all possible values of the scalars).  span({(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])}) is always a subspace of V, and {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} is always a spanning set for span({(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])}).

Basis

A set of vectors {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} in a vector space V is a basis for V if and only if it is linearly independent and spans V.  All bases for V have the same number of vectors in them - this number is the dimension of V (written dim(V)).  Given a set of vectors in V of the right size you can check to see if it is a basis by seeing if is linearly independent and spans V.

For (ℝ^n) the “standard basis” consists of the rows (or columns) of (Subscript[I, n]) (so for (ℝ^3) this is (1,0,0), (0,1,0), and (0,0,1)).  So dim((ℝ^n))=n.

For (Subscript[P, n])(ℝ) the “standard basis” is the set {1,x,..., (x^n)}.  So dim((Subscript[P, n])(ℝ))=n+1.

For (Subscript[M, a b])(ℝ) the “standard basis” are the set of matrices which are 1 in a single entry and 0 everywhere else.  So dim((Subscript[M, a b])(ℝ)=a b.

If V={0} (the trivial vector space), by convention dim(V)=0.

If dim(V)=n, and set of k>n vectors in V must be linearly independent and any set of k<n vectors cannot span V.

In (ℝ^n), the dimension of span({(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])}) is found by forming the matrix A whose rows are the vectors, finding its row echelon or reduced row echelon form, and counting the number of leading 1’s (so dim( span({(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])})=rank(A)).  

As a special case, in (ℝ^n) a set of n vectors is a basis if and only if the matrix A whose rows are the vectors row reduces to (Subscript[I, n]).  This is equivalent to det(A)≠0 and (A^(-1)) existing.  Since the determinant can be quickly computed when n=2,3 in these cases the determinant is a quick way to check for a basis.

Note that some vector spaces can be both real vector spaces and complex vectors, and the dimension depends on if you consider it as real or complex.  For example ℂ={a+b 𝕚|a,b∈ℝ} is 2 dimensional as a real vector space (basis {1,𝕚}) but only 1 dimensional as a complex vector space (basis {1}).  In such cases the “real dimension” is twice the “complex dimension”.

In (ℝ^n), a vector subspace of dimension k≤n can be written as a solution of a system of n-k linearly independent homogenous equations (linearly independent in the sense that the coefficient matrix for the system has rank n-k).  This can be a useful way about thinking of  subspaces of (ℝ^n) - for example in (ℝ^8) any 5-dimensional subspace can be thought of as a solution to 8-5=3 linearly independent homogenous equations.

If B={(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} is a basis for V, then every vector v∈V can be uniquely written as a linear combination v= (Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])+…+(Subscript[c, k])(Subscript[v, k]) (that is, the values for the c’s are unique).  Written as a vector the numbers ((Subscript[c, 1]), (Subscript[c, 2]),...(Subscript[c, n])) are the coordinates of v with respect to B.

For example the vectors {(2,1), (5,3)} form a basis for (ℝ^2) (this is easy to check as det((({{2, 1}, {5, 3}})))=1≠0).  To find the coordinates of (1,4) with respect to this basis we set (1,4)=(Subscript[c, 1])(2,1)+(Subscript[c, 2])(5,3)=(2(Subscript[c, 1])+5(Subscript[c, 2]),(Subscript[c, 1])+3(Subscript[c, 2])).  So 2(Subscript[c, 1])+5(Subscript[c, 2])=1 and (Subscript[c, 1])+3(Subscript[c, 2])=4.  Using Kramer’s rule (Subscript[c, 1])=((det((({{1, 5}, {4, 3}}))))/(det((({{2, 1}, {5, 3}})))))=((-17)/1)=-17 and (Subscript[c, 2])=((det((({{2, 1}, {1, 4}}))))/(det((({{2, 1}, {5, 3}})))))=(7/1)=7.  So the coordinates of (1,4) with respect to the basis {(2,1), (5,3)} are (-17,7).  As a check, -17(2,1)+7(5,3)=(-34,-17)+(35,21)=(1,4).

If (Subscript[B, 1])={(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} and (Subscript[B, 2])={(Subscript[w, 1]),(Subscript[w, 2]),… (Subscript[w, k])} are two bases for a vector space, you can find a matrix which will convert coordinates with respect to (Subscript[B, 1]) into coordinates with respect to (Subscript[B, 2]) (a change of basis matrix).  Such a matrix will be k×k, be invertible, and its (k^th) column is the coordinates of (Subscript[v, k]) with respect to (Subscript[B, 2]).  If A is the matrix for changing bases from (Subscript[B, 1]) to (Subscript[B, 2]), then (A^(-1)) is the matrix for changing bases from (Subscript[B, 2]) to (Subscript[B, 1]).

Linear Transformations

Let V and W be vector spaces.  Then a linear transformation from V to W is a function T:V->W for which T((Subscript[v, 1])+(Subscript[v, 2]))=T((Subscript[v, 1]))+T((Subscript[v, 2])) and T(c (Subscript[v, 1]))=c T((Subscript[v, 1])) for all vectors (Subscript[v, 1]),(Subscript[v, 2]) and all scalars c (essentially T is an “operation preserving” function between vector spaces).  Some examples of linear transformations are:

T:(R^3)->(R^2) given by T( (x,y,z) )=(x,z)
T:(Subscript[P, 4])->(Subscript[P, 4]) given by T(f(x))=f''(x)
T:V->V (for any vector space V) given by T(v)=2v
If A is an m×n matrix, T:(ℝ^n)->(ℝ^m) given by T((({{(Subscript[x, 1])}, {⋮}, {(Subscript[x, n])}})))=A(({{(Subscript[x, 1])}, {⋮}, {(Subscript[x, n])}}))

If T is a linear transformation from V to W, the kernel of T (sometimes denoted ker(T)) is the set {v∈V|T(v)=(Subscript[0, W])}.  The kernel of T is always a subspace of V, and its dimension is called the nullity of T.  For the examples above:

T( (x,y,z) )=(x,z):  ker(T)={ (0,y,0) |y∈ℝ} and the nullity is 1.
T(f(x))=f''(x): ker(T)={a+b x|a,b∈ℝ} and the nullity is 2.
T(v)=2v: ker(T)={0} and the nullity is 0.
 T((({{(Subscript[x, 1])}, {⋮}, {(Subscript[x, n])}})))=A(({{(Subscript[x, 1])}, {⋮}, {(Subscript[x, n])}})):  ker(T) is the solution set to the homogenous SOLE A(({{(Subscript[x, 1])}, {⋮}, {(Subscript[x, n])}}))=(Subscript[0, m×1]), known as the nullspace of A.  The nullity is the number of free variables in the solution to the homogenous solution, which is n-rank(A).
 
A linear transformation T is 1-1 if and only if ker(T)={0}.

If T:V->W is a linear transformation and {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} is a basis for V, then the values for T are completely determined by the values T((Subscript[v, 1])),T((Subscript[v, 2])),…, T((Subscript[v, k])) (that is, if you know those k values you can find any value).  This is because as {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} is a basis for V any vector v can be written as v=(Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])+…+(Subscript[c, k])(Subscript[v, k]) for specific numbers (Subscript[c, 1]),…,(Subscript[c, k]); the definition of what it means to be a linear transformation tells you that T(v)=T((Subscript[c, 1])(Subscript[v, 1])+(Subscript[c, 2])(Subscript[v, 2])+…+(Subscript[c, k])(Subscript[v, k]))=(Subscript[c, 1])T((Subscript[v, 1]))+(Subscript[c, 2])T((Subscript[v, 2]))+…+(Subscript[c, k])T((Subscript[v, k])).  This is a common problem on exams.  For example suppose T:(ℝ^2)->(ℝ^2) and T( (2,1) )=(1,3) and T(-1,1)=(5,2) - what is T( (3,2) )?  Setting a(1,3)+b(1,1)=(3,2) quickly yields a=(5/3), b=(1/3).  So T( (3,2) )=T((5/3)(2,1)+(1/3)(1,1))=(5/3)T((2,1))+(1/3)T( (1,1) )=(5/3)(1,3)+(1/3)(5,2)=((10/3),(17/3)).

If T is a linear transformation from V to W the range of T is always a subspace of W, and nullity of T + dim(range(T))=dim(V).  This formula can be very useful as in many cases either the kernel or range of T can be found without too much work - which means you can find dim(range(T)) or the nullity very quickly.  For example range(T)=dim(V) if and only if the nullity is 0.

If T:V->W is a linear transformation from V to W and V has a basis {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} and W has a basis {(Subscript[w, 1]),(Subscript[w, 2]),… (Subscript[w, n])}, then it is possible to represent T as a n×k matrix as follows:

For each vector (Subscript[v, i]), find the coordinates of T((Subscript[v, i])) with respect to the basis {(Subscript[w, 1]),(Subscript[w, 2]),… (Subscript[w, n])}.  Stack these coordinates as an n×1 column vector.

Create the matrix A whose columns are the vectors found in the previous step (in order) - A will the the n×k matrix.

A represents T as follows: for any v in V, find the its coordinates with respect to {(Subscript[v, 1]),(Subscript[v, 2]),… (Subscript[v, k])} as a column vector C.  Then the matrix product A C are the coordinates of T(v) with respect to {(Subscript[w, 1]),(Subscript[w, 2]),… (Subscript[w, n])}.

A special case of this is where V=(ℝ^k) and W=(ℝ^n) and the standard bases are used for both.  In this case the matrix A is the n×k matrix whose columns are the values T( (1,0,0,...) ), T( (0,1,0,...) ), ..., T( (0,0,..., 1) ).  For example suppose T:(ℝ^3)->(ℝ^4) given by T( (x,y,z) )=(x+y,x-y, x+y+z, 2x-2y+5z).  The standard basis for (ℝ^3) is (1,0,0), (0,1,0), and (0,0,1).  T( (1,0,0) )=(1,1,1,2), T( (0,1,0) )=(1,-1,1,-2), and T( (0,0,1) )=(0,0,1,5).  So the matrix A which represents T with respect to the standard bases is A=(({{1, 1, 0}, {1, -1, 0}, {1, 1, 1}, {2, -2, 5}})).  As a quick check, (({{1, 1, 0}, {1, -1, 0}, {1, 1, 1}, {2, -2, 5}}))(({{x}, {y}, {z}}))=(({{x+y}, {x-y}, {x+y+z}, {2x-2y+5z}})), which is the matrix form of the formula T( (x,y,z) )=(x+y,x-y, x+y+z, 2x-2y+5z).

Suppose T:(ℝ^n)->(ℝ^n) and A is the matrix for T.  If there is a basis B for (ℝ^n) which consists of non-zero eigenvectors for A, then the matrix for T with respect to the basis B is a diagonal matrix and therefore very easy to work with (this is one of the reasons eigenvalues and eigenvectors are important)

Inner Product Spaces

A special kind of vector space is an inner product space - one that in addition to its other properties has an operation which combines two vectors to get a scalar and essentially works like the dot product.  For real vector spaces, unless there is a specially defined symbol (like · for the standard dot product) the inner product of two vectors is typically written as <u,v>.  The properties a real inner product must have are:

<c u,v> = c<u,v> and <u,c v> = c<u,v> for all vectors u and v and all scalars c

<(Subscript[u, 1])+(Subscript[u, 2]),v> = <(Subscript[u, 1]),v>+<(Subscript[u, 2]),v> and <u,(Subscript[v, 1])+(Subscript[v, 2])> = <u,(Subscript[v, 1])>+ <u,(Subscript[v, 2])> for all vectors u,v, (Subscript[u, 1]), (Subscript[u, 2]), (Subscript[v, 1]), and (Subscript[v, 2])

<u,v> = <v,u> for all vectors u and v

For all vectors u, <u,u>≥0 and <u,u> =0 if and only if u=0

Complex vector spaces can have an inner product but it is unlikely you will see these on a timed test (the main difference is that two properties using the second input use complex conjugation - <u,c v> = (OverBar[c])<u,v> and <u,v> = (OverBar[<v,u>]) ).  The complex version of the standard dot product on (ℂ^n) is <(Subscript[a, 1]),(Subscript[a, 2]),…,(Subscript[a, n])>·<(Subscript[b, 1]),(Subscript[b, 2]),…,(Subscript[b, n])> = (Subscript[a, 1])(OverBar[(Subscript[b, 1])])+(Subscript[a, 2])(OverBar[(Subscript[b, 2])])+… +(Subscript[a, n])(OverBar[(Subscript[b, n])]).

The archetypical example of real inner product spaces is (ℝ^n) with the standard dot product (so <u,v> = u·v).  Another common example is the set of continuous functions on an interval [a,b] (which is a vector space under function addition and scalar multiplication by numbers) with the inner product <f(x),g(x)> = (∫_a^b)f(x)g(x)dx.

If V is an inner product space then it automatically inherits some common geometric ideas involving length, angle, and direction:

becomes equipped with notions of length, an angle between two vectors, and orthogonality:

The length of u is ||u||=(Sqrt[<u,u>])

The angle between two non-zero vectors u and v is defined by  cos(θ)=((<u,v>)/(||u|| ||v||))

u and v are orthogonal if and only if <u,v> =0

The projection of u onto v (i.e. the part of u that lies in the direction of v) is (Subscript[proj, v])(u)=((<u,v>)/(<v,v>))v

If the vector space V is (ℝ^2) or (ℝ^3) these formulas reduce to the ones you learned in Calculus 2 and Multivariate Calculus, but inner products let you use these ideas in other vector spaces as well.  For example

In (ℝ^5), ||<3,1,4,1,5>||=(Sqrt[<3,1,4,1,5>·<3,1,4,1,5>])=(Sqrt[9+1+16+1+25])=(Sqrt[52])

In the continuous functions on [0,1], ||(x^2)||=(Sqrt[(∫_0^1)(x^(2 ))(x^2)dx])=(Sqrt[(∫_0^1)(x^4)dx])=(Sqrt[(1/5)])=(1/(Sqrt[5]))
In (ℝ^4), the angle between <1,0,0,1> and <1,0,0,0> is defined by cos(θ)=((<1,0,0,1>·<1,0,0,0>)/(||<1,0,0,1>|| ||<1,0,0,0>||))=(1/(Sqrt[2])), so θ=(π/4).

In the continuous functions on [-π,π], sin(x) and cos(x) are orthogonal as <sin(x),cos(x)> =(∫_(-π)^π)sin(x)cos(x)dx=(1/2)(∫_(-π)^π)sin(2x)dx=0 (as sin(2x) is an odd function).

In (ℝ^4), the projection of <2,7,1,8> onto <1,1,0,0> is ((<2,7,1,8>·<1,1,0,0>)/(<1,1,0,0>·<1,1,0,0>))<1,1,0,0> =(9/2)<1,1,0,0> = <(9/2),(9/2),0,0>.