Transcription
Review from Bootcamp: Linear AlgebraD. Alex HughesOctober 27, 2014
1Properties of Estimators2Linear AlgebraAddition and SubtractionTransposeMultiplicationCross ProductTrace3Special MatricesMatrix InversionDeterminantsD. Alex HughesLinear Algebra ReviewOctober 27, 20142 / 46
Properties of Estimators1Unbiasedness:E (θ̂) θ 02Asymptotic Unbiasedness:lim P( E [θ̂] θ ) 0; 0n 3Efficiency11 MSEE [(θ̂ θ)2 ]4Consistencylim P( θ̂ θ ) 0 0n D. Alex HughesLinear Algebra ReviewOctober 27, 20143 / 46
Linear algebra Motivation: Linear algebra or matrix algebra avoids the mess and letsus solve for things we care about quickly, cleanly and easily. This is no different than algebra. Consider the difference in thefollowing formulas for the mean:x̄ x̄ x1 x2nx1 x2 x3nx1 x2 x3 x4x̄ nPxix̄ nD. Alex HughesLinear Algebra ReviewOctober 27, 20144 / 46
Linear algebra Similarly, matrix algebra is a form of notation that cleans up the messwhen working with more complex formulas. So suspend disbelief andconcern, and treat this as a new language you are learning. Think of this as algebra on steroids.D. Alex HughesLinear Algebra ReviewOctober 27, 20145 / 46
MotivationWhy are we studying matrix algebra? We will use matrix algebra to derive the least squares estimator Matrices are an intuitive way to think about data. We have a set ofobservations (perhaps individuals) on the row, and observe manydifferent characteristics (such as race, gender, PID, etc.)corresponding to columns Matrices are useful for solving systems of equations, like multipleregression Notation is much more compact and conciseD. Alex HughesLinear Algebra ReviewOctober 27, 20146 / 46
Definition of Matrices and VectorsDefinitionA matrix is simply an arrangement of numbers in rectangular form.Generally, a (j k) matrix A can be written a11 a12 · · · a21 a22 · · · A . .aj1 aj2 · · ·as follows: a1ka2k . . ajkNote that there are j rows and k columns, defining the dimensionality(order) of the matrix. Also note that the elements are double sub-scripted,with the row number first, and the column number second. In generalterms, the A above is of order (j, k).D. Alex HughesLinear Algebra ReviewOctober 27, 20147 / 46
ExamplesExample W 1 32 6 is of order (2, 2). This is also called a square matrix.There are also rectangular matrices (j 6 k), such as:Example 1 4 1 3 Γ 1 2 0 3 which is of order (4,2).D. Alex HughesLinear Algebra ReviewOctober 27, 20148 / 46
Notation Matrices are usually written using capital, bold-faced Roman or Greekletters. Roman is typically data, and Greek is typically parameters.This is not universal.D. Alex HughesLinear Algebra ReviewOctober 27, 20149 / 46
VectorDefinitionVectors are matrices that have either one row or one column. is the sameas a scalar – a regular number.Row vectors have a single row and multiple columns. α α1 α2 α3 · · · αkColumn vectors are those that have a single column and multiplerows. y1 y2 y . . . ykD. Alex HughesLinear Algebra ReviewOctober 27, 201410 / 46
Operations on MatricesAddition and SubtractionScalar addition is simply:m n 2 5 7Addition is similarly defined for matrices.If matrices or vectors are of the same order, then they can be added. Oneperforms the addition element by element.D. Alex HughesLinear Algebra ReviewOctober 27, 201411 / 46
AdditionA B C: a11 a12b11 b12a11 b11 a12 b12c11 c12 a21 a22b21 b22a21 b21 a22 b22c21 c22D. Alex HughesLinear Algebra ReviewOctober 27, 201412 / 46
SubtractionA B D: 1 4 2 3 2 84 2 10 5 3 32 2 33 56D. Alex HughesLinear Algebra ReviewOctober 27, 201413 / 46
Properties of Matrix Addition A B B A. Matrix addition is commutative. (A B) C A (B C). Matrix addition is associative.D. Alex HughesLinear Algebra ReviewOctober 27, 201414 / 46
TransposeDefinitionTo transpose a matrix is to exchange order subscripts. An order (j, k)matrix becomes an order (k, j) matrix.Transposition is denotedsuperscript T . q1,1 Q q2,1q3,1by placing a prime after a matrix or by placing a q1,2q2,2 q3,20Q q1,1 q2,1 q3,1q1,2 q2,2 q3,2 Note that the subscripts in the transpose remain the same, they are justexchanged.D. Alex HughesLinear Algebra ReviewOctober 27, 201415 / 46
ExampleExample 1 3 ω 2 5D. Alex Hughesω0 1 3 2 5Linear Algebra Review October 27, 201416 / 46
Some DefinitionsThere are a few results regarding transposition that are important toremember: An order (j, j) matrix A is said to be symmetric iff A A0 . 1 .2 .5W .2 1 .4 .5 .4 1 1 .2 .5W0 .2 1 .4 .5 .4 1 (A0 )0 A For a scalar k, (kA)0 kA0 . For two matrices of the same order, the transpose of the sum is equalto the sum of the transposes. (A B)0 A0 B0D. Alex HughesLinear Algebra ReviewOctober 27, 201417 / 46
Matrices and MultiplicationScalar times a matrix. In words, a scalar α times a matrix A equals thescalar times each element of A. Thus, a1,1 a1,2αa1,1 αa1,2αA α a2,1 a2,2αa2,1 αa2,2So, for: A D. Alex Hughes4 8 26 8 10 1A 2Linear Algebra Review 2 4 13 4 5 October 27, 201418 / 46
Matrices and MultiplicationDefinitionGiven A of order (m, n) and B of order (n, r ), then the product AB C isthe order (m, r ) matrix whose entries are defined by:ci,j nXai,k bk,jk 1where i 1, . . . , m and j 1, . . . , r and k1 n2D. Alex HughesLinear Algebra ReviewOctober 27, 201419 / 46
Matrices and Multiplication A 2 1 34 1 6 3 24 B 21 3 2 · 3 1 · 2 3 · 1 2 · ( 2) 1 · 4 3 · ( 3)AB 4·3 1·2 6·14 · ( 2) 1 · 4 6 · ( 3) 1 1 20 22D. Alex HughesLinear Algebra Review October 27, 201420 / 46
Matrices and MultiplicationIs multiplication of matrices commutative? A D. Alex Hughes 2 1 34 1 6 3 24 B 21 3Linear Algebra Review October 27, 201421 / 46
Matrices and Multiplication 141 3630 BA 12 14 2 15No: Multiplication of matrices is not commutative. In other words:AB 6 BA.D. Alex HughesLinear Algebra ReviewOctober 27, 201422 / 46
Matrices and MultiplicationImportant results Matrix multiplication is not commutative: AB 6 BA. Matrix multiplication is associative:(AB)C A(BC) Matrix multiplication is distributive:A(B C) AB AC The transpose of a product can be written as(AB)0 B0 A0D. Alex HughesLinear Algebra ReviewOctober 27, 201423 / 46
Vectors and MultiplicationInner product of vectors e0 e e1 e2 · · ·eN e1e2. eNe0 e e1 e1 e2 e2 · · · eN eN NXei2i 1Alt: outer productD. Alex HughesLinear Algebra ReviewOctober 27, 201424 / 46
Other Useful Vector ProductsLet i denote an order (N, 1) vector of ones, and x denote an order (N, 1)vector of data.Xi0 x (x1 x2 · · · xN ) xiFrom this, it follows that:1 X1 0ix xi x̄NND. Alex HughesLinear Algebra ReviewOctober 27, 201425 / 46
Cross ProductA B n̂ A B cos(θ) n̂: perpendicular unit vector A : Length of A θ: angle between A & BD. Alex HughesLinear Algebra ReviewOctober 27, 201426 / 46
TraceSum of the diagonal elements of a square matrix.a11 a12 · · ·a21 a22 · · ·A .an1 an2 · · ·tr (A) D. Alex HughesXa1na2n.annaii a11 a22 · · · annLinear Algebra ReviewOctober 27, 201427 / 46
Special Matrices and Their PropertiesWhen performing scalar algebra, we know that x · 1 x, which is knownas the identity relationship.There is a similar relationship in matrix algebra: AI A.What is I?It can be shown that I is a diagonal, square matrix with ones on the maindiagonal, and zeros on the off diagonal.For example, the order three identity matrix is: 1 0 0I3 0 1 0 0 0 1D. Alex HughesLinear Algebra ReviewOctober 27, 201428 / 46
Special Matrices and Their PropertiesNotice that I is oftentimes subscripted to denote its dimensionality. Here isan example of the use of an identity matrix: 1 21 01 2 3 40 13 4D. Alex HughesLinear Algebra ReviewOctober 27, 201429 / 46
Special Matrices and Their PropertiesOne of the nice properties of the identity matrix is that it is commutativewith respect to multiplication. That is,AIB IAB ABI ABAn identity in scalar algebra is x 0 x.This generalizes to matrix algebra, with the definition of the null matrix,which is simply a matrix of zeros, denoted 0j,k .Here is an example: 1 20 01 2A 02,2 A3 40 03 4D. Alex HughesLinear Algebra ReviewOctober 27, 201430 / 46
Matrix InversionDefinition:ZZ 1 IThis is roughly akin to division in non-matrix algebra. Actually calculatingthe inverse of a matrix takes several steps and has several prerequisites.D. Alex HughesLinear Algebra ReviewOctober 27, 201431 / 46
Matrix InversionGeneral solution for a square matrix A:A 1 So we need to figure outD. Alex Hughes1 A 1adjA A and adj(A).Linear Algebra ReviewOctober 27, 201432 / 46
Matrix InversionThe first of these, A , is called “the determinant”. There’s lots to learnabout determinants, but we’ll stick to the basics.Most importantly, if the determinant is NOT zero, a square matrix isinvertible.The determinant is a scalar, that is, a single number, like 5.For a two by two matrix, the determinant is: a11 a12 A a11 a22 a21 a12a21 a22So you multiply the corners and subtract one product from the other.D. Alex HughesLinear Algebra ReviewOctober 27, 201433 / 46
Graphical Intuition for Determinant of a 2x2 MatrixD. Alex HughesLinear Algebra ReviewOctober 27, 201434 / 46
D. Alex HughesLinear Algebra ReviewOctober 27, 201435 / 46
D. Alex HughesLinear Algebra ReviewOctober 27, 201436 / 46
Graphical Linear Algebra For a 2x2 matrix, the determinant is two times the area of thetriangle defined by the row vectors. Think about this for a matrix like: 1 21 2 1 25 10 A Or. A Most linear algebra functions can be represented graphically. Ask me for citations if you want a book that illustrates all these.D. Alex HughesLinear Algebra ReviewOctober 27, 201437 / 46
Calculating DeterminantsWith three by three matrices, the determinant is still quite manageable: A a11 a12 a13a21 a22 a23a31 a32 a33 a11 a22 a33 a12 a23 a31 a13 a21 a32 a11 a23 a32 a12 a21 a33 a13 a22 a31Graphically, this means adding the diagonal products from left to right,and subtracting the diagonal products from right to left. (example)D. Alex HughesLinear Algebra ReviewOctober 27, 201438 / 46
More DeterminantsFor bigger matrices, we have to use alternative methods, typically theLaplace Expansion. Basically, we break the matrix into sub-matricies, andcalculate determinants of these submatricies, then combine our results.Steps:1Pick a row or column to work with.2For each element in that row, calculate the subdeterminant, alsocalled the minor.3Multiply each element by its subdeterminant, determine signs, andadd.D. Alex HughesLinear Algebra ReviewOctober 27, 201439 / 46
Higher Order Determinants - Example5 6 12 3 07 3 0D. Alex Hughes 5 3 0 3 0 6 Linear Algebra Review2 07 0 1 2 37 3October 27, 201440 / 46
Properties of Determinants A A0 Interchanging any two rows or columns will alter sign but not value ofdeterminant. Multiplication of one row by k will change A to k A . Addition/subtraction of a multiple of any row to another row willleave the value of the determ unaltered (works for col too). If one row or columns is a multiple of another, the value of thedeterminant will be zero - matrix is singularD. Alex HughesLinear Algebra ReviewOctober 27, 201441 / 46
Tricks for DeterminantsPick a good row or column.5 6 12 3 03 3 0 5 3 0 3 0 6 2 07 05 6 12 3 07 3 0 1 2 37 3 0 5 67 3D. Alex HughesLinear Algebra Review 1 0 2 37 35 62 3October 27, 201442 / 46
Tricks for DeterminantsManipulate rows if possible5 6 12 4 73 6 11D. Alex Hughes 5 6 10 12 4 4 73 6 6 11Linear Algebra Review 5 4 12 073 0 11October 27, 201443 / 46
Back to inverting a matrixA 1 1adj(A) A (adj means adjugate)where adj(A) D. Alex Hughes C11 C12 · · · C21 C22 · · ·. Cn1 Cn2 · · ·Linear Algebra Review C1n C2n . 0 Cnn October 27, 201444 / 46
CofactorsCij is a matrix cofactor - the determinant of the matrix when excludingrow i and column j, and adj(A) is the transpose of the matrix of cofactors.The determinant of the cofactor submatrix is multiplied by 1 when i j iseven, and by -1 when i j is odd.D. Alex HughesLinear Algebra ReviewOctober 27, 201445 / 46
Easy Inversion - 2X21adj(A) A 1 2X 1 3 13 2 1 1 3 2 A 1 X 1D. Alex HughesLinear Algebra ReviewOctober 27, 201446 / 46
Linear algebra Similarly, matrix algebra is a form of notation that cleans up the mess when working with more complex formulas. So suspend disbelief and concern, and treat this as a new language you are learning. Think of this as algebra on steroids. D. Alex Hughes Linear Algebra Review October 27, 2014 5 / 46
