Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Matrix Algebra, part 2

Ming-Ching Luoh

2005.9.12

1 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Characteristic Roots and Vectors Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix Quadratic Forms and Denite Matrices Calculus and Matrix Algebra Optimization

2 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

Characteristic Roots and Vectors

Also called Eigenvalues and Eigenvectors. The solution to a set of equations

Ac = c
where is the eigenvalue and c is the eigenvector of a square matrix A. If c is a solution, then kc is also a solution for any k . Therefore, it is usually normalized so that c c = 1.
3 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

To solve for the solution, the above system implies Ac = I K c. Therefore,

( A I )c = 0
This has non-zero solution if and only if the matrix ( A I ) is singular or has a zero determinant, or | A I | = 0. The polynomial in is the characteristic equation of A.

4 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

For example, if A =

5 1 2 4

, then
5 1 2 4

| A I | =

= (5 )(5 ) 2 = 2 9 + 18 = 0 = 6, 3.

5 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

For the eigenvectors,

5 1 2 4

c1 c2 c1 c2 c1 c2

= =
0 0

0 0 0 0

= 6, = 3,

1
2 2 1 2 1

1 2

, c1 c2

c1 c2

=
1 5

2 1 2

2 5

6 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

The general results of eigenvalues and eigenvectors of a symmetric matrix A. A K K symmetric matrix has K distinct eigenvectors, c1 , c2 , , c K . The corresponding eigenvalues 1 , 2 , , K are real, but need not to be distinctive. Eigenvectors of a symmetric matrix are orthogonal, i.e., ci c j = 0, i = j .

7 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

We can collect the K eigenvectors in the matrix

C = [c1 , c2 , , c K ],
and the K eigenvalues in the same order in a diagonal matrix,

1 0 0 0 2 0 = . . . . . . . . . . . . 0 0 K

8 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

From Aci = i ci , we have AC = C . Since eigenvectors are orthogonal and ci c = 1,

c1 c1 c1 c2 c1 c K c2 c1 c2 c2 c2 c K =I CC = . . . . . . . . . . . . c K c1 c K c2 c K c K
This implies C = C 1 and CC = CC 1 = I .

9 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

Diagonalization and Spectral Decomposition

Since AC = C , pre-multiply both sides by C , we have

C AC = C C

1 0 0 0 2 0 = . . . . . . . . . . . . 0 0 K

10 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

On the other hand, if we post-multiply both sides by C 1 = C , we have

1 0 0 c 1 0 2 0 . . A = C C = [c1 , , c K ] . . . . . . . . . . . . . cK 0 0 K
K

=
i =1

i ci ci

This is called the spectral decomposition of matrix A.

11 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

Rank of a Matrix Rank of a Product. For any matrix A and nonsingular matrices B and C , then rank of B AC is equal to the rank of A. pf: It is known that if A is n K and B is a square matrix of rank K , then rank( AB )=rank( A), and rank( A)=rank( A ). Since C is nonsingular, rank( B AC )=rank(( B A)C )=rank( B A). Also we have rank( B A)=rank(( B A) )=rank( A B ). It follows that rank( A B )=rank( A ) since B is nonsingular if B is nonsingular. Therefore, rank( B AC )=rank( A )=rank( A).
12 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

Using the result of diagonalization that C AC = and C and C are nonsingular (|| = |C || A||C | = 0), we have, rank(C AC ) = rank( A) = rank( ) This leads to the following theorem.

Theorem
Rank of a Symmetric Matrix. The rank of a symmetric matrix is the number of nonzero eigenvalues it contains.

13 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

It appears that the above Theorem applies to symmetric matrix only. However, note the fact that rank( A)=rank( A A), A A will always be a symmetric square matrix. Therefore,

Theorem
Rank of a Matrix. The rank of any matrix A is the number of nonzero eigenvalues in A A.

14 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

Trace of a Matrix
The trace of a square K K matrix is the sum of its K diagonal elements, tr( A) = i =1 aii . Some results about trace are as follows. tr(c A) = ctr( A), c is a scalar. tr( A ) = tr( A). tr( A + B ) = tr( A) + tr( B ). tr( I K ) = K

15 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

tr( AB ) = tr( B A). proof: Let A : n K , B : K n , C = AB and D = B A. K Since cii = a i bi = k =1 aik bki , and dkk = bk ak = in=1 bki aik ,
n n K

tr( AB ) =
i =1 K

cii =
i =1 k =1 n

aik bki
K

=
k =1 i =1

bki aik =
k =1

dkk = tr( B A)

16 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

This permutation rule can be applied to any cyclic permutation in a product: tr( ABC D ) = tr( BC D A) = tr(C D AB ) = tr( D ABC ). Along the diagonalization rule of matrix A, we can derive a rule for the trace of a matrix. tr(C AC ) = tr( ACC ) = tr( AI )
K

= tr( A) = tr( ) =
k =1

Theorem
Trace of a Matrix. The trace of a matrix equals the sum of its eigenvalues.
17 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

Determinant of a Matrix One particular simple way of calculating the determinant of a matrix is as follows. Since C AC = , |C AC | = | |.

|C AC | = |C | | A| |C | = |C | |C | | A| = |C C | | A|
K

= | I | | A| = | A| = | | =
k =1

Theorem
Determinant of a Matrix. The determinant of a matrix equals the product of its eigenvalues.
18 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

Powers of a Matrix With the result that A = C C , we can show the following theorem.

Theorem
Eigenvalues of a Matrix Power. For any nonsingular symmetric matrix A = C C , A K = C K C , K = , 2, 1, 0, 1, 2, . In other word, eigenvalues of A K is iK while eigenvectors are the same. This is for all intergers.

19 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Diagonalization and Spectral Decomposition of a Matrix Rank of a Matrix Trace of a Matrix Determinant of a Matrix Powers of a Matrix

For real number, we have

Theorem
Real Powers of a Positive Denite Matrix. For a positive denite matrix A = C C , Ar = C r C , for any real number, r . A positive denite matrix is a matrix with all eigenvalues being positive. A matrix is nonnegative denite if all the eigenvalues are either positive or zero.

20 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Quadratic Forms and Denite Matrices

Many optimization problems involve the quadratic form
n n

q = x Ax =
i =1 j =1

xi x j ai j

where A is a symmetric matrix, x is a column vector. For a given A, If x Ax > (<)0 for all nonzero x , then A is positive (negative) denite. If x Ax ()0 for all nonzero x , then A is nonnegative denite or positive semi-denite (nonpositive denite).
21 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

We can use results we have so far to check a matrix for deniteness. Since A = C C ,
n

x Ax = x C C x = y
where y = C x is another vector.

y=
i =1

i yi2

Theorem
Denite Matrices. Let A be a symmetric matrix. If all the eigenvalues of A are positive (negative), then A is positive (negative) denite. If some of them are zeros, then A is nonnegative (nonpositve) if the remainder are positive (negative).
22 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Nonnegative denite of Matrices If A is nonnegative denite, then | A| 0. Since determinant is the product of eigenvalues, it is nonnegative. If A is positive denite, so is A1 . The eigenvalues of A1 are the reciprocals of eigenvalues of A which are all positive. Therefore A1 is also positive denite. The identity matrix I is positive denite. x I x = x x > 0 if x = 0.
23 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

(Very important) If A is n K with full rank and n > K , then A A is positive denite and A A is nonnegative denite. By assumption that A is with full rank, Ax = 0. So

x A Ax = y y =
i

yi2 > 0.

For the latter case, since A have more rows than columns, there is an x such that A x = 0. We only have y y 0.
24 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

If A is positive denite amd B is nonsingular, then B AB is positive denite. x B ABx = y Ay > 0, where y = Bx . But y can not be zero because B is nonsingular. For A to be negative denite, all As eigenvalues must be negative. But te determinant of A is positive if A is in even order and negative if it is in odd order.

25 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

Calculus and Matrix Algebra

We have some denitions to begin with.

x1 x2 y = f (x ), x = . . . xn
where y is a scalar and x is a column vector.

26 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

Then the rst derivative or gradient is dened as

f1 f f 2 = . = . . . . x . n 1 y fn
xn

y x1 y x2

27 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

And the second derivatives matrix or Hessian is,

2 f H= = xx n n

2 y x1 x1 2 y x2 x1 2 y xn x1

. . .

2 y x1 x2 2 y x2 x2 2 y xn x2

. . .

. . .

2 y x1 xn 2 y x2 xn 2 y xn xn

. . .

28 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

In the case of a linear function y = a x = x a = in=1 ai xi , where a and x are both column vectors,

y (a x ) = = x x

y x1 y x2

n 1

xn

a1 a 2 = . = a. . . . . . y an

29 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

In a set of linear functions,

y
n 1

A x ,y = n n n 1

y1 y2 . . . yn

,x =

x1 x2 . . . xn a1 a2 . . .
n

, = [a1 , a2 , , an ] ai j x j = a i x

A =

a11 a12 a1n a21 a22 a2n = . . . . . . . . . . . . an 1 an 2 ann

an

yi = ai 1 x1 + ai 2 x2 + + ain xn =
j =1

30 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

The derivatives are

yi yi = ai , = ai x x n 1 1n y y ( AX ) = = x x ( AX ) = A x
x y2 x
1

a1 a2 = A = . . . . . . yn an x

31 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

A quadratic form is written as

n n

y = x Ax =
i =1 j =1

xi x j ai j
n j =1 n j =1 n j =1

(x Ax ) = . . = x . n 1
y xn

y x1 y x2

a11 a12 a21 a22 . . . . . . an 1 an 2

n x j an j + i =1 x i ain a 1n x1 a 2n x 2 . . . . . . . . . ann xn

x j a1 j + x j a2 j + . . .

n i =1 x i ai 1 n i =1 x i ai 2

32 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

a11 a21 an 1 a12 a22 an 2 . . . . . . . . . . . . a1n a2n ann

x1 x2 . . . xn

= Ax + A x = ( A + A )x = 2 Ax if A is symmetric .

The Hessian matrix is

2 (x Ax ) = 2A xx
n n
33 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

Optimization

In a single variable case, y = f (x ). The rst order dy (necessary) condition for an optimum is d = 0. And the x second order (sufcient) condition is maximum and
d2 y d2x d2 y d2x

< 0 for a

> 0 for a mimimum.

34 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

For the optimization of a function with several variable y = f (x ) where x is a vector. The rst order condition is

f (x ) =0 x
And the second order condition is that the Hessian matrix

2 f (x ) H= xx
must be positive denite for a minimum and negative denite for a maximum.
35 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

One example, we want to nd x to maximize R = a x x Ax , where x and a are n 1, A is symmetric n n . The rst order condition is

R 1 = a 2 Ax = 0, x = A1 a . x 2
n 1

And the Hessian Matrix is

2 R = 2 A . xx
n n

If A is in quadratic form and is positive denite, then 2 A is negative denite. This ensures a maximum.
36 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

An Example: OLS Regression

yi =
11

xi x

+ u i i = 1, 2, , n
11

1 K K 1

y
n 1

+ u .
n 1

n K K 1

The OLS estimator mininizes the sum of squared residuals n 2 i =1 u i = u u . In other words,
min S () = ( y x) ( y x)

= y y x y y x + x x
37 / 38

Outline Characteristic Roots and Vectors Quadratic Forms and Denite Matrices Calculus and Matrix Algebra

Optimization

The rst order condition is

S () = 2x y + 2x x = 0
K 1

O L S = (x x )1 x y = (x x )1 x (x + u ), = + ( x x ) 1 x u
And the Hessian matrix is

2 S () = 2x x
K K

The Hessian matrix is positive denite. This ensures a minimum.

38 / 38