Lecture Notes on Math for Electrical Engineers

Dr. Eleftherios Gkioulekas

Copyright c 2009 Eleftherios Gkioulekas. All rights reserved. Permission is granted to make and distribute verbatim copies of this document provided the copyright notice and this permission notice are preserved on all copies.

Contents MEE1: Brief introduction to Logic and Sets MEE2: Basic Linear Algebra MEE3: Graph Theory. Part 1 MEE4: Eigenvalues and eigenvectors MEE5: Graph Theory. Part 2 MEE6: Applications of linear systems Programming with Matlab Algorithms with Matlab More on linear ODE systems 1 13 72 129 159 214 235 257 274

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

Appendices

236

237

Programming with Matlab

Eleftherios Gkioulekas Mathematical Sciences Computing Center University of Washington December, 1996

Starting Matlab

Matlab is an interactive tool that includes facilities for dealing with numerical analysis, matrix computation, signal processing and graphics. It is meant to be used to understand and test mathematical concepts interactively before coding in a real programming language. Throughout this tutorial, we will give you an overview of various things and refer you to Matlabs on-line help for more information. The best way to learn is by experimentation, and the best way this tutorial can help you is by telling you the basics and giving you directions towards which you can take o exploring on your own. To start Matlab type matlab on the shell prompt. You get a greeting on your screen, then a window pops up and goes away 1 and nally you get a >> prompt. At this point you can start typing in commands on interactive mode or you can quit by typing quit at the prompt. You can also navigate around the lesystem, execute the Matlab programs you have written, and get online help. If you want to log your what you do throughout your matlab session type >> diary filename where in filename you put the lename you want to use for your log. To stop logging type >> diary off Moreover you can get on-line help by typing >> help on the prompt. A list of topics appears. You can then type help again followed with the name of the topic and get more information about what you want to know. For example suppose you type: >> help [..lots of stuff..] Matlab/matfun - Matrix functions - numerical linear algebra. [..lots of other stuff..]
The reason this happens is because Matlab tries to see if you have an X display. It can do that without popping up a window, but then you wouldnt be as impressed
1

238

There you go! Say you want to learn more about the matrix stu. You type >> help matfun Matrix functions - numerical linear algebra. Matrix analysis. cond - Matrix condition number. norm - Matrix or vector norm. [etc...etc...] and you are given a list of commands that are available to you for working with matrices. Then, to learn about the cond command you type >> help cond and you will be told how to use it to compute condition numbers. For more information about getting help try: >> help help Another way of getting help is with the lookfor command. Suppose you want to see what Matlab can do with eigenvalues. Typing: >> lookfor eigenvalue will return to you a list of commands in which the word eigenvalue occurs. Note that this command is not very articially intelligent. If you are looking for your son, typing >> lookfor son will not give you what you want If you are on a machine that can run a web browser, try also: >> doc Finally, if Matlab ever does something you dont understand type: >> why for a succinct explanation. Like we said, at the prompt you can also execute your programs. So, lets do that!. Bring up an editor 2 and type in the following Matlab program:
On a Unix system you can use the following editors: emacs, vi and if available pico. On DOS there is edit. Other systems have (or are supposed to have) a text editor. Warning: You can NOT use a word-processor to type in Matlab programs, unless the word processor allows you to save your document as pure text.
2

239

% % These are comments. In Matlab you can add comments with a % character % This is the standard hello world program disp(Hello world!); Save the program with the lename hello.m. You want all your Matlab programs to be saved with the extension .m at the end. Then start Matlab under the same directory where you saved the le and type hello at the Matlab prompt. Then you should see the following: >> hello Hello world! >> When you terminate a Matlab command with a semicolon, the command will execute silently. When you dont, the command will print back a response. In a matlab program, the former behaviour is desirable. When using Matlab interactively you may want to see these responses. As a general rule of thumb, when you write Matlab programs terminate every statement with a semi-colon and produce the output of interest by invoking the commands whose job is to print things. Such a command would be disp for example, which will print hello even though it is being silenced with a semicolon. We will learn about another printing command later on. Notice that within the Matlab environment you dont have to type any funny commands to load your program. When you start Matlab from your shell, Matlab will load all the *.m les that happen to be under the present working directory at startup time. So, all you have to do when the prompt shows up is to type the name of the program (without the *.m extension) and it will get executed. Note that Matlab will only look for les with the *.m extension, so you are forced to use it. There are no work-arounds for this. Still, Matlab has a provision for the situation where your les are scattered in more than one directory. You can use the Unix cd, ls, pwd commands to navigate in the le system and change your current working directory. Also, if you want to be able to have access at the les on two or more seperate directories simultaneously type >> help path for more information.

Matlab variables

Matlab has three basic data types: strings, scalars and matrices. Arrays are just matrices that have only one row. Matlab has also lots of built-in functions to work with these things. You have already seen the disp function in our hello-world program. Starting with strings, you can assign a string to a variable like this: name = Indiana Jones; Note that it is a syntax error to quote the string with anything other than the forward quote marks. So, the following are wrong! 3

240

name = "Indiana Jones"; name = Indiana Jones;

wrong! wrong!

In a Matlab program you can prompt the user and ask him to enter in a string with the input command: % This is a rather more social program % yourname = input(Hello! Who are you? ,s); dadname = input(Whats your daddy name? ,s); fprintf(1,Hail oh %s son of %s the Great! \n,yourname,dadname); The input command takes two arguments. The rst argument is the string that you want the user to be prompted with. You could stick in a variable instead of a xed string if you wanted to. The second argument tells Matlab to expect the user to enter a string. If you omit the second argument, then Matlab will be expecting a number, and upon you entering your name, Matlab will complain. Finally, it returns the value that the user enters, and that value is passed on through assignment to the variable yourname. The fprintf command gives you more exibility in producing output than the disp command. There is a lot to learn about fprintf so please type help fprintf to learn all you need to know. Briey, fprintf takes two or three or more arguments. The rst argument is a le descriptor. File descriptors are integers that reference places where you can send output and receive input from. In Matlab, le descriptor 1 is what you use when you want to send things to the screen. The terminology you may hear is that le descriptor 1 sends things to the standard output. The rest of the arguments depend on what you want to print. If all you want to print is a xed string, then you just put that in as your second argument. For example: fprintf(1,Hello world!\n); The \n sequence will switch you over to the next line. disp will do this automatically, in fprintf you must explicitly state that you wish to go to a new line. This is a feature, since there may be situations where you do not want to go to a new line. If you want to print the values of variables interspersed with your string then you need to put appropriate markers like %s to indicate where you want your variables to go. Then, in the subsequent arguments you list the variables in the appropriate order, making sure to match the markers. There are many markers and the Matlab online help will refer you to a C manual. The most commonly used markers are the following: %s Strings %d Integers (otherwise you get things like 5.0000) %g Real numbers in scientic notation. In our example above, we just used %s. You will see further examples later on. Note that if you merely want to print a variable, it is better to use disp since it will format it for you. fprintf is more useful for those occasions where you want to do the formatting yourself as well as for sending things to a le. Scalars can be assigned, inputed and printed in a similar fashion. Here is an example: 4

241

% yet another one of these happy programs age = input(Pardon for asking but how old are you?); if (age < 75) life_left = 365.25*24*(75 - age); fprintf(1,You have %g hours left of average life expectancy.\n,life_left); else fprintf(1,Geez! You are that old?!\n); end fprintf(1,Live long and prosper!\n); Note the following: String and numeric variables look the same. You dont have to declare the type of the variable anywhere. Matlab will make sure to do the right thing (tm). When we use input to get the value of a numeric variable we omit the second s argument. This way, Matlab will do error-checking and complain if you entered something thats not a number. You can use fprintf to print numeric variables in a similar fashion, but you got to use the %g marker. If you are printing an integer you must use the %d marker, otherwise Matlab will stick in a few zeroes as decimal places to your integer. It is obvious that you can mix strings and numbers in an fprintf command, so long as you dont mix up the order of the variables listed afterwards. On line life_left = 365.25*24*(75 - age); we see how you can do simple computations in Matlab. Its very similar to C and Fortran and to learn more about the operators you have available type >> help ops >> help relops Finally, we have an example of an if statement. We will talk of that more later. The meaning should be intuitively obvious. In addition to ordinary numbers, you may also have complex numbers. The symbols i and j are reserved for such use. For example you can say: z = 3 + 4*i; or z = 3 + 4*j; 5

242

where i and j represent 1. If you are already using the symbols i and j as variables, then you can get a new complex unit and use it in the usual way by saying: ii = sqrt(-1); z = 3 + 4*ii;

Arrays in Matlab

Next we talk about arrays. In Matlab arrays are dynamic and they are indexed from 1. You can assign them element by element with commands like: a(1) = 23; a(2) = input(Enter a(2)); a(3) = a(1)+a(2); It is a syntax error to assign or refer to a(0). This is unfortunate since in some cases the 0-indexing is more convenient. Note that you dont have to initialize the array or state its size at any point. The array will make sure to grow itself as you index higher and higher indices. Suppose that you do this: a(1) = 10; a(3) = 20; At this point, a has grown to size 3. But a(2) hasnt been assigned a value yet. In such situations, during growth any unset elements are set to zero. It is good programming practice however not to depend on this and always initialize all the elements to their proper values. Notice that for the sake of eciency you might not like the idea of growing arrays. This is because every time the array is grown, a new chunk of memory must be allocated for it, and contents have to be copied. In that case, you can set the size of the array by initializing it with the zeros command: a = zeros(100); This will set a(1),a(2),...,a(100) all equal to zero. Then, so long as you respect these boundaries, the array will not have to be grown at any point. Here are some other ways to make assignments to arrays: x = [3 4 5 6]; will set x equal to an array of 4 values. You can recursively add elements to your array x in various ways if you include x on the right hand side. For example, you can make assignments like x = [x 1 2] % append two elements at end of the array x = [1 2 x 3 ] % append two elements at front, one at back How about making deletions? Well, rst of all notice that we can access parts of the array with the following indexing scheme: 6

243

y = x(2:4); will return the an array of x(2), x(3), x(4). So, if you want to delete the last element of the array, you just have to nd the size of the array, which you can do with the size command. Yet another way to setup arrays is like this: x = 3 : 1 : 6; This will set x equal to an array of equidistant values that begin at 3, end at 6 and are separated from each other by steps of 1. You can even make backwards steps if you provide a negative stepsize, like this: x = 6 : -1 : 3; It is common to set up arrays like these when you want to plot a function whose values are known at equidistant points. Finally, to conclude, you may want to know how to load arrays from les. Suppose you have a le that contains a list of numbers separated with carriage returns. These numbers could be the values of a function you want to plot on known values of x (presumably equidistant). You want to load all of these numbers on a vector so you can do things to them. Here is a demo program for doing this: filename = input(Please enter filename:,s); fd = fopen(filename); vector = fscanf(fd,%g,inf); fclose(fd); disp(vector); Here is how this works: The rst line, prompts the user for a lename. The fopen command will open the le for reading and return a le descriptor which we store at variable fd. The fscanf command will read in the data. You really need to read the help page for fscanf as it is a very useful command. In principle it is a little similar to fprintf. The rst argument is the le descriptor from which data is being read. The second argument tells Matlab, what kind of data is being read. The %g marker stands for real numbers in scientic notation. Finally the third argument tells Matlab to read in the entire le in one scoop. Alternatively you can stick in an integer there and tell Matlab to load only so many numbers from the le. The fclose command will close the le descriptor that was opened. Finally the disp command will show you what has been loaded. At this point you could substitute with somewhat more interesting code if you will.

244

Another common situation is data les that contain pairs of numbers separated by carriage returns. Suppose you want to load the rst numbers onto one array, and the second numbers to another array. Here is how that is done: filename = input(Please enter filename: ,s); fd = fopen(filename); A = fscanf(fd,%g %g\n,[2,inf]); x = A(1,:); y = A(2,:); fclose(fd); disp(Here comes x:); disp(x); disp(Here comes y:); disp(y); Again, you need to read the help page for fscanf to understand this example better. You can use it in your programs as a canned box until then. What we do in this code snippet essentially is to load the le into a two-column matrix, and then extract the columns into vectors. Of course, this example now leads us to the next item on the agenda: matrices.

Matrices in Matlab

In Matlab, arrays are matrices that have only one row. Like arrays, matrices can be dened element by element like this: a(1,1) = 1; a(1,2) = 0; a(2,1) = 0; a(2,2) = 1; Like arrays, matrices grow themselves dynamically as needed when you add elements in this fashion. Upon growth, any unset elements default to zero just like they do in arrays. If you dont want that, you can use the zeros command to initialize the matrix to a specic size and set it equal to zero, and then take it from there. For instance, the following example will create a zero matrix with 4 rows and 5 columns: A = zeros(4,5); To get the size of a matrix, we use the size command like this: [rows,columns] = size(A); When this command executes, the variable rows is set equal to the number of rows and columns is set equal to the number of columns. If you are only interested in the number of rows, or the number of columns then you can use the following variants of size to obtain them: rows = size(A,1); columns = size(A,2); Since arrays are just matrices with one row, you can use the size(array,2) construct to get hold of the size of the array. Unfortunately, if you were to say: 8

245

s = size(array);

% wrong!

it would be wrong, because this returns both the number of rows and columns and since you only care to pick up one of the two numbers, you pick up the number of rows, which for arrays is always equal to 1. Not what you want! Naturally, there are a few other ways to assign values to a matrix. One way is like this: A = [ 1 0 0 ; 0 1 0 ; 0 0 1] This will set A equal to the 3 by 3 identity matrix. In this notation you list the rows and separate them with semicolons. In addition to that you can extract pieces of the matrix, just like earlier we showed you how to extract pieces of the arrays. Here are some examples of what you can do: a b c d = = = = A(:,2); A(3,:); A(1:4,3); A(:,[2 4 10]); % % % % this this this this is is is is the the a 4 the 2nd column of A 3rd row of A by 1 submatrix of A 2nd, 4th and 10th columns of A stacked

In general, if v and w are arrays with integer components, then A(v,w) is the matrix obtained by taking the elements of A with row subscripts from v and column subscripts from w. So: n = size(A,2); A = A(:,n:-1:1); will reverse the columns of A. Moreover, you can have all of these constructs appear on the left hand side of an assignment and Matlab will do the right thing. For instance A(:,[3 5 10]) = B(:,1:3) replaces the third, fth and tenth columns of A with the rst three columns of B. In addition to getting submatrices of matrices, Matlab will allow you to put together block matrices from smaller matrices. For example if A,B,C,D are a square matrices of the same size, then you can put together a block matrix like this: M = [ A B ; C D ] Finally, you can get the transpose of matrix by putting a mark next to it. For example: A = [ 2 4 1 ; 2 1 5 ; 4 2 6]; Atrans = A; Matlab provides functions that return many special matrices. These functions are listed in Figure 1 and we urge you to look these up with the help command and experiment. To display the contents of matrices you can simply use the disp command. For example, to display the 5 by 5 Hilbert matrix you want to say:

246

zeros ones rand randn eye compan diag gallery hadamard hilb invhilb pascal toeplitz vander

Returns the zero matrix Returns a matrix in which all entries are set equal to one A matrix with uniformly distributed random elements A matrix with normally distributed random elements The identity matrix Computes the companion matrix Extract one of the diagonals of a matrix Returns a couple of small test matrices. See help page. Returns a Hadamard matrix of any order where N, N/12 or N/20 is a power of 2 Returns the Hilbert matrices Returns the inverse of Hilbert matrices Returns Pascals triangle Returns Toeplitz matrices Returns Vandermonde matrices. Figure 1: Matrix commands

>> disp(hilb(5)) 1.0000 0.5000 0.5000 0.3333 0.3333 0.2500 0.2500 0.2000 0.2000 0.1667

0.3333 0.2500 0.2000 0.1667 0.1429

0.2500 0.2000 0.1667 0.1429 0.1250

0.2000 0.1667 0.1429 0.1250 0.1111

The Hilbert matrix is a famous example of a badly conditioned matrix. It is also famous because the exact inverse of it is known analytically and can be computed with the invhilb command: >> disp(invhilb(5)) 25 -300 -300 4800 1050 -18900 -1400 26880 630 -12600 1050 -18900 79380 -117600 56700 -1400 26880 -117600 179200 -88200 630 -12600 56700 -88200 44100

This way you can interactively use these famous matrices to study concepts such as ill-conditioning.

Matrix/Array Operations
Matrix addition/subtraction: Matrices can be added and subtracted like this: A = B + C; A = B - C; It is necessary for both matrices B and C to have the same size. The exception to this rule is when adding with a scalar: 10

247

A = B + 4; In this example, all the elements of B are increased by 4 and the resulting matrix is stored in A. Matrix multiplication: Matrices B and C can be multiplied if they have sizes n p and p m correspondingly. The product then is evaluated by the well-known formula
p

Aij =
k=1

Bik Ckj , i {1, . . . , n}, j {1, . . . , m}

In Matlab, to do this you say: A = B*C; You can also multiply all elements of a matrix by a scalar like this: A = 4*B; A = B*4;

% both are equivalent

Since vectors are matrices that are 1 n, you can use this mechanism to take the dot product of two vectors by transposing one of the vectors. For example: x = [2 4 1 5 3]; y = [5 3 5 2 9]; p = x*y; This way, x is now a n 1 matrix and y is 1 n and the two can be multiplied, which is the same as taking their dot product. Another common application of this is to apply n n matrices to vectors. There is a catch though: In Matlab, vectors are dened as matrices with one row, i.e. as 1 n. If you are used to writing the matrix product as Ax, then you have to transpose the vector. For example: A x y y = = = = [1 3 2; 3 2 5; 4 6 2]; [2 5 1]; A*x; A*x; % % % % define a matrix define a vector This is WRONG! This is correct :)

Array multiplication: This is an alternative way to multiply arrays: Cij = Aij Bij , i {1, . . . , n}, j {1, . . . , m} This is not the traditional matrix multiplication but its something that shows up in many applications. You can do it in Matlab like this: 11

248

C = A.*B; Array division: Likewise you can divide arrays in matlab according to the formula: Cij = Aij , i {1, . . . , n}, j {1, . . . , m} Bij

by using the ./ operator like this: C = A./B; Matrix division: There are two matrix division operators in Matlab: / and \. In general X = A\B X = A/B is a solution to is a solution to A*X = B X*A = B

This means that A\B is dened whenever B has as many rows as A. Likewise A/B is dened whenever B has as many columns as A. If A is a square matrix then it is factored using Gaussian elimination. Then the equations A*X(:,j) = B(:,j) are being solved for every column of B. The result is a matrix with the same dimensions as B. If A is not square, it is factored using the Householder orthogonalization with column pivoting. Then the corresponding equations will be solved in the least squares t sense. Right division A/B in Matlab is computed in terms of left division by A/B = (A\B). For more information type >> help slash Matrix inverse: Usually we are not interested in matrix inverses as much as applying them directly on vectors. In these cases, its best to use the matrix division operators. Nevertheless, you can obtain the inverse of a matrix if you need it by using the inv function. If A is a matrix then inv(A) will return its inverse. If the matrix is singular or close to singular a warning will be printed. Matrix determinants: Matrix determinants are dened by
n

det(A) =
Sn

sign( )
i=1

Ai(i)

where Sn is the set of permutations of the ordered set (1, 2, . . . , n), and sign( ) is equal to +1 if the permutation is even and 1 if the permutation is odd. Determinants make sense only for square matrices and can be computed with the det function: a = det(A);

12

249

Matrix exponential function: These are some very fascinating functions. The matrix exponential function is dened by exp(A) = Ak k! k=0
+

where the power Ak is to be evaluated in the matrix product sense. Recall that your ordinary exponential function is dened by + k x x e = k! k=0 which converges for all x (even when they are complex). It is not obvious from this that the corresponding matrix expression also converges. But it does, and the result is the matrix exponential. The matrix exponential can be computed in Matlab with the expm function. Its usage is as simple as: Y = expm(X); Matrix exponentials show up in the solution of systems of dierential equations. Matlab has a plethora of commands that do almost anything that you would ever want to do to a matrix. And we have only discussed a subset of the operations that are permitted with matrices. The following help calls should be helpful in exploring what Matlab oers: >> help elmat >> help matfun >> help sparfun We will go into a detailed discussion about matrices and linear algebra in Matlab, on a separate tutorial.

Flow control in Matlab

So far, we have spent most of our time discussing the data structures available in Matlab, and how they can be manipulated, as well as inputted and outputed. Now we continue this discussion by discussing how Matlab deals with ow control. For loops: In Matlab, a for-loop has the following syntax: for v = matrix statement1; statement2; .... end 13

250

The columns of the matrix are stored one at a time in the variable, and then the statements up to the end statement are executed. If you wish to loop over the rows of the matrix then you simply transpose it and plug it in. It is recommended that the commands between the for statement and the end statement are indented by one space so that the reader of your code can visually see that these statements are enclosed in a loop. In many cases, we use arrays for matrices, and then the for-loop reduces to the usual for-loop we know in languages like Fortran and C. In particular using expressions of the form X:Y will eectively make the loop variable be a scalar that goes from X to Y. Using an expression of the form start:step:stop will allow you to loop a scalar from value start all the way to value stop with stepsize step. For instance the Hilbert matrix is dened by the equation: Aij = 1 i+j+1

If we didnt have the hilb command, then we would use for-loops to initialize it, like this: N = 10; A = zeros(N,N); for i = 1:N for j = 1:N A(i,j) = 1/(i+j-1); end end The same code can be rewritten more consisely like this: N = 10; A = zeros(N,N); for i = 1:N , for j = 1:N , A(i,j) = 1/(i+j-1); , end, end Note that the for and end statements in the long-winded version of the example are being separated by newlines and not semicolons. To obtain the consise version, we merely substitute those newlines with commas. While loops: In Matlab while loops follow the following format: while variable statement1; statement2; .... statementn; end where variable is almost always a boolean expression of some sort. In Matlab you can compose boolean expressions as shown in Figure 2. Here is an example of a Matlab program that uses the while loop: 14

251

a == b a > b a < b a <= b a >= b a = b a & b a | b a xor b a

True True True True True True True True True True

when a equals b when a is greater than b when a is smaller than b when a is smaller or equal to b when a is greater or equal to b when a is not equal to b when both boolean expressions a and b are true when at least one of a or b is true. only when only one of a or b is true. when a is false.

Figure 2: Some boolean expressions in Matlab n = 1; while prod(1:n) < 1.e100 n = n + 1; end disp(n); This program will display the rst integer for which n! is a 100-digit number. The prod function takes an array (or matrix) as argument and returns the product of its elements. In this case, prod(1:n) is the factorial n!. If and Break statements: The simplest way to set-up an if branch is like this: if variable statement1; .... statementn; end The statements are executed only if the real part of the variable has all non-zero elements. 3 Otherwise, the program continues with executing the statements right after the end statement. The variable is usually the result of a boolean expression. The most general way to do ifbranching is like this: if variable statement1; ......
Note that in the most general case, variable could be a complex number. The if statement will only look into its real part
3

15

252

statementn; elseif variable2 statement1; ...... statementn; [....as many elseifs as you want...] else statement1; ...... statementn; end In this case, if variable is true, then the statements right after it will execute until the rst else or elseif (whichever comes rst), and then control will be passed over to the statements after the end. If variable is not true, then we check variable2. Now, if that one is true, we do the statements following thereafter until the next else or elseif and when we get there again we jump to end. Recursively, upon consecutive failures we check the next elseif. If all the elseif variables turn out to be false, then we execute the statements after the else. Note that the elseifs and/or the else can be omitted all together. Here is a general example that illustrates the last two methods of ow control. % Classic 3n+1 problem from number theory while 1 n = input(Enter n, negative quits. ); if n <= 0, break, end while n > 1 if rem(n,2) == 0 , n = n/2; else, n = 3*n+1; end end end This example involves a fascinating problem from number theory. Take any positive integer. If it is even, divide it by 2; if it is odd, multiply it by 3 and add 1. Repeat this process until your integer becomes a 1. The fascinating unsolved problem is: Is there any integer for which the process does not terminate? The conjecture is that such integers do not exist. However nobody has managed to prove it yet. The rem command returns the remainder of a Euclidean division of two integers. (in this case n and 2)

16

253

Functions in Matlab

Matlab has been written so that it can be extended by the users. The simplest way to do that is to write functions in Matlab code. We will illustrate this with an example: suppose you want to create a function called stat which will take an array and return its mean and standard deviation. To do that you must create a le called stat.m. The le has to have the same name as the function you are dening, and the function denition is the only thing you can put in that le!. Then in that le, say the following: function [mean, stdev] = stat(x) % stat -- mean and standard deviation of an array % The stat command returns the mean and standard deviation of the % elements of an array. Typical syntax is like this: % [mean,dev] = stat(x); % % See also: foo, gleep, bork [m,n] = size(x); if m == 1 m = n; end mean = sum(x)/m; stdev = sqrt(sum(x.^2)/m - mean.^2); The rst line of the le should have the keyword function and then the syntax of the function that is being implemented. Notice the following things about the function syntax: Functions can have an arbitrary number of arguments. In this case there is only one such argument: x. The arguments are being passed by value: The variable that the calling code passes to the function is copied and the copy is being given to the function. This means that if the function internally changes the value of x, the change will not reect on the variable that you use as argument on your main calling code. Only the copy will be changed, and that copy will be discarded as soon as the function completes its call. Of course it is not desirable to only pass things by value. The function has to communicate some information back to the calling code. In Matlab, the variables on the right hand side, listed in brackets, are also being passed to the function, but this is done by reference: The function is not given a copy of the variables but it is actually given the variables themselves. This means that that any changes made to those variables while inside the function will reect on the corresponding variables on the calling code. So, if one were to call the function with: a = 1:20; m = 0; s = 0; [m,s] = stat(a); 17

254

then the values of the variables m and s would change after the call to stat. The lines afterwords are comments. However, these comments are what will be spit out if you type >> help stat on your prompt. You usually want to make the rst comment line be a summary of what the function does because in some instances only the rst line will get to be displayed, so it should be complete. Then in the lines afterwords, you can explain how the function is meant to be used. After you are done with documentation you type in your usual Matlab code that will implement the function. The main problem with Matlab function denitions is that you are forced to put every function in a separate le, and are even restricted in what you can call that le. Another thing that could cause you problems is name collision: What if the name you choose for one of your functions happens to be the name of an obscure built-in Matlab function? Then, your function will be completely ignored and Matlab will call up the built-in version instead. To nd out if this is the case use the which command to see what Matlab has to say about your function. Many of the examples we saw earlier would be very useful if they were to implemented as functions. For instance, if you commonly use Matlab to manipulate data that come out in (x, y ) pairs you can make our earlier example into a function like this: function [x,y] = load_xy(filename) % load_xy -- Will allow you to load data stored in (x,y) format % Usage: % Load your data by saying % [x,y] = load_xy(filename) % where filename is the name of the file where the data is stored % and x and y are the vectors where you want the data loaded into fd = fopen(filename); A = fscanf(fd,%g %g\n,[2,inf]); x = A(1,:); y = A(2,:); fclose(fd); You would have to put this in a le called load xy.m of course. Suppose that after making some manipulations you want Matlab to save your data on le again. One way to do it is like this: function save_xy(filename,x,y) % save_xy -- Will let your save data in (x,y) format. % Usage: % If x and y are vectors of equal length, then save them in (x,y) 18

255

% format in a file called filename by saying % save_xy(filename,x,y) fd = fopen(filename,w); A(1,:) = x; A(2,:) = y; fprintf(fd,%g %g\n,A); fclose(fd); Notice that it is not necessary to use a for loop to fprintf or fscanf the data one by one. This is explained in detail in the on-line help pages for these two commands. In many other cases Matlab provides ways to eliminate the use of for-loops and when you make use of them, your programs will generally run faster. A typical example is array promotion. Take a very simple function function y = f(x) which takes a number x and returns another number y. Typical such functions are sin, cos, tan and you can always write your own. Now, if instead of a number you plug in an array or a matrix, then the function will be applied on every element of your array or matrix and an array of the same size will be returned. This is the reason why you dont have to specify in the function denition whether x and y are simple numbers, or arrays in the rst place! To Matlab everything is a matrix as far as functions are concerned. Ordinary numbers are seen as 1x1 matrices rather than numbers. You should keep that in mind when writing functions: sometimes you may want to multiply your x and y with the .* operator instead of the * to handle array promotion properly. Likewise with division. Expect to be surprised and be careful with array promotion. Lets look at an example more closely. Suppose you write a function like this: function x = foo(y,z) x = y+z; Then, you can do the following on the Matlab prompt: >> disp(foo(2,3)) 5 >> a = 1:1:10; >> b = 1:2:20; >> disp(foo(a,b)) 2 5 8

11

14

17

20

23

26

29

What you will not be allowed to do is this: >> a = 1:1:10 >> b = 1:1:20 >> disp(foo(a,b)) ??? Error using ==> + Matrix dimensions must agree.

19

256

Error in ==> /home/lf/mscc/matlab/notes/foo.m On line 2 ==> x = y+z; The arguments a and b can not be added because they dont have the same size. Notice by the way that we used addition as our example on purpose. We challenge you to try * versus .* and see the eects! One use of functions is to build complex algorithms progressively from simpler ones. Another use is to automate certain commonly-used tasks as we did in the example of loading and saving (x, y ) pairs. Functions do not solve all of the worlds problems, but they can help you a lot and you should use them when you have the feeling that your Matlab program is getting too long and complicated and needs to be broken down to simpler components.

20

257

258

Algorithms with Matlab

Eleftherios Gkioulekas Mathematical Sciences Computing Center University of Washington December, 1996

Introduction

Numerical analysis is the branch of mathematics whose goal is to gure out how computers can solve problems in a way thats fast, ecient and accurate. Linear algebra is a large part of numerical analysis, because many problems eventually reduce to one of the following linear algebra problems: We want to solve a system of linear equations. We want to solve an eigenvalue problem. For this reason, researchers think of these problems as elementary the way you would think of addition and multiplication elementary enough to use a calculator. Likewise, researchers will use a calculator in some sense of the word. Matlab can serve as such a calculator. Beyond Matlab you have the option of using software libraries. Such libraries exist for most commonly used languages like C++ (e.g. LAPACK++) and FORTRAN (e.g. LINPACK, EISPACK). For more information about the latter visit the Netlib repository at http://www.netlib.org/ or send email to netlib@ornl.gov with one line that says send index. In this tutorial we are assuming that you have read and understood the Programming with Matlab tutorial. The purpose of this tutorial is to review the mathematical concepts of linear algebra to give you a feel for the big picture and at the same time show you how you can experiment with these concepts using Matlab. As you will see, these problems are not quite elementary

Basic concepts

A complex matrix A of size n m is a mapping A : {1, . . . , n} {1, . . . , m} C where C is the set of complex numbers. In other words, a matrix will take two integers i and j such that 1 i n and 1 j m and return back a complex number. That number we denote with aij (we use the lowercase letter). The set of all n m complex matrices is denoted as C nm . If we restrict ourselves to real values only, then we are dealing with real matrices and the set of these is 1

259

denoted as Rnm . Finally, for the sake of notation, we will denote the set of all integers from 1 to n with the symbol: [n] = {1, . . . , n} In Matlab all data structures are matrices. In particular ordinary numbers are 1 1 matrices, vectors (or arrays in general) are 1 n matrices. The Programming with Matlab tutorial has covered the basics of how all these data structures are setup in Matlab. There exist various operations dened for matrices: Denition 1 (a) Let A, B C nm be two matrices. The sum and dierence of these matrices is: C = A + B cij = aij + bij , (i, j ) [n] [m] C = A B cij = aij bij , (i, j ) [n] [m] (b) Let A C nm and B C mp . The product of these matrices is
m

C = AB cij =
k=1

aik bkj , (i, j ) [n] [p]

(c) The transpose of a matrix A C nm is: B = AT bij = aji , (i, j ) [n] [m] (d) The hermitian or complex conjugate transpose is: B = AH bij = a ji , (i, j ) [n] [m] Moreover, there are the following important classes of matrices that we will see mentioned later on: Denition 2 A matrix A C nn is: (a) symmetric A = AT (b) hermitian A = AH (c) orthogonal AAT = AT A = I (d) unitary AAH = AH A = I (e) normal AAH = AAH For real matrices, hermitian means the same as symmetric, unitary means the same as orthogonal, and both of these distinct classes are normal. In Matlab matrices that are stored in A and B can be added or multiplied quite simply by saying: C = A + B; C = A - B; C = A * B; 2

260

The transpose and the hermitian of a matrix can be obtained by B = A.; B = A; % transpose % hermitian

The usual rules apply to these operations except for the following: Multiplication is not commutative for all matrices. That is there exist matrices such that AB = BA Square matrices dont always have a multiplicative inverse. A multiplicative inverse is dened by: B = A1 AB = BA = I The rst exception is not very interesting, but the second one is related to one of the two Big Questions in linear algebra: Solving linear systems of equations.

Matrix inversion is not easy

Suppose that A C nn is a square matrix and x, b C n1 are vectors (actually, column matrices). If A and b are known, then we want to nd an x such that Ax = b If A has an inverse then x is unique and given by: x = A 1 b otherwise x will either not exist, or will not be unique. In order for a matrix to have an inverse, the matrix has to be square. Given that, the existence of matrix inverses is determined by a quantity called determinant. The determinant is dened in terms of permutations so we must explain what these are rst. A permutation Sn of order n is a bijection that maps: : [n] [n] The set of permutations of order n is written as Sn and it has n! elements. In simple terms, take an ordered set of integers say (1, 2, 3, 4, 5) and reorder it to (4, 1, 3, 2, 5). Our concept of how we reordered an ordered set of things is what is being modeled by permutations. There are as many permutations as there are ways in which we can reorder things. One way to represent permutations is by showing reorderings of (1, 2, . . . , n). For example the permutations of order 3 are: S3 = {(1, 2, 3), (2, 3, 1), (3, 1, 2), (1, 3, 2), (3, 2, 1), (2, 1, 3)}

261

Permutations have an important property called parity. If Sn is a permutation then we dene the parity to be equal to:
n1 n

s( ) = sign
j =1 i=j +1

( (i) (j ))

When the parity is +1 we talk of even permutations. When it is -1 we talk of odd permutations. You can verify that the even permutations in S3 are {(1, 2, 3), (2, 3, 1), (3, 1, 2)} and the odd ones are {(1, 3, 2), (3, 2, 1), (2, 1, 3)}. Given these denitions, the determinant of a matrix A C nn is dened by:
n

det(A) =
Sn

s( )
i=1

ai,(i)

Unfortunately, this is not the most useful denition in terms of numerics. There are n! terms to add, and every one of these terms involves n multiplications. This will overwhelm any computer soon enough, not to mention round-o errors. You have probably seen many recursive denitions of the determinant in other courses. Those are not helpful either because they also involve work that increases by n!. Most computations of determinants are based on the following theorem: Theorem 1 Let A, B C nn be two square matrices. Then, det(AB) = det(A) det(B) We write our matrix as a product of matrices whose determinants are easy to compute and apply the theorem. More on this later. Now the reason why determinants are important is because of this theorem: Theorem 2 Let A C nn be a square matrix. A1 exists det(A) = 0 Notice by the denition that the determinant varies continuously as you vary one of the elements of the matrix. This means that determinants are quite reasonable functions of n2 variables which has the following implication: Most of the time, the determinant will be quite nonzero and an inverse will exist. Sometimes, the determinant will be very close to zero and an inverse will exist but will be so sensitive to the original matrix that it will be hard to compute. In this case we say we have an ill-conditioned matrix. Finally, on very rare matrices the determinant will be exactly zero and then the inverse just doesnt exist. In this case we have a singular matrix. These last two cases cause trouble from a numerical point of view. To make matters worse, deciding numerically which case is which is not trivial either. In Matlab computing determinants and inverses may appear innocuously simple. If A is your matrix then 4

262

x = det(A); will compute the determinant and B = inv(A); will return the inverse. However, for ill-conditioned matrices, inv will not give you the correct inverse. Sometimes Matlab will detect this, but not always. There exists a class of square matrices H(n) called the Hilbert matrices that are dened by: hij = 1 i+j1

The matlab command hilb will return a Hilbert matrix of any size. That is: A = hilb(10); will return a Hilbert matrix of size 10. These matrices have two properties: Inverting them is very sensitive to oating point error but there is an analytic equation that gives their exact inverse! The inverse is given by rij ij = h ji = h , 1 i j n i+j1 where rij is dened for j > i by the following recurrences:
2 ri,i = pi rij =

(n j + 1)(n + j 1) (n i + 1)(n i 1) ri,j 1 , j > i pi = pi1 , 1 i n 2 (j 1) (i 1)2

p0 = n

This computation is being done by the invhilb command. For example, for n = 10 you would say: A = invhilb(10); We can use these matrices to show you the limitations of the inv command. Try the following program: for i = 1:20 x = max(max(abs(invhilb(i)-inv(hilb(i))))); fprintf(1,For size i = %d deviation is x = %g \n,i,x); end This program will progressively increase the matrix size and attempt to invert the matrix. Then it will compare it with the exact known inverse and report the worst value of the element by element dierence. The output of the program is: For For For For size size size size i i i i = = = = 1 2 3 4 deviation deviation deviation deviation is is is is x x x x = = = = 0 3.55271e-15 6.82121e-13 5.96629e-10 5

263

For For For For For For For

size size size size size size size

i i i i i i i

= = = = = = =

5 deviation is x = 1.43918e-08 6 deviation is x = 0.000447804 7 deviation is x = 0.436005 8 deviation is x = 35.4408 9 deviation is x = 395134 10 deviation is x = 3.657e+08 11 deviation is x = 3.17671e+11

Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND = 3.659249e-17 For size i = 12 deviation is x = 2.81767e+14 [....etc....] Notice that the deviation managed to increase all the way to 3.657e+08 before Matlab started giving out warnings. Not good! There are two or three sets of errors involved here: The errors caused by representing hilb(n) The errors caused in the matrix inversion process The errors, if any, in representing invhilb(n) It turns out that the rst of these, which involves representing fractions like 1/3 and 1/5 in oatingpoint, is the most signicant. This error is propagated throughout the matrix inversion and is signicantly amplied. The moral of the story: Sometimes computers dont answer the question you think you are asking them. You think you are asking the computer to nd the inverse. When you use the inv function, you are asking it to apply an algorithm which you believe will give you the inverse. Exercise 1 Write a Matlab function that implements the recurrence formula for the inverse Hilbert matrix. Compare your results with the invhilb command. Exercise 2 Can we trust the det function in evaluating the determinant for the Hilbert matrix? Look at the errors in computing: det(hilb(n))*det(invhilb(n)) and det(hilb(n)*invhilb(n)) Which is more accurate? Why?

Algorithms related to solving linear systems

After spending all this time talking about inverses, we need to tell you a little secret: we rarely ever compute inverses explicitly, because even if we did, multiplying them with other vectors or with the original matrix itself is susceptible to many oating point errors. In most problems our main goal is to solve the system Ax = b and we can do this without computing the inverse. The purpose of our earlier discussion was to alert you that there are diculties we have to overcome: its hard to compute determinants, so

264

Cramer rule is out of the question; matrices can be nasty and matrix inversion can be error-prone. Now we come to the techniques that help us deal with these diculties. The techniques are quite involved, and Matlab wants to hide the complexity for the user. Therefore, Matlab uses the backslash operator to do the job x = A\b; This will work provided that the matrix A is well-behaved. However, in order for Matlab to be exible, they provide you with further access to particular techniques that you may want to use to do matrix inversions by yourself.

4.1

LU decomposition

One technique is with the LU decomposition. The idea here is that we write A as the product of a lower triangular matrix L with 1 in the diagonal, and an upper triangular matrix U. Then we reduce our problem to the following two systems of equations: Ax = b LUx = b Ly = b Ux = y

These systems can be solved directly without having to nd the inverse. As a matter of fact, given the LU decomposition of a matrix you can nd the inverse of A by rewritting the equation AA1 = I as n linear systems involving the unknown columns of A1 . Also, since the diagonal elements of L are 1 and since U is upper triangular, the determinant of A can be determined by:
n

det(A) = det(L) det(U) = det(U) =

i=1

uii

The work involved in computing the LU decomposition increases by O(n3 ) which is a vast improvement over O(n!). In Matlab, the LU decomposition can be computed with the command: [L,U] = lu(A); Unfortunately, there is a wrinkle with this method as well as with the matrix L returned by Matlab that has to do with the LU decomposition existance theorem. That theorem states: Theorem 3 Let A C nn be a square matrix and Ak C kk be the k k submatrix of A containing the upper left part. Then, A has an LU decomposition k [n] : det(Ak ) = 0 There are many cases where the condition of this theorem will almost fail: one of the submatrix determinants will be close to zero. The workaround is to permute the rows and columns of A in 7

265

such a way so that these determinants turn out okey. When you do that you essentially nd the LU decomposition of a new matrix PA where P is a permutation matrix. Permutation matrices have one 1 located in each row and column and when they are applied to another matrix, they permute the other matrixs rows or columns. Since permutations are easy to undo, P is yet-anothermatrix thats easy to invert: Its inverse is P1 = PT . When you run the matlab function lu as quoted above, it will not really return an actual lower triangular matrix in L. Instead it will do the decomposition PA = LU and return back P1 L and U!. If you dont like that, you can call lu like this: [L,U,P] = lu(A); Now, L will be an actual lower triangular matrix and P will be a permutation matrix, and the actual decomposition will be A = PT LU The solution to our system of linear equations then is: x = U1 L1 Pb Now we have a problem: Matlab doesnt have documented facilities for applying L1 and with the well-known back/forward-substitution algorithms. Typically, if you want to solve a system of equations with this method , the backslash operator will do it all in one scoop. However, suppose that you want to solve many systems of linear equations in which A is the same and only b vary. Then, you can have a more ecient program going if do the LU decomposition once and use the L,U,P arrays on each dierent vector b. You can write your own function to do that. Then again, if eciency is such a concern, then perhaps its time to switch to a programming language. U1

4.2

QR decomposition

Another technique is the QR decomposition. This method will decompose A to an orthogonal matrix Q and a right-triangular matrix R. That Q is orthogonal means that: Q1 = QT which makes it easy to invert. Also R can be easily inverted being a right-triangular matrix, so by rewriting a linear-systems problem as Rx = QT b we can solve it, following the footsteps of the LU technique. You can also compute determinants. It turns out that det(Q) = 1 so det(A) = det(R) =
i=1 n

rii

266

The QR method requires twice as many operations as LU, so LU is more commonly used. However, as you will learn in your numerical methods course, there are many other instances where QR decompositions can come in handy. One application of QR decompositions is in obtaining an orthonormal basis to a vector space. If you think of the columns of A and the columns of Q as vectors, then both sets of vectors span the same space. The dierence is that the vectors obtained through matrix Q are orthogonal with one another. To do a QR decomposition in Matlab call: [Q,R] = qr(A); This will produce two matrices Q and R such that A = QR. Matlab provides a few variations to this call: [Q,R,P] = qr(A); will produce a permutation matrix P, and return in Q and R the QR decomposition of AP. The permutation matrix is chosen so that the absolute values of the diagonal elements |rii | is decreasing as i increases.

4.3

Cholesky decomposition

When a matrix A Rnn is symmetric and positive denite it has a more ecient triangular decomposition. Symmetric means that aij = aji , i, j [n] and positive denite means that xT Ax > 0x Rn1 If both conditions are true, then A can be decomposed to A = RT R where R is an upper triangular matrix and RT is the transpose. In complex matrices, you must use the complex conjugate transpose instead. The Cholesky algorithm will attempt to compute this decomposition. The algorithm can self-detect when it fails, and in practice this ability is used to establish whether symmetric matrices are positive denite. In Matlab, you can invoke the Cholesky algorithm with the chol command: R = chol(A); If the algorithm fails, Matlab will issue a warning. You can suppress the warning by calling chol like this: [R,p] = chol(A) If p is non-zero, then this means that the cholesky algorithm failed and the original matrix is not positive denite. Positive deniteness is an important property. Many theorems in linear algebra are known for positive denite matrices, and many times it is useful to be able to decide numerically whether a symmetric matrix is positive denite. Also, when we know that the matrix A in a system Ax = b is positive denite, we can use this method to solve the system; it would be faster than LU. 9

267

4.4

SVD decomposition

To conclude, another very useful decomposition is the SVD decomposition. The theorem behind SVD decompositions states: Theorem 4 If A Rmn then there exist two orthogonal matrices U Rmm and V Rnn such that UT AV = diag (1 , 2 , . . . , p ) Rmn where p = min{m, n} and 1 2 . . . p 0. To compute the SVD decomposition of a matrix A in matlab you use the svd function as follows: [U,S,V] = svd(A); where S will be a diagonal matrix containing the i . If you only want to look at the i , then do s = svd(A); This call will return an array containing just the i . Given the SVD decomposition, the inverse of a square nonsingular matrix A can be computed with: A1 = V [diag (1/j )] UT and the solution to a linear system Ax = b becomes: x = V [diag (1/j )] UT b The reason why SVD is important is because it can diagnose two pathologies: ill-conditioning and singularity. That you can tell from the condition number which is dened by: cond(A) = max{i } min{i }

When the condition number is too large, i.e. close to 1/ where is the oating point accuracy, then we say that the matrix is ill-conditioned. When this happens, merely computing the equation above for the inverse is very susceptible to oating point errors. One remedy is to increase the oating point accuracy to that needed. However, there is a theorem that suggests a better approach. That theorem says that if you take those 1/i that are really bad and replace them with zero and then do the equation above as if nothing is wrong, then you can get a much more accurate answer than both the SVD solution and a direct method; accurate in the sense that the residual: r = |Ax b| will be much smaller than the numerical result yielded by a direct method or the ordinary SVD method. It takes some discretion to decide which 1/i to kill, and whether you get the desirable accuracy in the residual. A recommended tolerance is to eliminate those i such that i < max{m, n} max{i } 10

268

where m and n are the size of A, and the oating point precision. The matrix A1 obtained by the expression above in which the small i have been eliminated is called the pseudoinverse matrix. You can use the pinv command to compute the pseudoinverse using the SVD method. One way to invoke this command is by saying: B = pinv(A); in which case the tolerance mentioned above is used. Alternatively, you can specify your own tolerance tol by invoking the pinv command like this: B = pinv(A,tol); It is important to understand that the pseudoinverse matrix is not really the same as the inverse matrix. The point is that in an ill-conditioned problem, we are going to get a more accurate answer if we use the pseudoinverse, rather than if we attempt to compute the inverse and use that instead.If, in the worst case scenario, the residual we get by this method is still not good enough then your other option is to increase your oating point precision. To do this, assign the variable eps in matlab to your desired precision. For example, to get quad precision set: eps = 1e-32; When the condition number is innite, that is when some i = 0 exactly, the matrix has no inverse. Unfortunately, numerics being what it is, if you have an actual singular matrix, you probably wont see exact zeroes because of round-o error. So, it takes some discretion to decide whether you are dealing with a singular matrix or an ill-conditioned matrix as well. Matlab has a special command for estimating condition numbers: c = cond(A); The cond will compute the condition number. There are two other algorithms for condition estimation that can be invoked by the commands condest and rcond. See the help pages for more information. The orthogonal matrices U and V can also give us their share of information. If the n columns of A are vectors spanning a vector space of interest, then the columns of U will contain an orthogonal set of vectors spanning the same space. Nothing special so far, since you can do this with the QR decomposition too. The bonus is that with SVD you can detect whether the dimensionality of the vector space is the same as the amount of vectors that you use to span the space. If any zeroes (or nearly zeroes) occur in one of the i then this means that the vector space was not in fact n-dimensional. To deal with that case, you just eliminate the columns in U that correspond to the ill-behaved i . The vectors in U that remain will be an orthogonal basis for the vector space. An equivalent way of stating the previous paragraph is to say that SVD can be used to determine the rank of the matrix A. Rank is the dimensionality of the vector space spanned by the columns of a matrix. 1 We say that a square matrix A Rnn is full rank if it has rank n.
it doesnt actually make a dierence if you pick the rows of the matrix. A theorem tells you that you will get the same number, always
1

11

269

Many theorems in linear algebra apply only on matrices that are full-rank, so from the numerical perspective we want to have a way to diagnose matrices that are not full rank. SVD provides us with a method to do that. In addition to rank, we can also determine the null space of a matrix A C nm . The null space is dened to be null(A) = {x C m1 : Ax = 0} that is, the set of vectors x for which Ax is zero. For non-singular matrices, the null-space consists only of x = 0. For matrices of rank(A) = r < m the last m r columns of V give an orthonormal basis for the nullspace. Finally for matrices of rank(A) = r > m the null space is null(A) = {0} These concepts become useful when you are trying to solve linear systems Ax = b in which A C nm is no-longer a square matrix. In such cases, if A actually has a null space, to nd the set of x that satisfy the linear system, we nd one particular solution x0 . Then full solution set becomes: {x Rm1 : Ax = b} = {x + x0 : x null(A)} In practice, we dont care to know the full solution space. Instead we want to nd the unique x that minimizes the residual r(x) = |Ax b| The backslash operator will solve this problem. To nd the x that minimizes the residual do: x = A\b; Exercise 3 Write a matlab function which will accept a set of vectors spanning a linear space in the form of a matrix, and optionally a threshold for the i . Your function should return an orthogonal set of vectors that span the same space, but take care to remove the vectors whose i falls bellow the threshold.

Eigenvalue Problems

A completely dierent type of problem that shows up in many forms is the eigenvalue problem. Let A C nn be a square matrix, x C n1 be a vector and C be a complex number. We want to nd the pairs (, x) that satisfy the linear equation: Ax = x The set of for which corresponding x exist are the eigenvalues of the matrix. The x that correspond to each eigenvalue form the corresponding eigenvector space. Individual members of that space are called eigenvectors. Finding the eigenvalues, and then from these the eigenvectors is the problem at hand. One approach to solving this problem is to nd the eigenvalues rst, and then from them, nd the eigenvectors. To nd the eigenvalues we use this theorem: 12

270

Theorem 5 Let A C nn and C . is an eigenvalue of A det(I A) = 0 It turns out that if you expand that determinant, you obtain a polynomial
n

p() = det(I A) =
k=1

ak xk

which is called the characteristic polynomial, so to nd the eigenvalues we need nd the roots of that polynomial. For matrices up to size n = 4 there are direct methods that will give you the solution. So for such matrices, the eigenvalue problem is trivial. For larger polynomials there are no direct methods, and in many cases there is no simpler way to solve polynomial equations except by reducing them to eigenvalue problems! In fact, it is quite common to solve even 3rd order and 4th order polynomials this way. In Matlab we usually solve eigenvalue problems with the eig function. The call: e = eig(A); will return a vector containing the eigenvalues of A. The call: [V,D] = eig(A); produces a diagonal matrix D of eigenvalues and a full matrix V whose columns are the corresponding eigenvectors. The grand strategy of all eigenvalue algorithms is based on the fact that the matrix Z1 AZ has the same eigenvalues as A for all non-singular Z. What we want to do then is to apply the appropriate sequence of Z matrices that will diagonalize our matrix A. Then, the diagonal elements of the diagonal matrix will also be its eigenvalues. Of course, this is easier said than done. In what follows, we will sketch out how this is done, and introduce you to all the related nifty Matlab functions. You can use these functions to operate on a lower level, and experiment with the eigenvalue algorithms more directly.

5.1

Hessenberg Forms

The rst thing we want to do to our matrix A is bring it to the so-called Hessenberg Form. The Hessenberg form of a matrix is zero bellow the rst lower subdiagonal and non-zero everywhere else. The existence theorem for Hessenberg forms says that: Theorem 6 If A C nn then there exists a unitary P C nn such that PH AP = H where H is such that hij = 0, (i, j ) : i > j + 1

13

271

The theoretical emphasis here is that we can bring our A to this funky form by applying a sequence of unitary Z matrices. In your numerical analysis course you will learn that this guarantees the numerical stability of these repeatitive transformations. In Matlab, you can get this far by applying the hess function: H = hess(A); will return the matrix H. If you also want to look at P, call: [P,H] = hess(A);

5.2

Schur Decompositions

The next step is to converge to the Schur decomposition. Here are the relevant existence theorems: Theorem 7 If A C nn then there exists a unitary Q C nn such that QH AQ = T = D + N where D = diag(1 , 2 , . . . , n ) and N C nn is strictly upper triangular. Furthermore, Q can be chosen so that the eigenvalues i appear in any order along the diagonal. This decomposition is called the Complex Schur Form. There is another one for real matrices: Theorem 8 If A Rnn then there exists an orthogonal Q Rnn such that

Q AQ =
T

T11 T12 0 T22 . . . . . . 0 0

.. .

T1 m T2 m . . .

Tmm

where each Tii is either a 1 1 matrix which is an eigenvalue of A or a 2 2 matrix whose eigenvalues are also eigenvalues of A. This one is called the Real Schur Form. Matlab provides with a function called schur. The call [Q,T] = schur(A) will return the real schur form if the matrix is real, and the complex schur form if the matrix is complex. If you dont want the real schur form, then the function rsf2csf will convert the inputted real schur form to complex: [Q,T] = rsf2csf(Q,T); How do we get to Schur from Hessenberg form? The simplest way to do it is with the so called QR algorithm: 14

272

1. First of all, we get the Hessenberg form of A: H0 = P T 0 AP0 2. Then we apply the following recurrence: Hk = Qk Rk QR decomposition Hk+1 = Rk Qk It can be proven that these repetive iterations are actually equivalent to applying unitary matrices for Z and that they will take the Hessenberg form all the way to Real Schur form! In your numerical analysis course you will learn more details about the various algorithms that take you from Hessenberg to Schur. With Matlab you can actually experiment with many of these algorithms and see how they perform.

5.3

Balancing

Before applying the QR algorithm it is a good idea to balance the matrix. By balancing we rescale the matrix by transforming it to D1 AD where D is a diagonal matrix so that it it has approximately equal row and column norms. In matlab, you can balance a matrix with the balance function: B = balance(A); While experimenting with variations of the QR algorithm, you can see what happens as you include or not include balancing.

5.4

Miscellaneous

With symmetric matrices, the following miracles happen: the eigenvalues are real, the Hessenberg form is a tridiagonal matrix, and the QR algorithm doesnt merely take us to a real Schur decomposition, but it completely diagonalizes our matrix! An even more exciting development is that for square symmetric matrices the Schur decomposition and the SVD decomposition are the same thing! From this fact, plus a few theoretical results, one can obtain an algorithm for computing the SVD by reducing it to a symmetric eigenvalue problem. The details of this are described in Matrix Computations by Gene Golub. Another application of eigenvalue problems is solving polynomial equations. As you may know, direct methods for nding polynomial roots, exist only for polynomials up to 4th degree. Let
n1

P (x) =
k=1

ak xk + xn

15

273

be your polynomial. Form the matrix:

C=

0 0 1 0 0 1 . . .. . . . . . 0 0

0 0 0 . . .

a0 a1 a2 . . .

1 an1

The eigenvalues of C are the roots of P (x). C is called the companion matrix of the polynomial. In matlab, if an array P holds the coecients of a polynomial with p(1) being the highest order coecient, the companion matrix can be retrieved by the compan function. A = compan(p); Of course Matlab, in its never-ending eorts to make this all easier for the user, provides you with a function called roots that you can use to do all this automatically. So, if you describe your polynomial with an array p such that y = p(1)*x^d + p(2)*x^(d-1) + ... + p(d)*x + p(d+1) then the roots can be obtained by simply saying: r = roots(p); You will obtain an array of roots in r. To verify your roots, you can use the polyval function to evaluate the polynomial. Just call: y = polyval(p,x); where p is the array with the polynomial and x is a scalar. Finally, Matlab has commands that allow the user to solve more generalized eigenvalue problems. One such problem is the following. Given two square matrices A, B C nn nd C and x C n1 such that Ax = Bx. To solve this problem call: [V,L] = eig(A,B); l = eig(A,B); % eigenvectors -> V, eigenvalues -> L % eigenvalues -> l

In the rst version the eigenvectors are returned as columns of V and the eigenvalues as diagonals of L. In the second version an array of the eigenvalues is returned.

16

274

275



!"#$%&'

$())&'0&132!45!6

7897A@CBEDF7GEHPIRQSRTRHPIVUF897ATRWVQ X)Ya`bacedfbhgpif`grq'stYCuviwYAxryCCgpcCsuvistgifcC##xdtu`sYC`stYC`0gpdfgrq'stYC`%#xrsfdtu`9bhg y `ystuxredgfhgpc#xiyCgrst`wstYaxrsjstYauiguikx&Culh`dt`0ymskxbCbCdtgmxrnYoqpdtgp1stYC`gpya`3badtgrbhgpif`0ouy C pgpcCdjst`9sfqhgrgrsdXtYC`3xruy#grstuvurxstugyqpgrdjbCdt`qP`dfduyCwstYC`3#xsfdu(`x9bhgpya`0ysuvxrhuitstYaxs uvs'bCdtgyuruC`0itxcCyCuza`0(sYC`0grdf&stYaxrsjnxryoifgpvup`{r|p|RbCdtgpqa`0!i0}auyCn0vcCCuyCw!xpe~rftbadtgrqe `!idAgrdk`xCxrba`i}hif`0`dtgpyCitgpydwC`sxruv`0`9bhgpifustugpyogrqstYC`wstYa`0grdfuvi'wuvup`0y q9%tbhgpisgpyd `kxrymstsgifgpvup`x!uyC`xrdif9is`0grqRgrdtauyaxdfCulh`dt`0ymstuvxr`09caxstuvgpyCigrqstYC`qPgrdt F 5 p C tYa`dt` uvi'xsryCgjyqPgrdtn0uvyCw!qPcCyCnsugpyh}AxpyC Ruigx##xrsfdtugrqngpyCisxpys'n0gr`qv zanu`0ymstidhX)Ya`k`0ifif`ystuxrdt`0ifcCs)grqVstYC`k#xrsfdtu`x9bh p g a y `0ysuvxristYC`0gpdf&uvistYaxststYC`ifgpvcestugpy uitwpuvup`0y%q9 `9b `xrb a `xrb ' tYa`dt` juigsYC`uyCuvstuxrEn0gyCCuvstuvgpyh}tYCunY`sryCgykdwXtYC`zCdtiss`dtdt`bCd`0if`0ymsti'stYC` YCg!gpwp`0ya`0gpcCiifgpcCstugpyh}9tYC`0dt`xristYC`gif`n0gpyC&st`ddt`bCdt`if`0ymstistYC`'bAxdfsun0cCvxrdifgpvcestugpyd X)Ya`k#xsfdu&`xrbhgpyC`0ymstuvxpAuitwpuvup`0y%q9 `xrb E R grst`stYAxs3sYCuiuixiwp`0yC`0dxruyxstugpyigrqjstYC`&Xxrgrd!if`du`0iw`x9bxryCifugygrqjstYC`isxpyCaxdt `xrbhgpyC`0ymstuvxpqpcCyanstugpyd xstcedxrvr} uijx ( #xsfdudX)Ya`bCdgrqa`0xsjYaxpyCuitstg `urxrcAxst`'stYC`#xrsfdtu`x9bhgpya`0ysuvxrCjuvstYCgpcCsYAx0uruyCwstg#`uxpcaxst`gxry(uvyezayCuvs`jifca%}CtYCuvnY uityCgrs)bCdxrnstunyxred dgtPCrEAwEapCaF r X)Ya``0uxrvcaxstugykgrqasYC`xsfdtu`9bhgpyC`ystuxrrdt`09cCuvd`0iEsYC`cCif`grqCstYa`ngpyCn0`basFgrqa`uwp`0yr urxrcC`ixryC&`0uw`0yup`nstgrdti0d `'itx!stYaxsxxsfdtu Yaxrih`uwp`0ymuxpcC`jtustY#`0uvwp`0ymup`0nstgrd ui!itxsuiza`0dXg6zAyCstYC`(`uwp`0ymuxpcC`0i}Rpgcifgpup`%stYC` })uqtstYC``0mcAxstugpy qPgpgytuyaww`mcaxrstugpy e C`s &stg xryCifgvup`'stYaxsstg3zayC X)Ya`0yh}rpgpcifceqaifstuvstces`)stYC`'y9cCqh`disYaxspgpc&zayC stYa`kn0grdfd`0ibhgpyCCuvyCw`0uvwp`0ymup`0nstgrdd X)Ya`wp`0g!`sfdtuvn'uvyst`0dfbCdt`sxstugpy&uisYaxs)stYa`k`0uwp`yup`0n0stgrddt`badt`0if`0ymsti)x#Cuvdt`0n0stugpyifcCnY stYAxsxpy6up`nstgrdwstYaxswbhgpuymstiksgxdtCi3stYaxrswCudt`0nstuvgpy grduystYC`&grbCbhgpifust`!Cuvd`0nstugyA tuvyCgrsqh`w{pt%tYC`ypgcxbCba#stYC`g#xrsfdtugpy%uvs0dVs)tugpyCisfd`stnY&gpd)ifYedtuyCsA} xryastYC`gn0grdfd`0ibhgpyCCuvyCw`0uvwp`0ymuxrvcC`)tuvst`0Apgpc(YCgywcCnYdqRxrstYa`'`uwp`0ymuxpcC`0ixdt`


276

!"#$

CuvistuyCn0s0}EstYa`0y6pgpc% dt`a`xruyaw%tuvstYx(up`0dfyCun`##xstdtu}Fqh`0nxpcCif`pgpctuvhxritgYaxup` n0gpdfdt`0ibhgpyaCuyCwi`0uw`0yup`nstgrdtisYaxs&xrdt`(grdfsYCgpwpgpyAxrstg`yxrnYgrstYC`0d0d XtYCui##`xryCi#stYaxs `up`0df%grstYC`dup`nstgrd)nxpyq`C`0n0gp!bgif`0xri & ' ( V tYa`dt`jstYC`ty9cC3qh`dti ( xdt`tstYC`'n0grgrdtCuvyaxst`0iRgpqhstYC`)up`nstgrdcaifuyCwstYC`'`0uwp`0ymup`0nsgrdti grqEstYC`g#xsfdu xpix3qpdxr!`ggrqEdt`qP`dt`0yCn`rdEsqpgpvgjihstYAxsjYC`0y&pgpc%xbCbastYC`g#xsfdu stg(xpyixrdfqauvsfdxdf%up`0nsgrd0}xrstYaxrs'YaxrbCbh`0yCi'uigstYaxsgpgp c % dt`3cavstuvba9uyaw!uvstin0grgrdCuyaxs`0i tustY!n0`dtsxruyn0gpyaisxrymsymcaqh`dti0}pstYC`)`uwp`0ymuxpcC`0i }dt`0wmxdtC`ifiVgrqAjYCgkstYauiVup`0nstgpdhuid X)YauihgpyavwYagpCiYag`up`d0) ~1p232465ek7)989A@CBrf0pD4'mFEG@g65CtyrfH)6@A{04I1P47Qd AcedfstYC`0dt!grdt`p}9sYCui#xqPxruvhuvqEpgpcoYaxup`kdt`bh`xrst`0&`0uwp`0ymurxrcC`0i0d dSRChT UT!VaEECvF r X)Ya`(!gpisn0gpymup`0yCu`0ymsst`0nYCyCu9cC`&qPgrd#`urxrcaxrstuyCwostYC`%#xrsfdtu `xrbgyC`0ymstuvxrhxriuyr up`0ymst`0qmXWcest0`0d0dsjuvi)qAxrit`0(gpystYC`kqPgpvgtuvyCwsg#nvxru!i0dRuvdis0}estYC`0dt`tuvExrvx0ri `xeuiswymcaqh`dti`YbaifcCnYsYaxskstYa`##xstdtu`x9bhgpya`0ysuvxrh nxryqh`!)dtusfst`0y6xrixzayCuvst` ifca wpuvup`yqm e  a c d Ya a X)Ya`0if`#y9cC3qh`dtiwnxryqh`#nyxrn0cCxst`0q9cCifuyaw%stYC`#qpgpvgjuyCw%n0vxpug fihecebCbhgpif`sYaxs` C`0zayC`kxbhgpvryCgp#uvxrwpuvup`0yq9 e  a p ' q Ya a X)Ya`0yh}ifcebCbhgpif`kstYaxsjpgpcs9yCgy xrsYC`3`uwp`0ymuxpcC`0igrqRsYC`3xsfdtu djXtYC`0yh}qpgpdj`xrnY `0uvwp`0ymuxrvcC`k(pgc(#xgpqCsxruygpya`k`09caxstugyxpiqpggytid ifubA``0uwp`yurxrcC` r t s q  w d CgpceqA``0uwp`yurxrcC` r t s q  qu  sfdtuba`g`0uwp`yurxrcC` r s q  q u  q uvu A xritx#ir ist`0%}ej uhwpuvup`g pgc(stYC`y9cC3qh`dty `snrx d hrgpvuruyCwwstYa`0if``09caxstuvgpyCistgpw`stYC`d i Yad X)Yauijuvigx&istdxruwpYmsfqPgrdfxdt(nyxrn0cCxstugpysYaxs'xpvxrijgrdfsri d 'g`up`d}hstYaxs'Cgr`0if" y %s !`yxrystYaxrskpgpc6juxrvx9i3Yax0up`#stg(Cguvs0d!yn0`0dfsxruyinxpif`0i}F`nxrygrdfsuvsgceskuy wp`yC`dxrxrya C`dtuup`%ibh`0n0uvxrdt`ifcCvstiwstYAxstu)xpgy pgpc stg6nuvdtn0cC3up`0ymsstYC`%yC``0stg ifgvup`stYC`i9ifst`0grqVuya`xd`09caxstugyCid Agrd`xexpba`r}uvqRpgpced#xrsfdtuuigyCuvn0``0yagpcCwpYstgYax0up` CuisuyCnst`uwp`0ymuxpcC`0i}astYC`0y stYa`k#xsfdu&`xrbhgpyC`0ymstuvxpAnxpyq`nxrn0cavxst`0#qdgp stYC`gqPgpvgtuvyCw`x9badt`0ififuvgpyCid `xrb s a a V d a a  e aP X)Yauidt`0itcCvsq9&uvsif`0vqn0gup`0dti)!gpifstn0uvdtncC!isxpyCn0`0id

Hb$ D

"g$bb$ #$b


 gpisFcCif`qPcCpuiEstYC` nxpif`rdEyksYCuinxpif`r}5`CuistuvyCwpcCuitYjqh`s`0`0ysgtbhgpififuvqAuuvstuv`07 if stYAxsstYa`#`0uw`0yurxrca`0ikxdt`CuistuvyCns0}EgpdstYaxrsstYC`0xrdt`#`09caxrdXhecebCbhgpif`sYaxskpgpcYaxup` qPgpcCyC(stYC``0uw`0yurxrca`0i}exryC%stYa`&xd`3 dh X)YC`0y}CuvqR stYC`0y `xrb s t qh   stYC`0y s `x9b s y s s s s vup`xr badtgrqa`!i# X)Yaui'dt`0ifcCsjuigif! c n0u`0ymststgifg d "k`0`biuyi!uyC}AstYAxsjjYC`0yopgpc sryCgystYC`g#xstdtu#`xrbhgpyC`0ymstuvxr}stYaxrsxrvgjipgcstgw)dtuvst`gCgytyifgpvcestugpyaihqPgrd)xrym#sruyC grqqPgrdtn0uyawqpcCyanstugpyaid % d $'&CPCFahF) ('C0 &EF g `scaihn0gpyaifuC`dRsYC`)!`sYCgrcCif`0q9stYC`)st`xrsfqhg9gpsAdX)YCui#`stYCgrui`9bh`0Cuv`0ys gpyav tYC`0y xrtstYC``uwp`0ymuxpcC`0i#xrdt`CuvistuyCn0sH { @Apgpc CgyCgrsYAx0up`qPgrdtn0uyaw6uypgpced bCdgrqa`0(dstuijxrifg!ngpyCn0`bastcaxruvyst`0dt`0istuvyCwed hrcCbCbhgpif`(sYaxs uvi&xry `0uw`0yup`nstgrd#gpqstYC`#xsfdtuv tuvstY `0uw`0yurxrca`ohd Yaxs YaxrbCbh`0yCi&uvqpgpcxrbCbav stYa`xsfdtu gpy stYC`i`0uw`0yup`nstgrd uyCis`xr 132jifuvyCwstYC` pgpco#xitYCg stYaxs C`0zayCuvstuvgpygpqsYC`#xsfdu&`xrbhgpyC`0ymstuvxpe} 5p `xrb s bCdt gpqa hrcCbCbhgpif`gstYaxrspgpc%xrysjstgifgpup`stYC`cCyCqpgrdn0`0 `0  Va X)Ya`jsp`0uistgw)dtust`tsYC`juvyCuvstuvxpCn0gpyCauvstugpyxrix3vuyC`xdn0gp3qauyaxrstugpygrqstYC`'`0uw`0yup`nstgrdti ( 2'ifuyCw!stYCui}e`k#x%nxrncCvxst`jstYC`itgpcestuvgpy%xriqPgpgyt7 if `x9b `xrb ( & ( `xrb ( `xrb Yaxspgpc#w`shuihstYC`)!`0stYCgr!grqstYC`)st`xrsfqhg9gpsA5 d 4upgpc&if`0`)Yag stYC`)xsfdtu`xrbhgpyC`0yr stuxraCuvitxbCbh`xd`0 1X)YC`juC`xuihsYaxy s ) ~pgpcCgp" y % sYaxup`jqPgrdtnuyCwe}mxpyC ) ~stYC`j`0uwp`0ymurxrcC`0i {pfCuvistuyCn0s0}AsYC`0yipgpc6nyxryn0uvdtncCup`ysgstYC`ngpbacesxstugpyogrqstYC`#xstdtuo`xrbhgpyC`0ymstuvxp q9%`uxpcaxstuvyCwsYC`A ) 8m7 @CBt0pI 4uvyCist`xph}rtYCunY%uitxry`xrifu`0d)n0gpbacCsxstugyhd 4gp y % s3qPgrdtwp`0s3stYAxswuswuiw`xrif6stgCuvxrwpyagpif`!jYC`stYC`0dpgpced!itgpcestuvgpyCixd`n0gpdfdt`0ns3qm ifcCqaistuvscestuyCw3stYC`qAxrns&sgstYC`kgrduwpuyaxpA`09caxsugpyCid 6 8 7rChCEa 9A@CB E G DGFIHQPSRUTQPSRVDXW`Y a5bdcebdfGghYhiqpsr0tvuiwghYhxwfGyCDr Q vU I SeAR Q vHTf S HQed D AgfhjiIPqHkf @vlqlwm D 9 nB o B DGpD5 W qGPTrPgDsiqptup uydDuwPSvRxwjAy#zS Q PSRUTvfUhjiIPwHkVf @vlq{wl D

277

278

References The following references were consulted during the preparation of these lecture notes. (1) A. Pistodes (1992), Algebra IV, unpublished lecture notes. (2) K. Gkatzoulis and M. Karamavrou (1988), Linear Algebra, Ekdoseis ZHTH. (3) T.M. Apostol (1969), Calculus, Vol. 2, Wiley. (4) J. Aldous and R.J. Wilson (2004), Graphs and Applications, Springer. (5) J.A. Bondy and U.S.R. Murty (1976), Graph Theory with Applications, Elsevier Science Publishing. Lecture notes by Pistodes are available for download at http://www.math.utpa.edu/lf/OGS/pistodes.html