Numerical Methods

Dr. Ali Fuat ALKAYA

Numerical Analysis/Methods
What is numerical analysis/method?
Analysis and design of algorithms for numerically solving mathematical problems in science and engineering

Why do we care about numerical analysis?

Simulation of real-world phenomena and events Virtual prototyping of engineering designs

Analysis vs. Numerical Analysis

Consider solving x2=2 Analytically, we know that is a root of the equation Numerically, how do we find the root of the equation using a computer program? Computer can only do arithmetic operations Design a procedure consisting of only arithmetic operations to find the root

Numerically Solving x =2
2

Other Examples
Science and Engineering convert a physical phenomenon into a mathematical model. Usually this leads to problems with no analytical solution:

Course Overview
Approximation and errors Solving nonlinear equations Solving sets of equations

Course Overview (cont.)

Interpolation and curve fitting Find intermediate values from a table of data Fit curves to data If the curve passes all data points, we call it interpolation.

Course Overview (cont.)

Approximation of functions with polynomials or ratio of polynomials Numerical differentiation and integration
approximate derivative values of a function approximate definite integral, even when no analytical form exists

Numerical solution of ordinary differential equations Optimization

Applications of Numerical Methods

Computer graphicsroot finding, interpolation, curve fitting, optimization, ODE solver, PDE solver, finite element method
Physics-based animation Geometry modeling

Computer visionoptimization, curve fitting, linear equations

Stereo vision Shape from shading

Applications of Numerical Methods (cont.)

Machine learningcurve fitting, linear equations, function approximation
Pattern recognition Neural network

Simulation for prototypingODE solver, PDE solver, optimization, numerical integration, interpolation, finite element method
Circuit design Mechanical design CAD/CAM

What is this course about?

This is not a course to teach you to code. This is a course to teach you computer algorithms for analyzing and solving science and engineering problems in numerical ways. Numerical analysis is the study of procedures for solving problems with a computer. Numerical analysis is always

TYPES OF ERRORS
Truncation error (finite speed and time) - An example:

Round-off error (finite word length): All computing devices represent numbers with some imprecision, except for integers. Human errors:
Mathematical equation/model. Computing tools/machines. Error in original data. Propagated error.

MEASURE OF ERRORS
The accuracy of any computation is always of great importance. Two ways to express the size of error in a computed result:
Absolute error Relative error

Floating-Point Arithmetic
A computer stores numbers as floating-point quantities that resemble scientific notation.
For example 13.524 is often displayed as .13524E2 Also -0.0442=-.442E-1 normalized (that is leading digit is made nonzero)

For storing floating-point numbers the IEEE standard is the most common.

Floating-Point Arithmetic (cont.)

A computer number has three parts
the sign (+ or -) the fraction part (called the mantissa) the exponent part

There are three level of precision and these are the number of bits used for Length Sign Mantissa Exponent Range mantissa and exponent.
Single Double Extended 32 64 80 1 1 1 23 52 64 8 11 15 1038 10308 104931

Floating-Point Arithmetic (cont.)

In 10-base system 156.78 could be represented as 0.15678x103 Also consider 1/34=0.029411765 .
Assume 4 digits to be stored. Then 1/34=0.0294x100. But we should normalize such that 0.2941x10-1

Consequence of normalization is that the absolute value of m (mantissa) is limited. That is (1/b)m<1 where b is the

A Hypothetical Example

MATLAB
Matlab is a software package that allows you to program the mathematics of an algorithm without getting too bogged down in the details of data structures, pointers, and reinventing the wheel. It also includes graphing capabilities, and there are numerous packages available for all kinds of functions, enabling relatively high-level

MATLAB
The basic Matlab data structure is a matrix; a scalar is a 1x1 matrix, a vector is an nx1 matrix. Vectors and matrices in Matlab are indexed starting from 1; and not from 0; as is more common in modern programming languages. Moreover, the last index of a vector is denoted by the special symbol end.

Programming in Matlab
If you are going to do any serious programming in Matlab, you should keep your commands in a file. Matlab loads commands from `.m' files. If you have the following in a file called myfunc.m:

function [y1,y2] = myfunc(x1,x2) % comments start with a `%' % this function is useless, except as an example of functions. % input: % x1 a number % x2 another number % output: % y1 some output % y2 some outpu y1 = cos(x1) .* sin(x2); y2 = norm(y1);

Programming in Matlab
then you can call this function from Matlab, as follows:
> myfunc(2,3) ans = -0.058727 > [a,b] = myfunc(2,3) a = -0.058727 b = 0.058727 > [a,b] = myfunc([1 2 3 4],[1 2 3 4]) a= 0.45465 -0.37840 -0.13971 0.49468 b = 0.78366

Control Structures in Matlab

%compute the sign of x if x > 0 s = 1; elseif x == 0 s = 0; else s = -1; end while (sin(x) > 0) x = x * pi; end %a `regular' for loop for i=1:10 sm = sm + i; end

%an `irregular' for loop for i=[1 2 3 5 8 13 21 34] fsm = fsm + i; end

Some Useful Commands in Matlab

help is the most useful command. floor(X) returns the largest integer not greater than X ceil(X) returns the smallest integer not less than X sin(X), cos(X), tan(X), atan(X), sqrt(X), returns the sine, cosine, tangent, arctangent, square root of X, computed elementwise. exp(X) returns eX. abs(X) returns |X| norm(X) returns the norm of X; if X is a vector, this is the L2 norm:

Some Useful Commands in Matlab

zeros(m,n) returns an mxn matrix of all zeros. eye(m) returns the mxm identity matrix. [m,n] = size(A) returns the number of rows, columns of A. Similarly the functions rows(A) and columns(A) return the number of rows and columns, respectively. length(v) returns the length of vector

Solving Nonlinear Equations

An important problem in applied mathematics is to solve f(x)=0 where f(x) is a function of x. The values of x that make f(x)=0 are called the roots of the equations. They are also called the zeros of f(x).

Interval Halving (Bisection)

It begins with two values for x that bracket a root. It determines that they do in fact bracket a root because the function f(x) changes signs at these two x-values and, if f(x) is continuous, there must be at least one root between the values. A plot of f(x) is useful to know where to start. The bisection method then successively divides the initial interval in half, finds in which half the root(s) must lie and reports with the endpoints of the smaller

Interval Halving (Bisection) cont.

Given a function f(x) in [a, b] satisfying f(a)f(b)<0, find a zero of f(x) in [a, b]. Step 0: Let xa := a; xb := b; n = 1. Step 1: Cut the interval in the middle, i.e., find xn = (xa+xb)/2 Step 2: Define Step 3: If xn is close enough to x0, stop. Otherwise, n:=n+1 & go to Step 1.

Interval Halving (Bisection) cont.

Advantages:
It is guaranteed to work if f (x) is continuous in [a, b] and a zero actually exists. A specific accuracy of iterations is known in advance. Few other root-finding methods share this advantage.

Disadvantages:
The estimate of root may be better at an earlier iteration than at later ones. It requires the values of a and b. The convergence of interval halving is very slow. Multiple zeros between a and b can cause

Example (Bisection)
Let f(x)=x2-1. Find its zero in [0, 1.5] Of course, we know f (x) has a root at x0=1. Let us find it using the Bisection Method: xa= 0, xb=1.5, n =1 x1=(0+1.5)/2=0.75 x2=(0.75+1.5)/2=1.125 x3=(0.75+1.125)/2=0.9375 x4=(0.9375+1.125)/2=1.03125 ...

Example (Bisection)
fx='3*x+sin(x)-exp(x)' fx = 3*x+sin(x)-exp(x) bisec(fx,0,1,13) 1.0000 0 1.0000 0.5000 0.3307 2.0000 0 0.5000 0.2500 -0.2866 3.0000 0.2500 0.5000 0.3750 0.0363 4.0000 0.2500 0.3750 0.3125 -0.1219 5.0000 0.3125 0.3750 0.3438 -0.0420 6.0000 0.3438 0.3750 0.3594 -0.0026 7.0000 0.3594 0.3750 0.3672 0.0169 8.0000 0.3594 0.3672 0.3633 0.0071 9.0000 0.3594 0.3633 0.3613 0.0023

function rtn=bisec(fx,xa,xb,n) x=xa;fa=eval(fx); x=xb;fb=eval(fx); for i=1:n xc=(xa+xb)/2;x=xc;fc=e val(fx); X=[i,xa,xb,xc,fc]; disp(X) if fc*fa<0 xb=xc; else xa=xc; end end

The Secant Method

This method begins by finding two points on the curve of f(x), hopefully near to the root we seek. A graph or a few applications of bisection might be used to determine the approximate location of the root. If f(x) were truly linear, the straight line would intersect the x-axis at the root. But not. So, the intersection of the line with the x-axis is not at x=r

The Secant Method (cont.)

In order to find x2, we can write an equation as follows: (x1-x2)/f(x1)=(x0-x1)/(f(x0)-f(x1)) x2=x1-f(x1)(x0-x1)/(f(x0)-f(x1)) Because f(x) is not exactly linear, x2 is not equal to r, but it should be closer than either of the two points we began with. If we repeat we have xn+1=xn-f(xn)(xn-1-xn)/(f(xn-1)-f(xn))

The Secant Method (cont.)

Advantage:
Convergence is fast and It does dot need derivative.

Disadvantage:
The method may fail if the function is far from linear and near the root the successive iterates can fly off to points far from the root.
r x0 x1 x2

False Position
To avoid the mentioned pathology is to ensure that the root is bracketed between the successive pairs. When this is done, this method is known as false position. This method is similar to bisection except that the next iterate is taken at the intersection of a line between the pair of x-values and x-axis rather than at the midpoint. This yields faster convergence, but with a more complicated algorithms.

Newtons Method
Like the previous ones, this method is also based on a linear approximation of the function, but does so using a tangent to the curve. tan =f(b)=f(b)/(b-x1)x1=b-f(b)/f(b)

Newtons Method (cont.)

Starting from a single estimate, b, that is not far from a root, we move along the tangent to its intersection with the x-axis and take that as the next approximation. This is continued until either the successive x-values are sufficiently close or the value of the function is sufficiently near zero. We continue the above equation by computing xn+1=xn-f(xn)/f(xn), n=1,2,

Newtons Method (cont.)

Advantages
More rapidly convergent than any of the methods discussed before. The method is quadratically convergent, that is the number of decimal places of accuracy nearly doubles at each iteration Starting point can be arbitrary

Disadvantage
Needs f(x).

Example (Newtons Method)

Let f(x)=x2-1. Find its zero in [0, 1.5]. Start with any initial value, say, 0.1

nd

Example

f(x)= 3x+sin(x)-ex Start with x0=0

x=g(x), Fixed-Point Method

Start from f(x)=0 and derive a relation x=g(x). The iteration xn+1=g(xn), n=0,1,2,3, converges to the fixed point r, a root of f(x). Example f(x)=x2-2x-3 x1,2=-1,3 x=g1(x)=2x+3
Start with x0=4 x1= 11=3.31662 x2= 9.63325=3.10375 x3= 9.20750=3.03439 x4= 9.06877=3.01144 x5= 9.02288=3.00381

x=g(x), Fixed-Point Method (cont.)

x=g2(x)=3/(x-2)
Start with x0=4 x1= 1.5 x2=-6 x3=-0.375 x4=-1.263158 x5= -0.919355 x6= -1.02762 x7= -0.990876 x8= -1.00305 converging to -1

x=g3(x)=(x2-3)/2
Start with x0=4 x1= 6.5 X2=19.625 x3=191.070 Seems diverging

x=g(x), Fixed-Point Method (cont.)

When does it converge? Convergence Theorem Consider a function f(x) and suppose it has a zero on the interval [a, b]. Also, consider the iteration scheme xn+1=g(xn) derived using fixed-point method. Then this scheme converges if the following condition are satisfied:
|g(x)|<1 for all x in [a,b]. g(x) is also said to be contraction in [a, b]. Start any initial point x0 in [a,b].

Remark: If the above conditions are not satisfied, the iteration scheme might still converge as the above theorem only gives sufficient conditions

Converge?Diverge
Converge Diverge

x=g(x), Fixed-Point Method (cont.)

Example
Solve f(x) = 0 by rewriting as x = g(x) and iterating xn+1 = g(xn) f(x) = x3 sin(x) = 0 (has 3 roots)

Example
Find x which satisfies x = ex. Can re-write this as log x = x

Summary of Methods
Bisection Method
Condition: Continuous function f (x) in [a, b] satisfying f (a) f (b) < 0. Convergence: Slow but sure. Linear.

False position method

Condition: Continuous function f (x) in [a, b] satisfying f (a) f (b) < 0. Convergence: Slow (linear).

Newton Method
Condition: Existence of nonzero f '(x) Convergence: Fast (quadratic).

Secant Method
Condition: Existence of nonzero f (xn+1) f (xn) Convergence: Fast (quadratic).

Fixed-point Method

Muellers Method

Multiple Roots
A multiple root corresponds to a point where a function is tangent to the x axis. For example
f(x)=(x-3)(x-1)(x-1)
1 3

f(x)=(x-3)(x-1)(x-1)(x-1)
1 3

Multiple Roots (cont.)

For multiple roots we can not use Bisection, False Position methods. Why? Because they are based on opposite signs around roots, but multiple roots does not guarantee this requirement. We can use Newtons method or Secant method to solve multiple roots. It is shown that Newtons method is linearly convergent for multiple roots. Modifications have been proposed to alleviate this problem.

Modified Newtons Method for Multiple Roots

Ralsten and Rabinowitz (1978) suggested to define a new function u(x) as u(x)=f(x)/f(x) This function has roots at all the same locations as f(x). Therefore we can write xi+1=xi-u(xi)/u(xi)

xi+1=xi- (f(xi).f(xi))/([f(xi)]2-

Example
f(x)=(x-3)(x-1)(x-1)

Solving Sets of Linear Equations

A matrix is a rectangular array of numbers in which not only the value of the number is important but also its position in the array. A matrix of n rows and m columns is said to be nxm. Double scripting is the common way of indexing elements. The first subscript denotes the row, second denotes the column in which the

Matrices and Vectors

A= =[aij] i=1,2,,n j=1,2,,n

Two matrices of the same size may be added or subtracted. Let A=[aij] and B=[bij], then C=A+B=[aij+ bij] = [cij].

Matrices and Vectors (cont.)

4 7 5 A= 4 2 12 1 5 4 B= 2 6 3

5 12 1 C = A+ B = 2 4 15

3 2 9 D = A B = 6 8 9

a11b11 + a12 b21 + + a1m bm1 a b + a b + + a b 22 21 2 m m1 21 11 a n1b11 + a n 2 b21 + + a nm bm1

a11b1r + + a1m bmr a21b1r + + a2 m bmr a n1b1r + + a nm bmr

Augmented Form
4 2 1 x1 15 4x1 - 2x 2 + x 3 = 15 3 1 4 x = 8 - 3x - x + 4x = 8 1 2 3 2 1 1 3 x3 13 x1 - x 2 + 3x 3 = 13

4 2 1 15 3 1 4 8 Augmented Form : 1 1 3 13

Elimination Methods
Aim: try to convert an augmented form matrix to an upper triangular form by using elementary row operations. Elementary row operations
We may multiply any row of the augmented coefficient matrix by a constant. We can add a multiple of one row to a multiple of any other row.

Example
1 15 4 2 1 15 4 2 3 1 4 8 3R + 4 R 0 10 19 77 1 2 1 1 3 13 R1 4 R3 0 2 11 37
1 15 4 2 0 10 19 77 R2 + 5 R3 0 0 36 108

x3=3 x2=-2 x1=2

Zeroes on the Diagonal

During the triangularization step, if a zero is encountered on the diagonal, we can not use that row to eliminate coefficients below that zero element. However, in that case, we can continue by interchanging rows and eventually achieve an upper triangular matrix of coefficients. The real stumbling block is finding a zero on the diagonal after we have triangularized. If that occurs, there is no solution and determinant is zero.

Gaussian Elimination
Gaussian elimination is an elimination method that cares for round off errors and ill-conditioning or division by zero. A useful strategy to avoid zero divisors is to rearrange the equations so as to put the coefficient of largest magnitude on the diagonal at each step. This is called pivoting.

Gaussian Elimination (cont.)

Complete pivoting may require both row and column interchanges. Partial pivoting, which places a coefficient of larger magnitude on the diagonal by row interchanges only, will guarantee a nonzero divisor if there is a solution to the sets of equation and will have the added advantage of giving improved arithmetic precision.

Example

LU Matrix and Multiple RightHand Side

Suppose we have solved the system Ax=b by Gaussian elimination. That is, we know the LU equivalent of A: A=L*U Consider we want to solve Ax=b with some new b vector. Ax=b=L*U*x=b,U*x=yL*y=b We can easily solve this for y by back substitution since L is a lower triangular matrix and b is known.

LU Matrix and Multiple Right-Hand Side (cont.)

Let Ux=y=b. Then we can solve for x easily since U is upper triangular. What if we had reordered the rows of matrix A by pivoting. This is not a problem if we save the order vector that tells the final order of the rows of the matrix.

Example

Gauss-Jordan Method
It is a variation of Gauss elimination. In it, the elements above the diagonal are made zero at the same time that zeros are created below the diagonal. The diagonal elements are normalized (made ones) at the same time that the reduction is performed. Thus, the elimination step results in an identity matrix rather than a triangular matrix. Consequently, it is not necessary to employ back substitution to obtain the solution. Pivoting is used to preserve accuracy.

Example

Pathological Systems
An arbitrary set of equations may not have such a guaranteed solution. There are several such possible situations, which is called pathological. In each case there is no unique solution to the set of equations.

Example

Singular Matrices
A matrix that does not have an inverse is defined as singular. Properties of a singular matrix: A singular matrix has
A determinant of zero Has rows that are linearly dependent Has columns that are linearly dependent Has no unique solution The rank is less than n, number of rows

Redundant and Inconsistent Systems

A system is redundant if any two rows are dependent (previous example is redundant) If there is no solution that satisfies the equations it is called inconsistent. Let the last column in the above 2 4 1 7 example 10.5 [55 7 2]Ti.e. see that the be 5. and 12 (0.5) x30=-1/30= (0.5) (0.333) is0 1 solution 3 1/3

Redundant and Inconsistent Systems (cont.)

Redundant and Inconsistent systems arise from singular matrices.

Properties of singular matrices

It has no inverse, A-1 Determinant is zero No unique solution to the system Ax=b Gaussian elimination can not avoid a zero on the diagonal Rank is less than n Rows and columns are linearly dependent

Properties of nonsingular matrices

Its inverse, A-1, exists Determinant is nonzero A unique solution to the system Ax=b exists Gaussian elimination does not encounter a zero on the diagonal Ranks equals n Rows and columns are not linearly dependent

Ill-Conditioned Systems
A system whose coefficient matrix is nearly singular is ill-conditioned. When a system is ill conditioned, the solution is very sensitive to changes in the right-hand side vector. It is also sensitive to small changes in the coefficients. For an ill-conditioned system, small changes in the input make large changes in the output.

Example
1.01x+0.99y=2 0.99x+1.01y=2 solution : x=1, y=1 If b is [2.02 1.98]T solution becomes x=2, y=0. If b is [1.98 2.02]T we would have x=0, y=2. It must be apparent that while solving an ill-conditioned system, the

Interpolation-Fitting a polynomial to a data set

Suppose we have the following data at right at hand and we wonder the value of f(x) at x=3.0 Let us use four points to estimate f(3.0). Then a third degree polynomial will fit on these points and finding out that cubic polynomial will help us to find f(3.0).
x 3.2 2.7 1.0 4.8 f(x) 22.0 17.8 14.2 38.3

Interpolation
Let the cubic polynomial passing through these be such that ax3+bx2+cx+d and
a(3.2)3+b(3.2)2+c(3.2)+d=22.0 a(2.7)3+b(2.7)2+c(2.7)+d=17.8 a(1.0)3+b(1.0)2+c(1.0)+d=14.2 a(4.8)3+b(4.8)2+c(4.8)+d=38.3

Interpolation
These equations lead us to a set of linear equations with four unknowns and four equations. But we know how to solve it by: Gaussian elimination. This gives us the solution as
a=0.5275 b=6.4952 c=-16.1177 d=24.3499 and f(3.0)=20.212 with these values.

Interpolation
Now, assume that fifth data pair came out as above. For this new set we should form a quadratic (fourth degree) polynomial and solve for f(3.0) once more.
x 3.2 2.7 1.0 4.8 5.6 f(x) 22.0 17.8 14.2 38.3 51.7

Lagrangian Polynomials
The simplest way to exhibit the existence of a polynomial for interpolation with unevenly spaced data. Suppose at hand we have:
i 0 1 2 3 x x0 x1 x2 x3 f(x) f0 f1 f2 f3

Lagrangian Polynomials
( x x1 )( x x2 )( x x3 ) ( x x0 )( x x2 )( x x3 ) P3 ( x) = f0 + f1 ( x0 x1 )( x0 x2 )( x0 x3 ) ( x1 x0 )( x1 x2 )( x1 x3 ) ( x x0 )( x x1 )( x x3 ) ( x x0 )( x x1 )( x x2 ) + f2 + f3 ( x2 x0 )( x2 x1 )( x2 x3 ) ( x3 x0 )( x3 x1 )( x3 x2 )

The lagrangian form for this polynomial is

Lagrangian Polynomials
The lagrangian for other degrees of interpolating polynomials employs this same pattern of forming a sum of polynomials all of the desired degree; it will have n-1 terms when the degree is n. Try to see that this Lagrangian polynomial passes through each point. Example: Solve the previous example by Lagrangian and compute f(3.0)
See that it is the same result.

Lagrangian PolynomialsExample

Lagrangian Polynomials
Extrapolation (applying the interpolating outside the range of x values employed to construct it) will have larger errors than for interpolation. Caution: never fit a polynomial of a degree higher than 4 or 5 to a set of points. If you need to fit to a set of more than 6 points, be sure to break up the set into subsets and fit separate polynomials to these. Better way may be using spline curves.