The variational

Approach to Optimal
Control
Introduction:
We discussed in last chapters optimal control problems and the
methods of optimization.
In this chapter we shall apply variational methods to optimal control
problems.
We shall first derive Necessary conditions for optimal control
assuming that the admissible controls are not bounded.
These Necessary conditions are then employed to find the optimal
control law for the important linear regulator problems.
Next, Pontryagins minimum principle is introduced heuristically as a
generalization of the fundamental theorem of the calculus of
variations, and problems with bounded control and state variables.
The three concluding sections are time-optimal problems, minimum
control-effort systems, and problems involving singular intervals.
Necessary conditions for optimal control
Let us employ the techniques in last chapter to determine necessary
conditions for optimal control. the problem is to find an admissible
control u* that causes the system
x a( x(t ), u (t ), t ) (1)
To follow an admissible trajectory x* that minimizes the performance
measure tf

J (u ) h( x(t f ), t f ) g ( x(t ), u (t ), t )dt
t0
(2)

We shall initially assume that the admissible state and control regions
are not bounded, and that the initial conditions x(t0)=x0 and the initial
time t0 are specified.
 The only difference between the equation
tf

J (u ) h( x(t f ), t f ) g ( x(t ), u (t ), t ) dt
t0

And the functional considered in chapter (4) is the term

involving the final states and final time however assuming
that h is differentiable function, we can write:
t f

d
dt
[ h( x (t ), t )]dt h( x (t f ), t f ) h( x (t0 ), t 0 )
t 0 (3)
t f

d
h( x (t f ), t f )
t
dt
[ h( x (t ), t )]dt h( x (t0 ), t 0 )
0

If it is assumed that the second partial derivatives are

continuous, the order of differentiation can be interchanged,
and these terms add to zero.
 In the following we shall find it convenient to use the function H ,
called the Hamiltonian, defined as

(4)

Using this notation we can write the necessary conditions through as

follows

(5)

(6)

(7)
(8)
 Boundary conditions
In particular problem either g and h may be missing; in this case, we
simply strike out the terms involving the missing function. To
determine the boundary conditions is a matter of making the
appropriate substitutions in equation (8) in all cases it will be
assumed that we have the n equations x*(t0 )=x0.

Problems with fixed final time:

if the final time tf is specified, x(tf ) may be specified, free , or required
to lie on some surface in the state space.

Case I : Final state specified since x(tf ) and tf are specified, we

substitute xf =0 and tf =0 in eq ( 8 ) The required n equations are
x*(tf )=xf
Case II : Final state free, We substitute tf =0 in eq (8 ) since xf is
arbitrary the n equations

Must be satisfied
 Case III : Final state lying on the surface defined by m(x(t))=0.
since this is a new situation, We substitute tf =0 in eq (8 ) and The
tangent line is normal to the gradient vector at the point (x(tf),tf).
Thus, x(tf) must be normal to the gradient so that:
 Problems with free Final Time:
If the final time is free, there are several situations that may occur.
CASE I Final state fixed. The appropriate substitution in Equation
(8) is xf =0. tf is arbitrary, so the (2n+1) relationship is

CASE II Final state Free. xf , tf are arbitrary and independent

therefore their coefficients must be zero that is:
 Making substitution in Equations ( 8 ) yield the equation
 Example:-

The system

Is to be controlled so its control effort is conserved; that

is the performance measure is to be minimized.

The admissible states and controls are not bounded. Find

necessary conditions that must be satisfied for optimal control.
 Solution:-

The first step is to form the Hamiltonian

From Equations (5,6,7) necessary conditions for optimality are

and
We obtain the linear algebraic equations
While the original performance measure is
minimized. As before, the solution of the state and costate equations
is given

And c1=c2=0 , the boundary conditions at t=2 are

Eliminating d and substituting t =2, we obtain the equations

Which have the solution c3=-0.894, c4=-1.379. the optimal trajectory

is then
 LINEAR REGULATOR PROBLEMS:
In this section we shall consider an important class of optimal control
problems- linear regulator systems. We shall that for linear regulator
problems the optimal control law can be found as a linear time
varying function of the system states. Under certain conditions, the
optimal control law becomes time invariant. The results presented
here are primarily due to R.E. Kalman.
The plant is described by the linear state equations:

Which may have time-varying coefficients. The performance measure

to be minimized is

The final time tf is fixed, H and Q are real symmetric positive semi-
definite matrices, and R is real symmetric positive definite matrix.
 It is assumed that the states and controls are not bounded, and x(tf) is
free. It is desired to maintain the state vector close to the origin
without an excessive expenditure of control effort.
The Hamiltonian is

And necessary conditions for optimality are:

 Can be solve the equation
The existence R-1 is assured, since R is a positive definite matrix. By
Substituting u*(t) in to linear system equation yields

Thus we have the set of 2n linear homogeneous differential equations

The solution to these equations has the form

Where is the transition matrix of the system.

Partitioning the transition matrix we have:

Where 11,12,21 and 22 are nn matrices.

From boundary conditions in entry 2 of the Table (1) we find that:

Substituting this for p*(tf) gives

Then we obtain

The figure shows the plant and the optimal controller.

To determine the feedback matrix F, we need the transition matrix.

If all of matrices (A,B,R,Q) are time invariant, the transition matrix can
be found by evaluating the inverse laplace transform of the matrix.
The matrix K satisfies the matrix differential equation

With the boundary condition K(tf)=H.

This matrix differential equation is of the Ricatti type.
 Example:
Find the optimal control law for the system

To minimize the performance measure

The admissible state and control values are unconstrained, the final
time T is specified, H>0, x(T) is free.
 from equations above gives:
A=a, B=1, R=0.25, Q=0,

Which has Transition Matrix

The optimal control law is
 Linear Tracking Problems
Next, let us generalize the results obtained for the
linear regulator problem to the tracking problem;
that is, the desired value of the state vector is not the
origin.

And the performance measure to be minimized is

 Where r(t) is the desired or reference value of the state vector. The
final time tf is fixed, x(tf) is free, and the states and controls are not
bounded. H and Q are real symmetric positive semi-definite matrices,
and R is real symmetric and positive definite.
The Hamiltonian is given by
 Substituting the equation in the state equations yields the state and
costate equations

Notice that the term Q(t)r(t) is forcing function; these differential

equations are linear and time varying, but not homogeneous. The
solution of this equation is

Where is the transition matrix of the system. if is partitioned and

the integral replaced by the 2n1 vector
 Replacing p*(tf) and then substituting x*(tf ) we obtain

The definitions of K(t) and s(t) are apparent by inspection.

Therefore, the optimal control law is:
 Where F(t) is the feedback gain matrix and v(t) is the
command signal. Notice that v(t) depends on the system
parameters and on the reference signal r(t). In fact, v(t)
depends on future values of reference signal, so we might
say that the optimal control has an anticipatory quality.
The diagram shows the plant and the optimal controller.
 We obtain

And

With the boundary conditions are

 The Riccati equation and differential equations for s are found
 And from boundary conditions equations

The optimal control law obtained from

 Summary
1. Procedure Summary of
A. Statement of the Problem

Given the plant as

x a( x(t ), u (t ), t )
the performance index as
tf

J (u ) h( x(t f ), t f ) g ( x(t ), u (t ), t ) dt
t0
and the boundary conditions as
x(t0) = x0 and tf and x(tf) = xf are free,

find the optimal control.

B. Solution of the Problem
Step 1 Form the Hamiltonian function

Step 2 Minimize H to find u*(t)

Step 3 Using the results of Step 2 in Step 1, find the optimal H*

H*(x*(t), u*(t), p*(t), t) = H*(x*(t), p*(t), t).
Step 4 Solve the set of 2n differential equations

with initial conditions x0 and the final conditions

Step 5 Substitute the solutions of x* (t), A" (t) from Step 4 into the
expression for the optimal control u*(t) of Step 2.
C. Types of Systems
See the table (1)
2. Procedure Summary of Linear Regulator System:
A. Statement of the Problem
Given the plant as

the performance index as

and the boundary conditions as x(to) = xo, tf is fixed, and x(tf) is free,
find the optimal control, state and performance index
B. Solution of the Problem
Step 1 Solve the matrix differential Riccati equation

with final condition K(t=tf)=H

Step 2 Solve the optimal state x* (t) from

Where p*(t)=K(t)x*(t) with initial condition x(t0) = x0.

Step 3 Obtain the optimal control u*(t) as
u*(t) = -F(t)x*(t) where, F(t) = R-1(t)B'(t)K(t).
Step 4 Obtain the optimal performance index from
J* = 0.5 x*'(t)K(t)x*(t).
3. Procedure Summary of Linear Tracking System
A. Statement of the Problem
Given the plant as

the performance index as

and the boundary conditions as x(t0) = x0, x(tf) is free,

find the optimal control, state and performance index.
 B. Solution of the Problem
Step 1 Solve the matrix differential Riccati equation

and the non-homogeneous vector differential equation

with final condition

Step 2 Solve the optimal state x* (t) from

x * (t ) ( A(t ) B(t ) R 1 (t ) BT (t ) K (t )) x * (t ) B(t ) R 1 (t ) BT (t )s(t )

with initial condition x(t0) = x0.
 Step 3 Obtain optimal control u*(t) from