LECTURES ON OPTIMAL CONTROL

THEORY
Terje Sund
August 9, 2012
CONTENTS
INTRODUCTION
1. FUNCTIONS OF SEVERAL VARIABLES
2. CALCULUS OF VARIATIONS
3. OPTIMAL CONTROL THEORY
INTRODUCTION
In the theory of mathematical optimization one try to nd maximum or
minimum points of functions depending of real variables and of other func-
tions. Optimal control theory is a modern extension of the classical calculus
of variations. Euler and Lagrange developed the theory of the calculus of
variations in the eighteenth century. Its main ingredient is the Euler equa-
tion
1
which was discovered already in 1744. The simplest problems in the
calculus of variation are of the type
max
_
t
1
t
0
F(t, x(t), x(t)) dt
where x(t
0
) = x
0
, x(t
1
) = x
1
, t
0
, t
1
, x
0
, x
1
are given numbers and F is a
given function of three (real) variables. Thus the problem consists of nding
1
the Euler equation is a partiall dierential equation of the form
F
x

d
dt
(
F
x
) = 0,
where F is a function of three real variables, x is an unknown function of one variable t
and x(t) =
dx
dt
(t).
F
x
stands for the partial derivative of F with respect to the second
variable x,
F
x
(t, x, x), and
F
x
are the partial derivatives with respect to the third variable
x.
1
functions x(t) that make the integral
_
t
1
t
0
F(t, x(t), x(t)) dt maximal or mini-
mal. Optimal control theory has since the 1960-s been applied in the study of
many dierent elds, such as economical growth, logistics, taxation, exhaus-
tion of natural resources, and rocket technology (in particular, interception
of missiles).
1 FUNCTIONS OF SEVERAL VARIABLES.
Before we start on the calculus of variations and control theory, we shall need
some basic results from the theory of functions of several variables. Let A
R
n
, and let F : A R be a real valued function dened on the set A. We
let F(x) = F(x
1
, . . . , x
n
), x = (x
1
, . . . , x
n
) A, ||x|| =
_
x
2
1
+ +x
2
n
. ||x||
is called (the Euklidean) norm of x.
A vector x
0
is called an inner point of the set A, if there exists a r > 0
such that
B(x
0
, r) = {x R
n
: ||x x
0
|| < r} A.
We let
A
0
= {x
0
A : x
0
is an inner point of A}
The set A
0
is called the interior of A. Let
e
1
, e
2
, . . . , e
n
be the vectors of the standard basis of R
n
, that is,
e
i
= (0, 0, . . . , 0, 1, 0, . . . , 0) (i = 1, 2, . . . , n),
hence e
i
has a 1 on the i-th entry, 0 on all the other entries.
Denition 1. let x
0
A be an inner point. The rst order partial
derivatives F

i
(x
0
) of F at the point x
0
= (x
0
1
, . . . , x
0
n
) with respect to the
i-th variable x
i
, is dened by
F

i
(x
0
) = lim
h0
1
h
[F(x
0
+he
i
) F(x
0
)]
= lim
h0
1
h
[F(x
0
1
, . . . , x
0
i
+h, x
0
i+1
, . . . , x
0
n
) F(x
0
1
, . . . , x
0
n
)],
if the limit exists.
2
We also write
F

i
(x
0
) =
F
x
i
(x
0
) = D
i
F(x
0
) (1 i n)
for the i-th rst order partial derivative of of F i x
0
.
Second order partial derivatives are written
F

i,j
(x
0
) =

2
F
x
j
x
i
(x
0
) = D
i,j
F(x
0
) (1 i n)
where we let

2
F
x
j
x
i
(x
0
) =

x
j
_
F
x
i
_
(x
0
) = F

j
(F

i
(x
0
)).
The gradient of the function F at x
0
is the vector
F(x
0
) = (
F
x
1
(x
0
), . . . , (
F
x
n
(x
0
))
Example 1. Let F(x
1
, x
2
) = 7x
2
1
x
3
2
+x
4
2
, x
0
= (2, 1) Then
F(x
1
, x
2
) = (14x
1
x
3
2
, 21x
2
1
x
2
2
+ 4x
3
2
),
F(x
1
, x
2
) = (14x
1
x
3
2
, 21x
2
1
x
2
2
+ 4x
3
2
)
= (28, 28).
Level surfaces: Let c R, F : R
n
R. The set
M
c
= {x : F(x) = c}
is called the level surface (or the level hypersurface ) of F corresponding to
the level c. if x
0
= (x
0
1
, x
0
2
, . . . , x
0
n
) M
c
, then we dene the tangent plane of
the level surface M
c
at the point x
0
as the solution set of the equation
F(x
0
) (x x
0
) = 0.
In particular, Fx
0
) is perpendicular to the tangent plane at x
0
, F(x
0
)
the tangent plane at x
0
.
Example 2. let F(x
1
, x
2
) = x
2
1
x
2
. then
F(x
1
, x
2
) = (2x
1
x
2
, x
2
1
) and F(1, 2) = (4, 1).
3
Tangent the equation at the point (1, 2) for the level surface (the level curve)
x
2
1
x
2
= 2 is F(1, 2) (x (1, 2)) = 0, hence (4, 1) (x
1
1, x
2
1) = 0,
4x
1
4 +x
2
2 = 0, that is 4x
1
+x
2
= 6.
Example 3. Let F(x
1
, x
2
, x
3
) = x
2
1
+ x
2
2
+ x
2
3
, and consider the level
surface
x
2
1
+x
2
2
+x
2
3
= 1,
the sphere with radius 1 and centre at the origin. (a) Then
F(x
1
, x
2
, x
3
) = 2(x
1
, x
2
, x
3
)
F(
1

3
,
1

3
,
1

3
) =
2

3
(1, 1, 1).
The tangent plane at the point
1

3
(1, 1, 1) has the equation:
2

3
(1, 1, 1) (x
1

3
, x
2

3
, x
3

3
) = 0,
2

3
(x
1
+x
2
+x
3
) =
2

3
3

3
,
x
1
+x
2
+x
3
=

3.
(b) Here
F(0,
3
5
,
4
5
) = 2(0,
3
5
,
4
5
) =
2
5
(0, 3, 4)
The tangent plane at (0,
3
5
,
4
5
) has the equation
2
5
(0, 3, 4) (x
1
0, x
2

3
5
, x
3

4
5
) = 0,
0 + 3(x
2

3
5
) + 4(x
3

4
5
) = 0,
3x
2
+ 4x
3
= 5.
Directional derivatives. Assume that F : A R
n
R. Let a R
n
and x A
0
be an inner point of A. The derivative of F along a at the point
x is dened as
F

a
(x) = lim
h0
1
h
[F(x +ha) F(x)]
if the limit exists.
4
Remark 1. The average growth of F from x of x +ha is
1
h
[F(x +ha) F(x)],
hence the above denition is natural if we think of the derivative as a rate of
change.
Denition 2. We say that F is continuously dierentiable, or C
1
, if the
function F

a
(x) is continuous at x for all a R
n
and all x A
0
, that is if
lim
y0
F

a
(x +y) = F

a
(x).
if F is C
1
, then we can show that the map a F

a
(x) is linear, for all
x A
0
. We show rst the following version of the Mean Value Theorem for
functions of several variables,
Proposition 1 (Mean Value Theorem). Assume that F is continuously
dierentiable on an open set A R
n
and that the closed line segment
[x, y] = {tx + (1 t)y : 0 t 1}
from x to y is contained in A. Then there exists a point w on the open line
segment (x, y) = {tx + (1 t)y : 0 < t < 1} such that
F(x) F(y) = F

xy
( w)
Proof. Set g(t) = F(tx + (1 t)y), t (0, 1). Then
1
h
[g(t +h) g(t)] =
1
h
[F((t +h)x + (1 (t +h))y) F(tx + (1 t)y)]
=
1
h
[F(t)x + (1 t)y +h(x y)) F(tx + (1 t)y)]

h0
F

xy
(tx + (1 t)y)
Since F was continuously dierentiable, it is clear that g

(t) is continuous
for 0 < t < 1, and g

(t) = F

xy
(tx + (1 t)y). Moreover, g(0) = F(y) and
g(1) = F(x). We apply the ordinary Mean Value Theorem to the function
g. Hence there exists a (0, 1) such that
g(1) g(0) = g

()
= F

xy
(x + (1 )y) = F

xy
( w),
5
where w = x +(1 )y lies on the open line segment (x, y). Hence we have
proved the Mean Value Theorem.
Proposition 2. Let x A
0
be given. If F is C
1
, then
(1) F

ca
(x) = cF

a
(x), for all c R and a R
n
and
(2) F

a+

b
(x) = F

a
(x) +F

b
(x),
that is, for each xed x, the map a F

a
(x) is a linear map of R
n
into R.
Proof. (1) if c = 0, it is clear that both sides of (1) are equal to 0.
Assume that c = 0. Then
F

ca
(x) = lim
h0
1
h
[F(x +hca) F(x)]
= c lim
h0
1
hc
[F(x +hca) F(x)]
k=hc
= c lim
k0
1
h
[F(x +ka) F(x)] = cF

a
(x).
(2) F is C
1
hence F

a
(x) and F

b
(x) exist. Since x A
0
, exists r > 0 such
that
||x y|| < r y A.
Choose h such that
||h(a +

b)|| < r and ||h

b|| < r.
Then
x +h(a +

b) A and x +h

b A,
hence
() F(x +h(a +

b)) F(x)
F(x +h(a +

b)) F(x +h

b) F(x)] + [F(x +h

b) F(x)]
We apply the Mean Value Theorem to the expression in the rst bracket.
Thus there exists a (0, 1) such that
() F((x +h

b) +ha) F(x +h

b) = F

ha
(x +ha +h

b).
6
By part (1) we have
F

ha
(x +ha +h

b) = hF

a
(x +ha +h

b)
If we divide the equation in () by h, then we nd
1
h
[F((x +h

b) +a)) F(x +h

b)] = F

a
(x +ha +h

b)
h0
F

a
(x),
since F is continuously dierentiable. Consequently, dividing by h in (), it
follows that
lim
h0
1
h
[F((x +h(a +

b)) F(x)]
= F

a
(x) + lim
h0
1
h
[F(x +h

b) F(x)] = F

a
(x) +F

b
(x)
We write a =

n
i=1
a
i
e
i
. Then
F

a
(x) =
n

i=1
a
i
F

e
i
(x) =
n

i=1
a
i
F
x
i
(x) = F(x) a.
As a consequence we have,
Corollary 1. If F is C
1
on A R
n
, then
F

a
(x) = F(x) a,
for all x A
0
.
If ||a|| = 1, then F

a
(x) is often called the directional derivative of F at
the point x in the direction of a. From Corollary 1 it follows easily that
Corollary 2. F is continuously dierentiable at the point x
0
F is
continuous at the point x
0
all the rst order partial derivatives of F are
continuous at x
0
.
Proof. Let a = (a
1
, . . . , a
n
) R
n
, ||a|| = 1. Then
|F

a
(x) F

a
(x
0
)| = |[F(x) F(x
0
)] a]|
=|F(x) F(x
0
)| ||a|| |F(x) F(x
0
)|.
This proves that F

a
(x) F

a
(x
0
)
xx
0
0 F(x) F(x
0
)
xx
0
0. Hence
F is continuously dierentiable at the point x
0
F is continuous at x
0
.
7
If we let a = e
i
, (1 i n), we nd that
F
x
i
(x) = F

e
i
(x) is continuous at
x
0
for i = 1, . . . , n if F is continuously dierentiable i x
0
. Conversely, if the
partial derivatives
F
x
i
(x
0
) are continuous for i = 1, . . . n, then
F

a
(x) =
n

i=1
a
i
F

e
i
(x) =
n

i=1
a
i
F
x
i
(x)
is continuous at x
0
for all a R
n
. Hence it follows that F is continuously
dierentiable at x
0
.
Convex sets. A set S R
n
is called convex if
tx + (1 t)y S
for all x, y S, and all t [0, 1].
In other words, S is convex if and only if the closed line segment [x, y]
between x and y is contained in S for all x, y S.
Example 4. Let us show that the unit disc with centre at the origin,
S = {(x
1
, x
2
) : x
2
1
+x
2
2
1} is a convex subset of R
2
.
Let x = (x
1
, x
2
), y = (y
1
, y
2
) S, 0 t 1. We must show that
z = tx + (1 t)y = t(x
1
, x
2
) + (1 t)(y
1
, y
2
) S.
Using the triangle inequality and that ||tu|| = |t| ||u||, u R
n
, we nd that
||z|| = ||tx + (1 t)y||
||tx|| +||(1 t)y|| = t||x|| + (1 t)||y||
t 1 + (1 t) 1 = 1
Hence z S. This shows that S is convex.
Exercise 1. Let a > 0 and b > 0. Show that the elliptic disc
S = {(x
1
, x
2
) :
x
2
1
a
2
+
x
2
2
b
2
1}
er a convex subset of R
2
.
Concave and convex functions. A C
2
function (that is, a func-
tion possessing a continuous second derivative) f : [a, b] R is concave if
f

(x) 0 for all x in the open interval (a, b). The function f is convex if
8
f

(x) 0 for all x (a, b). If f is convex, we see see geometrically that the
chord [(x, f(x))] always lies under or on the graph of f. Equivalently, the
inequality
() tf(x) + (1 t)f(y) f(tx + (1 t)y),
holds for all x, y (a, b), and all t [0, 1]. The converse is also right: If
() holds, then f is concave. For functions of several variables, the second
derivative at a vector x is no real number, but a bilinear function. As a
consequence, we will use the inequality () when we dene concavity for
functions of several variables.
Denition 3. Let S be a convex subset of R
n
and let f be a real valued
function dened on S. We say that f is concave if f satises the inequality
() f(tx + (1 t)y) tf(x) + (1 t)f(y),
for all x, y S and all t [0, 1]. A function f is called convex if the opposite
inequality holds. The function is strongly concave (respectively strongly
convex) if strict inequality holds in ().
Remark 2. That f is concave, means geometrically that the tangent
plane of the surface z = f(x) lies over (or on) the surface at every point
(y, f(y)): the equation for the tangent plane through the point (y, f(y))is
z = f(y) (x y) + f(y),
hence
f(x) z = f(y) (x y) +f(y)
for all x on the tangent plane.
Proposition 3. Assume that the functions f and g are dened on a
convex set S R
n
. Then the following statements hold
(a) If f and g are concave and a 0, b 0, then the function af +bg is
concave.
(b) If f and g are convex and a 0 then the function af +bg is convex.
Proof. (b): Assume that f and g are convex, a 0, b 0. Set h(x) =
af(x) +bg(x) (x S), and let t (0, 1). For all x, y S then
h(tx + (1 t)y) = af(tx + (1 t)y) +bg(tx + (1 t)y)
a[tf(x) + (1 t)f(y)] +b[tg(x) + (1 t)g(y)] = th(x) + (1 t)h(y).
9
Accordingly h convex.
Part (a) is shown in the same way.
The following useful 2nd derivative test holds:
Proposition. Let f : S R
2
R, where S is open and convex. If all
the 2nd order partial derivatives of f are continuous (that is, f is C
2
on S),
then
(a) f is convex

2
f
x
2
1
(x) 0,

2
f
x
2
2
(x) 0 and

2
f
x
2
1
(x)

2
f
x
2
2
(x)

2
f(x)
x
1
x
2
2
0 for all x S.
(b) f is concave

2
f
x
2
1
(x) 0,

2
f
x
2
2
(x) 0 and

2
f
x
2
(x)

2
f
x
2
2
(x)

2
f(x)
x
1
x
2
2
0 for all x S.
We vil prove this proposition below (see Proposition 8). First we shall
need some results on symmetric 2 2 matrices Let
Q =
_
a b
b c
_
be a symmetric 2 2-matrix, and let
Q(x) = x
t
Qx = (x
1
, x
2
)Q
_
x
1
x
2
_
= ax
2
1
+ 2bx
1
x
2
+cx
2
2
, x R
2
.
Denition 4. We say that Q is
(a) positive semidenite if x
t
Qx 0 for all x R
2
.
(b) positive denite if x
t
Qx > 0 for all x R
2
\ {0}.
(c) negative semidenite if x
t
Qx 0 for all x R
2
.
(d) negative denite if x
t
Qx < 0 for all x R
2
\ {0}.
We will also write Q 0 if Q is positive semidenite, Q > 0 if Q is
positive denite, and similarly for (c) and (d).
Proposition 4. Q is positive semidenite a 0, c 0 and acb
2
0.
10
Proof. : assume that a 0, b 0 and ac b
2
0. if a = 0, vil
ac b
2
= b
2
0
then b = 0. Hence
(x
1
, x
2
)Q
_
x
1
x
2
_
= (x
1
, x
2
)
_
0 0
0 c
__
x
1
x
2
_
= (x
1
, x
2
)
_
0
cx
2
_
= cx
2
2
0,
so that Q is positive semidenite.
Assume that a = 0. Then
(x
1
, x
2
)
_
a b
b c
__
x
1
x
2
_
= (x
1
, x
2
)
_
ax
1
+bx
2
bx
1
+cx
1
_
= ax
2
1
+ 2bx
1
x
2
+cx
2
2
= a[x
2
1
+ 2
b
a
x
1
x
2
+
c
a
x
2
2
]
= a[x
2
1
+ 2
b
a
x
1
x
2
+
b
2
a
2
x
2
2
+ (
c
a

b
2
a
2
)x
2
2
]
= a[(x
1
+
b
a
x
2
)
2
+
ac b
2
a
2
x
2
2
] 0
for all (x
1
, x
2
) R
2
. Hence is Q 0.
: Assume that Q 0, then x
t
Qx 0, x R
2
. In particular,
(1, 0)
_
a b
b c
__
1
0
_
= a 0
and
(0, 1)
_
a b
b c
__
0
1
_
= c 0
Assume that a > 0 : As we have seen above,
(x
1
, x
2
)
_
a b
b c
__
x
1
x
2
_
= a[(x
1
+
b
a
x
2
)
2
+
ac b
2
a
2
x
2
2
].
11
If we let x
1
=
b
a
x
2
, we derive that
acb
2
a
2
x
2
2
0, for all x
2
, henceac b
2
0.
Assume that a = 0 : Then
0 x
t
Qx = ax
2
1
+ 2bx
1
x
2
+cx
2
2
= 2bx
1
x
2
+cx
2
2
if b > 0, the fact that x
2
= 1, x
1
= n, implies
0 2bn +c
n
,
a contradiction. Hence b = 0, and ac b
2
= 0 0.
Further, it is easy to show
Proposition 5. Q is positive denite a > 0 (and c > 0) and ac b
2
>
0.
The following chain rule for real functions of several variables will be
useful.
Proposition 6. (Chain Rule) Let f be a real C
1
- function dened on an
open convex subset S of R
n
, and assume that r : [a, b] S is dierentiable.
Then the composed function f r = f (r
1
, . . . , r
n
) : [a, b] R is well
dened. Set g(t) = f(r(t)). Then g is dierentiable and
g

(t) =
f
x
1
(r(t))r

1
(t) + +
f
x
n
(r(t))r

n
(t) = f(r(t)) r

(t).
Proof. consider
()
g(t +h) g(t)
h
=
f(r(t +h)) f(r(t))
h
Since S is open, there exists an open sphere B
r
with positive radius and centre
at r(t) which is contained in S. We choose h = 0 so small that r(t +h) B
r
.
Set u = r(t +h), v = r(t). Then the numerator on the right hand side of ()
is f(u) f(v). By applying the Mean Value Theorem to the last dierence,
we obtain
f(u) f(v) = f(z) (u v),
where z = u + (u v) for a (0, 1). Hence the dierence quotient in ()
may be written
()
1
h
[g(t +h) g(t)] =
1
h
f(z) [(r(t +h)) (r(t))]
12
As h 0, we nd u v, so that z u. Sincef is C
1
, is f continuous,
f(z) f(u) = f(r(t)). Therefore the right side of () converges to
f(r(t))r

(t). This shows that g

(t) exists and is equal to the inner product

f(r(t)) r

(t).
If the function f is dened on an open subset S of R
2
for which the partial
derivatives of 2nd order exist on S, then we may ask if the two mixed partial
derivative are equal. That is, if

2
f
x
1
x
2
=

2
f
x
2
x
1
. This is not always the case
however, the following result which we state without proof, will be sucient
for our needs.
Proposition 7. Assume that f is a real valued function dened on an
open subset S of R
2
and that all the rst and second order partial derivatives
of f are continuouse on S. Then

2
f
x
1
x
2
(x) =

2
f
x
2
x
1
(x),
for all x i S.
Proof. See for instance Tom M. Apostol, Calculus Vol. II, 4.25.
Next we will apply the last four propositions to prove
Proposition 8. Let f be a C
2
function dened on an open convex subset
S of R
2
. Then the following statements hold,
(a) f is convex

2
f
x
2
1
0,

2
f
x
2
2
0 and
f
=

2
f
x
2
1

2
f
x
2
2
(

2
f
x
1
x
2
)
2
0
(b) f is concave

2
f
x
2
1
0,

2
f
x
2
2
0 and
f
0
(c)

2
f
x
2
1
> 0 and
f
> 0 f is strictly convex.
(d)

2
f
x
2
1
< 0 and
f
> 0 f is strictly concave.
Alle the above inequalities are supposed to hold at each point of S.
Proof. (a) : Choose two arbitrary vectors a,

b in S, and let t [0, 1].

We consider the function g given by
g(t) = f(

b +t(a

b)) = f(ta + (1 t)

b).
Put r(t) =

b +t(a

b), thus g(t) = f(r(t)). Let a = (a

1
, a
2
),

b = (b
1
, b
2
). The
13
Chain Rule yields
g

(t) = f(r(t)) r

(t)
f
x
1
(r(t)) (a
1
b
1
) +
f
x
2
(r(t)) (a
2
b
2
)
and
g

(t) =(

2
f(r(t))
x
2
1
,

2
f(r(t))
x
2
x
1
) (a

b)(a
1
b
1
)
+ (

2
f(r(t))
x
1
x
2
,

2
f(r(t))
x
2
2
) (a

b)(a
2
b
2
)
=[

2
f
x
2
1
(a
1
b
1
)
2
+

2
f
x
2
x
1
(a
1
b
1
)(a
2
b
2
)
+

2
f
x
1
x
2
(a
1
b
1
)(a
2
b
2
) +

2
f
x
2
2
(a
2
b
2
)
2
]
r(t)
=(a

b)
t
_

2
f
x
2
1

2
f
x
1
x
2

2
f
x
2
x
1

2
f
x
2
2
_
r(t)
(a

b)
From the conditions in (a) the Hessematrix H of f,
H =
_

2
f
x
2
1

2
f
x
1
x
2

2
f
x
2
x
1

2
f
x
2
2
_
is positive semidenite in S. Accordingly g

(t) 0, for all t (0, 1), so that

g is convex. In particular it follows that
f(ta + (1 t)

b) = g(t) = g(t 1 + (1 t) 0)
tg(1) + (1 t)g(0) = tf(a) + (1 t)f(

b).
Since a and

b were arbitrary in S, we deduce that f is convex.

: In light of Proposition 4 it suces to prove that the Hessematrix H
of f is positive semidenite in S. Let x S, h R
2
be arbitrary. As S is
open, there exists r > 0 such that |t| < r x + t

h S. Dene p on the
interval I = (r, r) ved p(t) = f(x +t

h).
Claim: p is a convex function.
14
In order to see this, let [0, 1], t and s I. Then
p(t + (1 )s)) = f(x +t

h + (1 )s

h)
= f((x +t

h) + (1 )(x +s

h))
f(x +t

h) + (1 )f(x +s

h)
= p(t) + (1 )p(s),
which proves the claim.
Since p is C
2
and convex, we deduce p

0. Let r(t) = x +t

h, t I. By
The Chain Rule,
p

(t) = f(r(t))

h
and as in the rst part of the proof,
p

(t) =

h
t
H(r(t))

h (

h R
2
arbitrary).
As p

(t) 0, it follows that H(r(t)) is positive semidenite, t I. In partic-

ular,
H(x) = H(r(0)) 0
for all x S. Hence part (a) follows.
The proofs of (b), (c) and (d) are similar.
The following gradient inequality and its consequenses are useful in the
calculus of variations.
Proposition 9. (Gradient inequality) Assume that S R
n
is convex,
and let f : S R be a C
1
-function. Then the following equivalence holds
f is concave
(1) f(x) f(y) f(y) (x y) =
n

i=1
f
x
i
(y)(x
i
y
i
)
for all x, y S.
Proof. : Assume that f is concave, and let x, y S. For all t (0, 1) :
we know that
tf(x) + (1 t)f(y) f(tx (1 t)y),
15
that is
t(f(x) f(y)) f(t(x y) +y) f(y)
hence
f(x) f(y)
1
t
[f(y +t(x y)) f(y)]
If we let t 0, we nd that the expression to the right approaches the
derivative of f at y along x y. Hence,
f(x) f(y) lim
t0
1
t
[f(y +t(x y)) f(y)] = f

xy
(y)
= f(y) (x y)
: Assume that the inequality (1) holds. Let x, y S, and let t (0, 1). We
put
z = tx + (1 t)y.
It is clear that z S, since S is convex. By (1),
(2) f(x) f(z) f(z) (x z)
and
(3) f(y) f(z) f(z) (y z)
Multiplying the inequality in (2) by t and the inequality in (3) by 1 t, and
then adding the two resulting inequalities, we derive that
(4) t(f(x) f(z)) + (1 t)(f(y) f(z))
f(z) [t(x z) + (1 t)(y z)]
Here the right side of (4) equals

0, since
t(x z) + (1 t)(y z) = tx + (1 t)y tz z +tz
= tx + (1 t)y z =

0.
Rearranging the inequality (4) we hence nd,
tf(x) + (1 t)f(y) f(z) + (1 t)f(z)
= f(z) = f(tx + (1 t)y),
which shows that f is convex.
16
Remark 3. We also have that f is strictly concave proper inequality
holds in (1). Corresponding results with the opposite inequality (respectively
the opposite proper inequality) in (1), holds for convex (respectively strongly
convex) functions.
Denition 5. Let f : S R
n
R, and let x

= (x

1
, . . . , x

n
) S.
A vector x

is called a (global) maximum point for f if f(x

) f(x) for
all x S. Similarly we dene global minimum point. x

is called a local
maximum point for f if there exists a positive r such that f(x

) f(x) for
all x S that satises ||x x

|| < r.
A stationary (or critical) point for f is a point y S such that f(y) =

0,
that is, all the rst order partial derivatives of f at the point y are zero,
f
x
i
(y) = 0 (1 i n).
If x

is a local maximum- or minimum point in the interior S

0
of S, then
f(x

) =

0 :
f
x
i
(x

) = lim
h0
+
1
h
[f(x

) f(x

+he
i
)] 0
and
f
x
i
(x

) = lim
h0

1
h
[f(x

) f(x

+he
i
)] 0,
and hence
f
x
i
(x

) = 0, (1 i n). For arbitrary C

1
-functions f the
condition f(x

) =

0 is not sucient to ensure that f has a loal maximum

or minimum at x

, however, if f is convex or concave, the following holds,

Proposition 10. Let f be a real valued C
1
function dened on a convex
subset S of R
n
, and let x

S. The following holds

(a) if f is concave, then x

is a local maximum point for f

f(x

) =

0.
(b) if f is convex, is x

a local minimum point for f

f(x

) =

0.
Proof. (a)
: We have seen above that if x

is a local maximum point for f, then

f(x

) =

0.
17
: Assume that x

is a stationary point and that f is concave. For all x S

it follows from the gradient inequality in Proposition 9 that
f(x) f(x

) f(x

) (x x

) = 0,
hence f(x) f(x

), for all x S.
the proof of (b) is similar.
The next result will also be useful,
Proposition 11. Assume that F is a real valued function dened on a
convex subset S of R
n
, and that G is a real function dened on an interval in
R that contains the image f(S) = {f(x) : x S} of f. Then the following
statements hold,
(a) f is concave and G is concave and increasing G f is concave.
(b) f is convex and G is convex and increasing G f is convex.
(c) f is concave and G is convex and decreasing G f is convex.
(d) f is convex and G concave and decreasing G f is concave.
Proof. (a) Let x, y S, and let t (0, 1). Put U = G f. Then
U(tx + (1 t)y) = G(f(tx + (1 t)y)
G(tf(x) + (1 t)f(y)) (f concave and G increasing)
tG(f(x)) + (1 t)G(f(y)) (G concave)
= tU(x) + (1 t)U(y),
hence U is concave.
(b) is proved as (a).
(c) and (d): Apply (a) and (b) to the function G. (Notice that G is convex
and decreasing G is concave and increasing.)
2 CALCULUS OF VARIATIONS.
We may say that the basic problem of the calculus of variations is to deter-
mine the maximum or minimum of an integral of the form
(1) J(x) =
_
t
1
t
0
F(t, x(t), x(t)) dt,
18
where F is a given C
2
function of three variables and x = x(t) is an unknown
C
2
function on the interval [t
0
, t
1
], such that
(2) x(t
0
) = x
0
, and x(t
1
) = x
1
,
where x
0
and x
1
are given numbers. Additional side conditions for the prob-
lem may also be included. Functions x that are C
2
and satisfy the endpoint
conditions (2) are called admissible functions.
Example 1. (Minimal surface of revolution.)
Consider curves x = x(t) in the txplane, t
0
t t
1
, all with the same
given, end points (t
0
, x
0
), (t
1
, x
1
). By revolving such curves around the taxis,
we obtain a surface of rvolution S.
Problem. Which curve x gives the smallest surface of revolution?
If we assume that x(t) 0, the area of S is
A(x) =
_
t
1
t
0
2x

1 + x
2
dt
Our problem is to determine the curve(s) x

for which min A(x) = A(x

),
when x(t
0
) = x
0
, x(t
1
) = x
1
. This problem is of the same type as in (1)
above.
We shall next formulate the main result for problems of the type (1).
Theorem 1. (Main Theorem of the Calculus of Variations)
Assume that F is a C
2
function dened on R
3
. Consider the integral
()
_
t
1
t
0
F(t, x, x) dt
If a function x

maximizes or minimizes the integral in () among all C

2
functions
x on [t
0
, t
1
] that satisfy the end point conditions
x(t
0
) = x
0
, x(t
1
) = x
1
,
then x

satises the Euler equation

(E)
F
x
(t, x(t), x(t))
d
dt
(
F
x
(t, x(t), x(t)) = 0 (t [t
0
, t
1
]).
If the function (x, x) F(t, x, x) is concave (respectively convex) for each
t [t
0
, t
1
], then a function x

that satises the Euler equation (E), solves

the maximum (respectively minimum) problem.
19
A proof of Theorem 1 will be given below. First we consider
Example 2. Solve
min
x
_
1
0
(x
2
+ x
2
) dt, x(0) = 0, x(1) = e
2
1.
We let F(t, x, x) = x
2
+ x
2
. Then
F
x
= 2x,
d
dt
F
x
=
d
dt
(2 x) = 2 x. Hence the
Euler equation of the problem is
x x = 0,
The general solution is
x(t) = ae
t
+Be
t
Here x(0) = 0 = A + B, B = A, x(1) = e
2
1 = Ae + Be
1
= A(e e
1
).
This implies that A = e, B = e, hence
x(t) = e
t+1
e
1t
is the only possible solution, by the rst part of Theorem 1. Here F(t, x, x) =
x
2
+ x
2
is convex as a function of (x, x), for each t. As a consequence of the
last part of Theorem 1, we conclude that x does in fact solve the problem.
The minimum is
_
1
0
(x
2
+ x
2
) dt =
_
1
0
[(e
t+1
e
1t
)
2
+ (e
t+1
+e
1t
)
2
] dt = . . . = e
4
1.
We notice that t does not occur explicitly in the formula for F in this example.
Next we shall take a closer look at this particular case.
The case F = F(x, x) (that is, t does not occur explicitly in the formula
of the function F).
If x(t) is a solution of the Euler equation, then the Chain Rule yields:
d
dt
[f(x, x) x
F
x
(x, x)] =
F
x
x +
F
x
x x
F
x
x
d
dt
F
x
= x[
F
x

d
dt
(
F
x
)] = x 0 = 0,
so that
F x
F
x
= C (is constant).
20
This equation is called a rst integral of the Euler equation.
In order to prove Theorem 1 we will need,
The Fundamental Lemma (of the Calculus of Variations). Assume
that f : [t
0
, t
1
] R is a continuous function, and at
_
t
1
t
0
f(t)(t) dt = 0
for all C
2
functions that satises (t
0
) = (t
1
) = 0. Then f(t) = 0 for all
t in the interval [t
0
, t
1
].
Proof. If there exists an s (t
0
, t
1
) such that f(s) = 0, say f(s) > 0,
then f(t) > 0 for all t in some interval [s , s +] of s since f is continuous.
We choose a function such that is C
2
and
0, (s ) = (s +) = 0, > 0/on (s , s +)
and = 0 outside the interval (s , s +). We may for instance let
_
(t (s ))
3
((s +) t)
3
, y [s , s +]
0, t / [s , s +].
which is C
2
. Then
_
t
1
t
0
f(t)(t) dt =
_
s+
s
f(t)(t) dt > 0.
since (t)f(t) > 0 and (t)f(t) is continuous on the open interval (s, s+).
Hence we hofe obtained a contradiction.
We shall also need to dierentiate under the integral sign:
Proposition 2. Let g be a C
2
function dened on the rectangle R =
[a, b] [c, d] i R
2
.
G(u) =
_
b
a
g(t, u) dt, u [c, d].
Then
G

(u) =
_
b
a
g
u
(t, u) dt.
21
Proof. Let > 0. Since
g
u
is continuous, it is also uniformly continuous
on R (that is, closed and bounded, hence compact). Therefore there exists a
> 0 such that
|
g
u
(s, u)
g
u
(t, v)| <

b a
,
for all (s, u), (t, v) in R such that ||(s, u) (t, v)|| < . By the Mean value
Theorem there is a = (t, u, v) between u and v such that
g(t, v) g(t, u) =
g
u
(t, )(v u).
Consequently,
|
_
b
a
[
g(t, v) g(t, u)
v u

g
u
(t, u)] dt|

_
b
a
|
g
u
(t, )
g
u
(t, u)| dt

_
b
a

b a
dt = , for |v u| < .
Hence
G

(u) = lim
vu
G(v) G(u)
v u
= lim
vu
_
b
a
g(t, v) g(t, u)
v u
dt
=
_
b
a
g
u
(t, u) dt.

Proof of Theorem 1. We will consider the maximum problem (the

minimum problem is similar)
(1)
_

_
max
x
_
t
1
t
0
F(t, x, x) dt
when x(t
0
) = x
0
and x(t
1
) = x
1
, x
0
, x
1
given numbers.
assume that x

is a C
2
function that solves (1). Let be an arbitrary
C
2
function on [t
0
, t
1
] that satises (t
0
) = (t
1
) = 0. For each real the
function
x = x

+
22
is an admissible function, that is, a C
2
function satisfying the endpoint
conditions x(t
0
) = x
0
and x(t
1
) = x
1
. Then we must have
J(x

) J(x +)
for all real . Let be xed. We will study I() = J(x

+ ) as a function
of . The function I has a maximum for = 0. Hence
I

(0) = 0.
Now
I() =
_
t
1
t
0
F(t, x

+, x

+ ) dt
Dierentiating under the integral sign (see Proposition 2 above) with respect
to we nd,
0 = I

(0) =
_
t
1
t
0
_
F

x
+
F

dt
where we put
F

x
=
F
x
(t, x

, x

) and
F

x
=
F

x
(t, x

, x

). Hence
0 = I

(0) =
_
t
1
t
0
F

x
dt +
_
t
1
t
0
F

x
dt
=
_
t
1
t
0
F

x
dt +
_
F

x

_
t
1
t
0

_
t
1
t
0
d
dt
_
F

x
_
dt (ved delvis integrasjon)
=
_
t
1
t
0
_
F

x

d
dt
_
F

x
_
dt (since (t
1
) = (t
0
) = 0)
hence
_
t
1
t
0
_
F

x

d
dt
_
F

x
_
dt = 0
for all such C
2
functions with (t
1
) = (t
1
) = 0. By the Fundamental
Lemma it follows that
(2)
F

x

d
dt
_
F

x
_
= 0,
hence x

satises the Euler equation. Using a quite similar argument we nd

that an optimal solution x

for the minimum problem also satises (2).

Suciency: assume that F(t, x, x) is concave with respect to the last
two variables (x, x), and let x

satisfy the Euler equation with the endpoint

23
conditions of (1). We shall prove that x

solves the the maximum problem.

Let x be an arbitrary admissible function for the problem. By concavity the
Gradient inequality yields,
F(t, x, x) F(t, x

, x

)
F

x
(x x

) +
F

x
( x x

)
d
dt
(
F

x
)(x x

) +
F

x
( x x

) =
d
dt
(
F

x
(x x

)),
for all t [t
0
, t
1
]. By integration it then follows that
_
t
1
t
0
_
F(t, x, x) F(t, x

, x

)
_
dt

_
t
1
t
0
d
dt
_
F

x
_
(x x

) dt =
_
F

x
(x x

)
_
t
1
t
0
= 0,
where we used the endpoint conditions for x and x

at the last step. Hence

_
t
1
t
0
F(t, x, x) dt
_
t
1
t
0
F(t, x

, x

) dt
as we wanted to prove.
Remark. In the above theorem, assume instead that F is concave with
respect to (x, x) (for each t [t
0
, t
1
]) only in a certain open and convex
subset R of R
2
. The above suciency proof still applies to the set V of
all admissible functions x enjoying the property that (x(t), x(t)) R for all
t [t
0
, t
1
]. Hence any admissible solution x

of the Euler equation that is

also an element of V will maximize the integral
_
t
1
t
0
F(t, x, x) dt among all
members of V . Consequently, we obtain a maximum relative to the set V .
Note that the proof uses the Gradient Inequality (Proposition 9 in Section
1) which requires the region to be open and convex.
Example 3. We will nd a solution x = x(t) of the problem
min
_
1
1
F(t, x, x) dt
where F(t, x, x) = t
2
x
2
+ 12x
2
and x(1) = 1, x(1) = 1.
(a) First we will solve the problem by means of Theorem 1. Here we nd
F
x

d
dt
_
F
x
_
= 24x
d
dt
_
t
2
2 x
_
= 24x 4t x 2t
2
x,
24
and the Euler equation takes the form
t
2
x + 2t x 12x = 0.
We seek solutions of the type x(t) = t
k
. This gives
(k(k 1) + 2k 12)t
k
= 0, for all t,
that is, k
2
+ k 12 = 0, which has the solutions k = 3 and k = 4.
Consequently, x(t) = At
3
+Bt
4
where
x(1) = A +B = 1
x(1) = A +B = 1
Hence B = 0 and A = 1, so that x(t) = t
3
is the only possible solution.
Furthermore,

2
F
x
2
= 2t
2
0,

2
F
x
2
= 24 > 0,

2
F
x x
= 0,
and the determinant of the Hesse matrix is 24 2t
2
= 48t
2
0. It follows that
F is convex with respect to (x, x). Hence, in view of Theorem 1, x(t) = t
3
gives a minimum.
(b) Suppose next that x is a solution of the Euler equation to the problem
which in addition satises the given endpoint conditions. If z is an arbitrary
admissible function, then = z x is a C
2
-function that satises (1) =
(1) = 0. Let us show that
J(z) J(x) = J(x +) J(x) 0.
From this it will follow that x minimizes J. Now
J(x +) J(x) =
_
1
1
[t
2
( x + )
2
+ 12(x +)
2
t
2
x
2
12x
2
] dt
=
_
1
1
[2t
2
x +t
2

2
+ 24x + 12
2
] dt =
_
1
1
[t
2

2
+ 12
2
] dt +I
25
where
I =
_
1
1
[2t
2
x + 24x] dt
=

1
1
2t
2
x
_
1
1
(2t
2
x + 4t x) dt +
_
1
1
24xdt (using integration by parts)
= 0
_
1
1
2[t
2
x + 2t x 12x] dt
_
(1) = (1) = 0
_
= 0,
where the last inequality follows from the fact that x satises the Euler
equation
t
2
x + 2t x 12x = 0.
Hence
J(x +) J(x) =
_
1
1
[t
2

2
+ 12
2
] dt > 0
if = 0, hence every solution of the Euler equation that satises the endpoint
conditions, yields a minimum. We have seen in (a) that x(t) = t
3
is the only
such solution.
Exercise 1. Consider the problem min J(x), where
J(x) =
_
2
1
x(1 +t
2
x) dt, x(1) = 3, x(2) = 5.
(a) Find the Euler equation of the problem and show that it has exactly
one solution x that satises the given endpoint conditions.
(b) Apply Theorem 1 to prove that the solution from (a) really solves
the minimum problem.
(c) For an arbitrary C
2
-function that satises (1) = (2) = 0, show
that
J(x +) J(x) =
_
2
1
t
2

2
dt,
where x is the solution from (a). Explain in light of this, that x minimizes
J.
Exercise 2. A function F is given by
F(x, y) = x
2
(1 + y
2
), (x, y) R
2
26
(a) Show that F is convex in the region
R = {(x, y) : |y|
1

3
}
(b) Show that the variation problem
min
_
1
0
x
2
(1 + x
2
) dt,
x(0) = x(1) = 1
has Euler equation
() x x + x
2
1 = 0
(c) Find the only possible solution x

of the variation problem. Let

V = {x : x is a C
2
function such that x(0) = x(1) = 1
and | x(t)| <
1

3
for all t [0, 1]}
Show that x

V . Explain that x

minimizes the integral in (1) among all

functions in V .
Weierstrass suciency condition.
2
Example 4. Consider the problem of maximizing or minimizing the
integral
J(x) =
_
1
0
( x
2
x
2
) dt, x(0) = x(1) = 0.
The Euler equation of the problem is
x +x = 0
The extremals (that is, the solutions of the Euler equation) are
x(t) = Asin t +Bcos t, A and B arbitrary constants.
Among all extremals only the zero function x

, x

(t) = 0 for all t [0, 1],

satises the endpoint conditions. It is easy to see, using the second-derivative
2
This section is optional.
27
test, that F(t, x, x) = x
2
x
2
is neither concave nor convex i (x, x). Hence
we cannot use Theorem 1 to decide if x

gives a maximum or a minimum for

the problem.
Exercise 3. Show that the function F(t, x, x) = x
2
x
2
of the last
Example is neither convex nor concave.
The suciency condition stated in Theorem 3 below, is often useful for
solving problems
(1) max or minJ(x)
where
J(x) =
_
t
1
t
0
F(t, x(t), x(t)) dt.
As above F is assumed to be a given C
2
function of three variables and
x = x(t) is an unknown C
2
-function on the interval [t
0
, t
1
], such that
(2) x(t
0
) = x
0
, x(t
1
) = x
1
,
where x
0
and x
1
are given numbers.
The solutions of the Euler equation of this problem (without any endpoint
conditions), are called extremals.
Denition. we say that an extremal x

for
J(x) =
_
t
1
t
0
F(t, x(t), x(t)) dt, x(t
0
) = x
0
, x(t
1
) = x
1
,
is a relative maximum if there exists an r > 0 such that J(x

) J(x) for all

C
2
-functions x with x(t
0
) = x
0
, x(t
1
) = x
1
that satises |x

(t) x(t)| < r

for all t [0, 1]. A relative minimum x

is dened similarly.
Theorem 3. (Weierstrass suciency condition). Assume that an ex-
tremal x

for the problem (1) satises the endpoint conditions (2) and that
28
there exists a parameter family x(, )
3
, < < , of extremals such that
(1) x(t, 0) = x

(t), t [t
0
, t
1
],
(2) x(, ) is dierentiable with respect to and
x

(t, )

=0
= 0,
t
0
t t
1
,
(3) if
1
=
2
, then the two curves given by x(,
1
) and x(,
2
)have no
common points,
(4)

2
F
x
2
(t, x, x) < 0 for all t [t
0
, t
1
] and all x = x(, ), (, )
Then x

is a relative maximum for J(x) with the given endpoint conditions.

If the condition
(4

)

2
F
x
2
(t, x, x) > 0 for all t [t
0
, t
1
] and all x = x(, ), (, ),
holds instead of (4), then x

is a relative minimum.
Consider the extremals
x(t) = Acos t +Bsin t, t [0, 1].
of Example 4. The parameter family of extremals x(, given by
() x(t, ) = cos t, (1, 1),
satises x(, 0) = x

= 0 and is dierentiable with respect to with

x(t,)

=0
=
cos t = 0 for all t [0, 1]. Moreover, it is clear that the family of curves given
by () have no common points. Finally we have

2
F
x
2
(t, x, x) =

x
(2 x) = 2 > 0.
Hence, by the Weierstrass suciency condition, x

= 0 is a relative minimum
point for J(x).
Exercise 4. Let
J(x) =
_
t
1
t
0

1 + x
2
dt, x(t
0
) = x
0
, x(t
1
) = x
1
, where t
0
= t
1
.
3
For xed , x(, ) denotes the function t x(t, )
29
J(x) gives the arc length of the curve x(t) between the two given points
(t
0
, x
0
) and (t
1
, x
1
). We wish to determine the curve x

that yields the

minimal arc length.
(a) Show that the Euler equation of the problem may be written
(E)
x

1 + x
= c, where c is a constant dierent from 1.
(b) Show that (E) has a unique solution x

that satises the given end-

point conditions.
(c) Show that among the solutions of (E) there exists a parameter family
given by x(t, ) = x

(t) +, 1 < < 1, where x

is as in (b), and that the

conditions of Theorem 3 are satied such that x

gives a (relative) minimum.

(d) Show that F(t, x, x) is convex i (x, x). Conclude by Theorem 1 that
x

solves the minimum problem.

Other endpoint conditions. Next we will consider optimation prob-
lems for which the left end point x(t
0
) of the admissible functions x is xed,
whereas the right end point x(t
1
) is free. In this case we have the following
result:
Theorem 4. Assume that x
0
is a given number. A necessary condition
for a C
2
-function
4
x

to solve the problem

max (min)
_
t
1
t
0
F(t, x(t), x(t)) dt, x(t
0
) = x
0
, x(t
1
) is free,
is that x

satises the Euler equation and, in addition, the condition

(T)
_
F
x
_
t=t
1
= 0.
If the function F(t, x, x) is concave (respectively convex) in (x, x) for each
t [t
0
, t
1
], then any admissible function x

that satises the Euler equation

and the condition (T), solves the maximum (respectively minimum) problem.
The condition (T) is called the transversality condition.
Proof. We will give a proof of the maximum problem. The proof of
the minimum problem is similar. Assume that x

solves the problem. In

particular all admissible (C
2
-) functions x which have the same value x

(t
1
)
4
As can be shown, it suces to consider C
1
-functions.
30
as x

at the right endpoint t = t

1
, satisfy J(x) J(x

), hence x

is optimal
among those functions. However, then x

satises the Euler equation. In the

proof of the Euler equation, see the proof of Theorem 1, we considered the
function
I() =
_
t
1
t
0
F(t, x

+, x

+ ) dt
After integration by parts and dierentiation under the integral, we con-
cluded that
0 = I

(0) =
_
t
1
t
0
[
F

x

d
dt
(
F

x
)] dt +
_
F

x
(t)

t
1
t
0
,
where
()
F

x
=
F
x
(t, x

(t), x

(t)) and
F

x
=
F
x
(t, x

(t), x

(t))
Here, at this point, we let be a C
2
-function such that (t
0
) = 0 and (t
1
) is
free. Since x

satises the Euler equation, the integral in () must be equal

to zero, hence
_
F

x
_
t=t
1
(t
1
) = 0.
If we choose a with (t
1
) = 0, it follows that (
F

x
)
t=t
1
= 0.
Suciency: Assume that F(t, x, x) is concave in (x, x), and that x

is
an admissible function which satises the Euler equation and the condition
(T). The argument led to the inequality
_
t
1
t
0
_
F(t, x, x) F(t, x

, x

)
_
dt

_
t
1
t
0
d
dt
_
F

x
_
(x x

) dt =
_
F

x
(x x

)
_
t
1
t
0
The last part of the proof of Theorem 1, goes through as before. Here we
nd
_
F

x
(x x

)
_
t=t
0
= 0
since x(t
0
) = x

(t
0
) = x
0
. Finally the condition (T) yields
_
F

x
(x x

)
_
t=t
1
= 0.
Thus it follows that J(x) J(x

), hence x

maximizes J.
31
3 OPTIMAL CONTROL THEORY.
In the Calculus of Variations we studied problems of the type
max/min
_
t
1
t
0
f(t, x, x) dt, x(t
0
) = x
0
, x(t
1
) = x
1
(or x(t
1
) free).
We assumed that all the relevant functions were of class C
2
(twice continu-
ously dierentiable). If we let
u = x,
then the above problem may be reformulated as
(0) max/min
_
t
1
t
0
f(t, x, u) dt, x = u, x(t
0
) = x
0
, x(t
1
) = x
1
(or x(t
1
) free).
We shall next study a more general class of problems, problems of the type:
(1)
_

_
max/min
_
t
1
t
0
f(t, x(t), u(t)) dt, x(t) = g(t, x(t), u(t)),
x(t
0
) = x
0
, x(t
1
) free,
u(t) R, t [t
0
, t
1
], u piecewise continuous.
Later we shall also consider such problems with other endpoint conditions
and where the functions u can take values in more general subsets of R.
Such problems are called control problems, u is called a control vari-
able (or just a control). The control region is the common range of the
possible controls u.
5
Pairs of functions (x, u) that satises the given end-
point conditions and, in addition, the equation of state x = g(t, x, u), are
called admissible pairs.
We shall always assume that the controls u are piecewise continuous, that
is, u may possess a nite number of jump discontinuities. (In more advanced
texts measurable contols are considered.) In the calculus of variations we
always assumed that u = x was of class C
1
.
In the optimal control theory it proves very useful to apply an auxiliary
function H of four variables dened by
H(t, x, u, p) = f(t, x, u) +pg(t, x.u),
called the Hamilton function (or Hamiltonian) of the given problem.
5
Variable control regions may also be considered
32
Pontryagins Maximum principle gives conditions that are necessary for
an admissible pair (x

, u

) to solve a given control problem:

Theorem (The Maximum Principle I)
Assume that (x

, u

) is an optimal pair for the problem in (1). Then

there exist a continuous function p = p(t), such that for all t [t
0
, t
1
], the
following conditions are satised:
(a) u

(t) maximizes H(t, x

(t), u, p(t)), u R, that is,

H(t, x

(t), u, p(t)) H(t, x

(t), u

(t), p(t)), for all u R.

(b) The function p (called the adjoint function ) satises the dierential
equation
p(t) =
H
x
(t, x

(t), u

(t), p(t)) (written

H

x
),
except at the discontinuities of u

.
(c) The function p obeys the condition
(T) p(t
1
) = 0 (the transversality condition)
Remark.
For the sake of simplicity we will frequently use the notations
H

x
=
H
x
(t, x

(t), u

(t), p(t))
and similarly,
f

= f(t, x

(t), u

(t)), g

= g(t, x

(t), u

(t)), H

= H(t, x

(t), u

(t), p(t))
Remark.
For the varition problem in (0), where x = u, the Hamiltonian H will be
particularly simple,
H(t, x, u, p) = f(t, x, u) +pu,
hence, by the Maximum Principle, it is necessary that
(i) p(t) =
H

x
=
f

x
33
Since u R, we must have (there are no endpoints to consider here)
0 =
H

u
=
f

u
+p(t).
or
(ii) p(t) =
f

u
=
f

x
,
in order that u = u

(t) shall maximize H. Equations (ii) and (i) now yield

p(t) =
d
dt
(
f

x
) =
f

x
,
Hence we have deduced the Euler equation
(E)
f

x

d
dt
(
f

x
) = 0
As p(t) =
f

x
, the condition (T) above yields the old transversality
condition (
f

x
)
t=t
1
= 0 from the Calculus of Variations. This shows that
The Maximum Principle implies our previous results from The Calculus of
Variations.
The following theorem of Mangasarian supplements the Maximum Prin-
ciple with sucient conditions for optimality.
Mangasarians Theorem (First version)
Let the notation be as in the statement of the Maximum Principle. If the
map (x, u) H(t, x, u, p(t)) is concave for each t [t
0
, t
1
], then each admis-
sible pair (x, u

) that satises conditions (a), (b), and (c) of the Maximum

Principle, will give a maximum.
Example 1
We will solve the problem
max
_
T
0
[1 tx(t) u(t)
2
] dt, x = u(t), x(0) = x
0
, x(T) free,
where x
0
and T are positive constants.
Solution.
The Hamiltonian is
H(t, x, u, p) = 1 tx u
2
+pu
34
In order that u = u

(t) shall maximize H(t, x

(t), u, p(t)), it is necessary that

H
u
(t, x

(t), u, p(t)) = 0,
that is 2u + p(t) = 0, or u =
1
2
p(t). This yields a maximum since

2
H

u
2
=
2 < 0. Hence
u

(t) = p(t)/2, t [0, T].

Furthermore,
H
x
= t = p(t),
so that p = t, and p(t) =
1
2
t
2
+ A. In addition, the condition (T) gives
0 = p(T) =
1
2
T
2
+A, hence A =
1
2
T
2
. Thus
p(t) =
1
2
(t
2
T
2
)
and
x

(t) = u

(t) =
1
2
p(t) =
1
4
(t
2
T
2
),
x

(t) =
1
12
t
3

1
4
T
2
t +x
0
.
Here H(t, x, u, p(t)) = 1 tx u
2
+ pu is concave in (x, u) for each t, being
the sum of the two concave functions (x, u) pu tx (which is linear)
and (x, u) 1 u
2
, (which is constant in x and concave as a function
of u). As an alternative, we could have used the second derivative test.
Accordingly Mangasarians Theorem shows that (x

, u

) is an optimal pair
for the problem.
Example 2
max
_
T
0
(x u
2
) dt, x = x +u, x(0) = 0, x(T)free, u R.
Here
H(t, x, u, p) = x u
2
+p(x +u) = u
2
+pu + (1 + p)x,
which is concave in (x, u). Let us maximize H with respect to u:
H
u
=
2u +p = 0 if and only if u =
1
2
p. This gives a maximum as

2
H
u
2
= 2 < 0.
As a consequence we nd x = x +
1
2
p, p =
H

x
= (1 +p) Hence
_
dp
1 +p
=
_
(1) dt, log |1 +p| = t +C, |1 +p| = Ae
t
35
From (T) we nd p(T) = 0, thus 1 = Ae
T,
A = e
T
. Accordingly, 1 + p =
e
Tt
, p(t) = e
Tt
1 (p(T) = 0 hence the plus sign must be used.) It
follows that x x =
1
2
p =
1
2
[e
Tt
1],
d
dt
(e
t
x) =
1
2
[e
T2t
e
t
],
x(t) =
1
4
e
Tt
+
1
2
+De
t
Now
x(0) = 0 =
1
4
e
T
+
1
2
+D, D =
1
2
(1 +
1
2
e
T
) =
1
4
e
T
) =
1
4
e
T

1
2
,
hence
x

(t) =
1
4
e
Tt
+
1
4
e
T+t

1
2
e
t
+
1
2
=
1
4
[e
T+t
e
Tt
] +
1
2
(1 e
t
)

Next we willl study control problems of the form

(2)
_

_
max/min
_
t
1
t
0
f(t, x(t), u(t)) dt, x(t) = g(t, x(t), u(t)),
x(t
0
) = x
0
, x(t
1
) free,
u(t) U, t [t
0
, t
1
], u piecewise continuous,
U a given interval in R
We shall assume that f and g are C
1
functions and that x(t
1
) satises
exactly one of the following three terminal conditions:
(3)
_

_
(i) x(t
1
) = x
1
, where x
1
is given
(ii) x(t
1
) x
1
, where x
1
is given
(iii) x(t
1
) is free
In the present situation it turns out that we must consider two dierent pos-
sibilities for the Hamilton function in order to obtain the correct Maximum
Principle.
(A) H(t, x, u, p) = f(t, x, u) +pg(t, x, u) (normal problems)
or
(B) H(t, x, u, p) = pg(t, x, u) (degenerate problems)
36
Case (A) is by far the most interesting one. Consequently, we shall almost
always discuss normal problems in our applications. Note that in case (B)
the Hamiltonian does not depend on the integrand f.
Remark. If x(t
1
) is free (as in (iii)), then the problem will always be
normal. More general, if p(t) = 0 for some t [t
0
, t
1
], then it can be shown
that the problem is normal.
Pontryagins Maximum Principle takes the following form.
Theorem (The Maximum Principle II) Assume that (x

, u

) is an
optimal pair for the control problem in (2) and that one of the terminal
conditions (i), (ii), or (iii) is satised. Then there exists a continuous and
piecewise dierentiable function p such that p(t
1
) satises exactly one of the
following transversality conditions:
_

_
(i

) no condition on p(t
1
), if x(t
1
) = x
1
(ii

) p(t
1
) 0 if x(t
1
) x
1
, (p(t
1
) = 0 if x(t
1
) > x
1
)
(iii

) p(t
1
) = 0 if x(t
1
) is free.
In addition, for each t [t
0
, t
1
], the following conditions must hold
(a) u = u

(t) maximizes H(t, x

(t), u, p(t)), u U, that is,

H(t, x

(t), u, p(t)) H(t, x

(t), u

(t), p(t)), for all u U.

(b) The function p (called the adjoint function) satises the dierential
equation
p(t) =
H
x
(t, x

(t), u

(t), p(t)) (written

H

x
),
except at the discontinuity points of u

.
(c) If p(t) = 0 for some t [t
0
, t
1
], then the problem is normal.
Once again concavity of the Hamiltonian H in (x, u) yields suciency for
the existence of an optimal pair:
Mangasarians Theorem II. Assume that (x

, u

) is an admissible
pair for the maximum problem given in (2) and (3) above. Assume further
that the problem is normal. If the map (x, u) H(t, x, u, p(t)) is concave for
each t [t
0
, t
1
], then each admissible pair (x, u

) that satises conditions

(a), (b), and (c) of the Maximum Principle II, will maximize the integral
_
t
1
t
0
f(t, x, x) dt.
37
In contrast to the Maximum Principle, the proof of Mangasarians Theo-
rem is neither particularly long nor dicult. For the proof we shall need the
following
Lemma. If is a concave real-valued C
1
function on an interval I,
then
has a maximum at x
0
I

(x
0
)(x
0
x) 0, for all x I.
Proof. We let a < b be the endpoints of I.
=: Assume that x : 0 is a max point for . Since the derivative

exists
there are exactly three possibilities:
(1) x
0
= a and

(x
0
) 0,
(2) a < x
0
< b and

(x
0
) = 0, and
(3) x
0
= b and

(x
0
) 0,
In all three cases we have

(x
0
)(x
0
x) 0.
=: Suppose that

(x
0
)(x
0
x) 0. Again there are three possibilities:
(1) x
0
= a: Then x
0
x 0, hence

(x
0
) 0. Since is concave, the
tangent lies above or on the graph of at each point. Hence

decreases
(this may also be shown analytically) so that

(x)

(a) =

(x
0
) 0, for all x I.
Hence decreases and x
0
= a is a maximum point.
(2) a < x
0
< b: For any x I which satises x
0
< x we have x
0
x < 0,
hence

(x
0
) 0. On the other hand, if x
0
> x, then x x
0
> 0, and hence

(x
0
) 0 too. It follows that (x
0
) = 0. As

is decreasing, x
0
must be a
max point for .
(3) x
0
= b: Then x
0
x 0, hence

(x
0
) 0. Since

is decreasing,

(x) 0 for all x I. Therefore is increasing and x

0
= b must be a
maximum point.
Proof of Mangasarians Theorem. Assume that (x

, u

) is an admis-
sible pair and satises conditions (a), (b), and (c) in the hypothesis of the
Maximum Principle. Assume further that the map
(x, u) H(t, x, u, p(t))
is concave for each t [t
0
, t
1
]. Let (x, u) be any admissible pair for the control
problem. We must show that
=
_
t
1
t
0
f(t, x

(t), u

(t)) dt
_
t
1
t
0
f(t, x(t), u(t)) dt 0
38
Let us introduce the simplied notations
f

= f(t, x

(t), u

(t)), f = f(t, x(t), u(t)),

and similarly for g

, g, H

, and H. Thus
H

= H(t, x

(t), u

(t), p(t)), H = H(t, x(t), u(t), p(t)).

Then
H

= f

+pg

,
hence
f

= H

pg

= H

p x

since x

= g

by condition (b))
and
f = H p x.
It follows that
=
_
t
1
t
0
(f

f) dt =
_
t
1
t
0
(H

p x

H +p x) dt
=
_
t
1
t
0
p( x

x) dt
_
t
1
t
0
(H H

) dt
Since H is concave with respect to (x, u) the gradient inequality (see Propo-
sition 9 of Section 1) holds
H H

x
(x x

) +
H

u
(u u

)
= p(x x) +
H

u
(u u

)
p(x x) (by the last Lemma),
except at the discontinuities of u

. For the last inequality above we used

that
H

u
(u

u) 0 which holds since H is concave in (x, u), hence in u,

Lemma. Consequently,
39

_
t
1
t
0
p( x x

) dt +
_
t
1
t
0
p(x x

) dt
_
t
1
t
0
d
dt
[p(x x

)] dt ( here we used that the problem is normal ).

= |
t
1
t
0
p(x x

)
= p(t
1
)[x(t
1
) x

(t
1
)] p(t
0
)[x(t
0
) x

(t
0
)]
= p(t
1
)[x(t
1
) x

(t
1
)] (since x(t
0
) = x
0
= x

(t
0
))
By the terminal conditions,
(i) x(t
1
) = x
1
= x

(t
1
) : 0 is clear.
(ii) x(t
1
) x
1
: If x

(t
1
) = x
1
, then
x(t
1
) x
1
= x

(t
1
), and since p(t
1
) 0 we nd that 0
(iii) x(t
1
) is free: p(t
1
) = 0, hence 0.
Hence we have shown that the pair (x

, u

) is optimal.
Example 3 (SSS 12.4 Eksempel 1)
We shall solve the control problem
max
_
1
0
x(t) dt, x = x+u, x(0) = 0, x(1) 1, u(t) [1, 1] = U, for all t [0, 1].
We will assume (as usual) that the problem is normal. Hence the Hamiltonian
is
H(t, x, u, p) = x +p(x +u) = (1 +p)x +pu
which is linear in (x, u)m hence it is concave (for all t [0, 1]). Thus Man-
gasarians Theorem applies and any admissible pair (x

, u

) that satises
the conditions of the Maximum Principle will solve the problem. Here the
transversality condition
(iii

) p(1) 0(p(1) = 0 if x(1) > 1)

applies. Since H is linear in u, the maximum is attained at one of the
endpoints u = 1 or u = 1, hence
u

(t) =
_

_
1, if p(t) > 0
1, if p(t) < 0
undetermined if p(t) = 0. We let u

(t) = 1 in this case.

40
Furthermore, for x = x

, u = u

,
H
x
= 1 +p = p (from condition (b) in the Maximum Principle)
Hence p +p = 1 which yields p(t) = Ae
t
1, where p(1) = Ae
1
1 0,
so that A e. Therefore
p(t) = Ae
t
1 e
1t
1 = h(t),
where
h

(t) = e
1t
< 0, and h(1) = 0, h(0) = e 1 > 0.
Hence h(t) [0, e 1]. In particular,
p(t) h(t) > 0 for t [0, 1), p(1) 0.
Accordingly
u = u

(t) = 1, t [0, 1].

It follows that
x

= 1, x

(t) = Be
t
1.
Further,
x

(0) = 0 B = 1 x

(t) = e
t
1.
Thus x

(1) = e 1 > 0, so that p(1) = 0, and A = e. We have found

x

(t) = e
t
1, u

(t) = 1, for all t [0, 1]

and this is an optimal pair by Mangasarians Theorem.
An inspection of the proof of Mangasarians Theorem reveals that it will
work locally if the function (x, u) H(t, x, u, p(t)) is concave only in some
open and convex subset V of R
2
. In fact, under these hypothesis the Gradient
Inequality (Proposition 1.9) is valid. This yields
Corollary. Assume that (x

, u

) is an admissible pair for the maximum

problem given in (2) and (3) above. Assume further that the problem is
normal. Let V be an open and convex subset of R
2
and let
W = {(x, u) : (x, u) is admissible and (x(t), u(t)) V for all t [t
0
, t
1
]}.
If for each t [t
0
, t
1
] the map (x, u) H(t, x, u, p(t)) is concave on V , then
each admissible pair (x

, u

) that satises conditions (a), (b), and (c) of the

41
Maximum Principle II, will maximize the integral
_
t
1
t
0
f(t, x, x) dt among all
admissible pairs (x, u) in W.
Arrows condition.
Mangasarians Theorem requires that the Hamilton function is concave with
respect to (x, u). In many important applications this condition is not sat-
ised. Consequently, we will consider a weaker but related condition that in
some cases is sucient for optimality. Dene

H(t, x, p) = max
uU
H(t, x, u, p)
where we assume that the maximum exists. Then we have
Arrows Suciency Condition. In the control problem (2) above as-
sume that the Hamilton function is given by
H(t, x, u, p) = f(t, x, u) +pg(t, x, u).
(that is, the problem is normal). If the conditions of the Maximum Principle
are satised for an admissible pair (x

, u

) and the function x

H(t, x, p(t))
is concave for each t [t
0
, t
1
], then (x

, u

) is optimal for the problem.

Example. We shall apply the Maximum Principle to nd a possible
optimal pair for the problem. Then we will show that this pair is indeed
optimal.
max
_
1
1
(tx u
2
) dt, where x = x +u
2
, u(t) [0, 1] for all t [1, 1],
and with the endpoint conditions x(1) = 2e
1
1, x(1) free.
Solution:
Assume that (x

, u

) is an optimal pair. The Hamilton function is

H(t, x, u, p) = tx u
2
+p(x +u
2
) = (t +p)x + (p 1)u
2
.
According to the Maximum Principle there exists a continuous function p(t)
such that for each t the control u = u

(t) maximizes H(t, x

(t), u, p(t)). Here

H
u
= 2(p 1)u = 0 if and only if u = 0 or p = 1.
The second derivative of H with respect to u is equal to 2(p 1) and it is
negative p < 1. Consequently, whenever
42
p = p(t) > 1 :
u = u

(t) = 0 yields a minimum for H. Further, u = u

(t) = 1 gives a
maximum, and for all such t the function x

must satisfy the equation

x

= 1, hence x

(t) = Ae
t
1. Next, if
p = p(t) < 1 :
u = u

(t) = 0 maximizes H, so that x

= 0, x

(t) = Be
t
.
Furthermore,
p(t) =
H
x
= t +p(t),
hence p(t) + p(t) = t, p(t) = 1 t + Ce
t
. Here p(1) = C = 0, since x(1)
is free. Therefore,
p(t) = 1 t, t [1, 1].
Thus p(t) > 1 t < 0, so that p(t) > 1 on [1, 0) and p(t) < 1 on (0, 1].
t [1, 0) :
p(t) > 1, x

(t) = Ae
t
1, where x(1) = Ae
1
1 = 2e
1
1, A = 2.
Hence x

(t) = 2e
t
1, u

(t) = 1.
t (0, 1] :
p(t) < 1, x

(t) = Be
t
. Continuity of x

at t = 0 gives x

(0) = B = 2 1 =
3. Therefore,
x

(t) = 3e
t
, u

(t) = 0
t = 1 :
u

(1) can be assigned any value in [0, 1], since H is constant in u. Let us
choose the value u

(1) = 1.
Then the pair (x

, u

) is the only possible candidate for a solution.

It remains to prove that (x

, u

) is in fact optimal. We see that H(t, x, u, p(t))

is concave in (x, u) p(t) 1 t [0, 1]. Hence Mangasarians Theorem
does not apply. However, we observe that

H(t, x, p(t)) = max

u[0,1]
H(t, x, u, p(t)) = (t +p(t)x) +p(t) 1 if p(t) > 1
and

H(t, x, p(t)) = (t +p(t))x, if p(t) 1,

whence

H is linear and is therefore concave in x. Accordingly, Arrows The-
orem implies that the pair (x

, u

) solves the problem.

Exercise. Solve the control problem in the last example if the end point
conditions instead are
x(1) = 0, x(1) = e
2
e
1+
1
e
.
43
Hint: Try with p(t) < 1 on an interval (t
0
, 1] (p denotes the adjoint function).
44
REFERENCES
[EP] Edwards & Penney, Elementary Dierential Equations, 2009. Upper
Saddle River, N.J. : Pearson Prentice Hall. ISBN: 978-0-13-235881-1.
[GF] Gelfand, I. M., Fomin, S. V., Calculus of variations, Prentice-Hall,
1963.
[O] Osgood, W. F., Sucient Conditions in the Calculus of Variations,
The Annals of Mathematics, 2nd Series, Vol. 2, No. 1/4, 1900-1901 (105-129)
(Online at: www.jstor.org/stable/2007189)
[PBGM] Pontryagin, Boltyanskii, Gamkrelidze, Mischchenko, The Math-
ematical Theory of Optimal Processes, Interscience Publishers (Wiley), 1962.
[SSS] Sydster K., Seierstad A., Strm, A., Matematisk Analyse, Bind
2, Gyldendal Akademisk 2004.
[SS] Seierstad, A., Sydster, K., Optimal Control Theory with Economic
Applications, North-Holland 1987. ISBN: 0-444-87923-4.
45