Convex functions

Renu M. R.

March 5, 2020

1 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,

2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,

f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)

2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)

f is strictly convex if a strict inequality holds for x 6= y and

0 < α < 1.

2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)

f is strictly convex if a strict inequality holds for x 6= y and

0 < α < 1.

f is concave if −f is convex.

2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)

f is strictly convex if a strict inequality holds for x 6= y and

0 < α < 1.

f is concave if −f is convex.

This inequality also goes by the name Jensen’s inequality

2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)

f is strictly convex if a strict inequality holds for x 6= y and

0 < α < 1.

f is concave if −f is convex.

This inequality also goes by the name Jensen’s inequality

Examples?

2 / 11
Convex functions
Let C ⊂ Rn , be a convex set.
f : C → R,
is a convex function if for every x, y ∈ C, and 0 ≤ α ≤ 1,
f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y)

f is strictly convex if a strict inequality holds for x 6= y and

0 < α < 1.

f is concave if −f is convex.

This inequality also goes by the name Jensen’s inequality

Examples?

How to check for convexity?

2 / 11
First order condition

3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?

3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?

Let f be differentiable, f is convex iff,

3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?

Let f be differentiable, f is convex iff,

3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?

Let f be differentiable, f is convex iff,

1 Domain of f (C) is a convex set, and

3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?

Let f be differentiable, f is convex iff,

1 Domain of f (C) is a convex set, and

2 ∀ x, y ∈ C,
f (y) ≥ f (x) + (y − x)T ∇f (x).

3 / 11
First order condition
For a convex function is there any specific placement of tangent
and the function?

Let f be differentiable, f is convex iff,

1 Domain of f (C) is a convex set, and

2 ∀ x, y ∈ C,
f (y) ≥ f (x) + (y − x)T ∇f (x).

Proof?

3 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.

4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x)  0, ∀x ∈ C.

4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x)  0, ∀x ∈ C.

Symbol  indicates psd.

4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x)  0, ∀x ∈ C.

Symbol  indicates psd.

If Hessian is pd ∀x, then f is strictly convex.

4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x)  0, ∀x ∈ C.

Symbol  indicates psd.

If Hessian is pd ∀x, then f is strictly convex.

Is the converse true for strictly convex functions?

4 / 11
Second order conditions
Let f be a twice differentiable function from C ⊂ Rn to R.
f is convex iff
∇2 f (x)  0, ∀x ∈ C.

Symbol  indicates psd.

If Hessian is pd ∀x, then f is strictly convex.

Is the converse true for strictly convex functions? No.

4 / 11
Problems
[P1] Check if the following functions (which we saw in the
previous classes) are convex?
1 f (x) =k Ax − b k2 ,
A ∈ Rm×n , m > n. A is full rank.
 
3 3 x
2 f (x) = x − x + y − y, where x = .
y

x6
3 f (x) = 2x2 − 1.05x4 + + xy + y 2 .
6
[P2] When is the following function convex? (x ∈ Rn , P is
symmetric)
1
f (x) = xT P x + qT x + r.
2
[P3] Show that linear and affine functions are convex.

5 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.

Proof.

6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.

Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ )  0.

6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.

Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ )  0.

f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,

6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.

Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ )  0.

f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,

S = {x | k x − x∗ k< }.

6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.

Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ )  0.

f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,

S = {x | k x − x∗ k< }.

Let y ∈ C and z = x∗ + α(y − x∗ ), 0 ≤ α ≤ 1.

6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.

Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ )  0.

f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,

S = {x | k x − x∗ k< }.

Let y ∈ C and z = x∗ + α(y − x∗ ), 0 ≤ α ≤ 1.

Choose α small enough so that z ∈ S,

6 / 11
Global minimum
Theorem
If C ⊂ Rn is convex and f : C → R is a convex function, then a
local minimum of f is also a global minimum.

Proof.
Let x∗ be a local minimum i.e. ∇f (x∗ ) = 0 and ∇2 f (x∗ )  0.

f (x∗ ) ≤ f (x), ∀x such that k x − x∗ k< ,

S = {x | k x − x∗ k< }.

Let y ∈ C and z = x∗ + α(y − x∗ ), 0 ≤ α ≤ 1.

Choose α small enough so that z ∈ S,

Use convexity to argue that x∗ is global a minimum

6 / 11
Coercive function

7 / 11
Coercive function
A continuous function f : Rn → R is coercive if

lim f (x) → ∞.
kxk→∞

7 / 11
Coercive function
A continuous function f : Rn → R is coercive if

lim f (x) → ∞.
kxk→∞

Check if the following functions are coercive.

1 f (x, y) = x2 + y 2

2 f (x, y) = x2 − 2xy + y 2

3 f (x, y) = x4 − 4xy + y 4

7 / 11
Coercive function
A continuous function f : Rn → R is coercive if

lim f (x) → ∞.
kxk→∞

Check if the following functions are coercive.

1 f (x, y) = x2 + y 2

2 f (x, y) = x2 − 2xy + y 2

3 f (x, y) = x4 − 4xy + y 4

Let a continuous function f (x) be coercive. Then f (x) has

at least one global minimum.

7 / 11
Additional problems
Check if the following functions are convex
P1 eax a, x ∈ R

P2 xa a, x ∈ R

P3 log x 0 6= x ∈ R

P4 x log x x≥0

P5 f (x) = max xi x ∈ Rn
i=1···n

P6 f (x, y) = x2 /y y > 0.

8 / 11
Additional problems
P7 Use the notion of convexity to prove
1 the AM-GM inequality

2 that for a convex function f , f (E(X)) ≤ E(f (X)), where X is a

discrete random variable and E(X) is the expectation of X.

P8 Use the AM-GM inequality to solve the following:

maximize f (x) = x1 x2 x3
subject to x1 x2 + 2x2 x3 + 2x1 x3 = S.

9 / 11
Proof slide

10 / 11
Proof
First order condition

11 / 11
Proof
First order condition

Say f (x) is convex.

11 / 11
Proof
First order condition

Say f (x) is convex.

Use the definition of directional derivative to prove the inequality.

11 / 11
Proof
First order condition

Say f (x) is convex.

Use the definition of directional derivative to prove the inequality.

Say, the inequality is true.

11 / 11
Proof
First order condition

Say f (x) is convex.

Use the definition of directional derivative to prove the inequality.

Say, the inequality is true.

Consider three points, x, y, and z, where y is a convex

combination of x and z. Using the gradient at y, show that f is
convex.

11 / 11
Proof
First order condition

Say f (x) is convex.

Use the definition of directional derivative to prove the inequality.

Say, the inequality is true.

Consider three points, x, y, and z, where y is a convex

combination of x and z. Using the gradient at y, show that f is
convex.

Second order condition

11 / 11
Proof
First order condition

Say f (x) is convex.

Use the definition of directional derivative to prove the inequality.

Say, the inequality is true.

Consider three points, x, y, and z, where y is a convex

combination of x and z. Using the gradient at y, show that f is
convex.

Second order condition

Use Taylor series expansion and use the first order condition.
Back to first order conditions.

11 / 11