Scribd
Upload
2 views

lecture_09

This document discusses the concepts of convex sets and convex functions, providing definitions, propositions, and inequalities related to them. It covers properties of convex functions, including their relationship with differentiation and operations that preserve convexity. Additionally, it introduces Jensen's inequality and provides examples and references for further reading on convex analysis.

Uploaded by

syyeung7
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
2 views

lecture_09

This document discusses the concepts of convex sets and convex functions, providing definitions, propositions, and inequalities related to them. It covers properties of convex functions, including their relationship with differentiation and operations that preserve convexity. Additionally, it introduces Jensen's inequality and provides examples and references for further reading on convex analysis.

Uploaded by

syyeung7
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Optimization and Computational Linear Algebra for Data Science

Lecture 9: Convex functions


Léo Miolane · leo.miolane@gmail.com

November 4, 2020

Warning: This material is not meant to be lecture notes. It only gathers the main concepts
and results from the lecture, without any additional explanation, motivation, examples, figures...

1 Convex sets
Definition 1.1 (Convex set)
A set C ⊂ Rn is convex if for all x, y ∈ C and all α ∈ [0, 1],

αx + (1 − α)y ∈ C.

Remark 1.1. Subspaces of Rn are convex sets.

Figure 1: Left: a convex set. Right: a non-convex set.

Definition 1.2 (Convex combination)


We say that y ∈ Rn is a convex combination of x1 , . . . , xk ∈ Rn if there exists α1 , . . . , αk ≥ 0
such that
k
X k
X
y= αi xi and αi = 1.
i=1 i=1

Proposition 1.1
If C is convex then all convex combination of elements of C remains in C.

2 Convex functions

1
Definition 2.1
A function f : Rn → R is convex if for all x, y ∈ Rn and all α ∈ [0, 1],

f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y). (1)

We say that f is strictly convex is there is strict inequality in (1) whenever x 6= y and
α ∈ (0, 1).
A function f is concave (respectively strictly concave) if −f is convex (respectively strictly
convex).

Figure 2: A convex function.

Notice that a linear function is also a convex function since it verifies (1) with equality, but
is not strictly convex.

Exercise 2.1. Let f : Rn → R a convex function and α ∈ R. Show that the “α-sublevel set”

Cα = x ∈ Rn f (x) ≤ α


is convex.

2.1 Convex function and differential

Proposition 2.1
A differentiable function f : Rn → R is convex if and only if for all x, y ∈ Rn

f (y) ≥ f (x) + ∇f (x)T (y − x).

Corollary 2.1
Let f : Rn → R be a differentiable convex function and x ∈ Rn . Then

x is a minimizer of f ⇐⇒ ∇f (x) = 0.

2
Proposition 2.2
Let f : Rn → R be a twice-differentiable function. We denote by Hf the Hessian matrix of
f . Then f is convex if and only if for all x ∈ Rn , Hf (x) is positive semi-definite.

When f : R → R is twice differentiable, we get that f is convex if and only if f 00 ≥ 0.


Remark 2.1. For strictly convex functions, the following result holds:
If for all x ∈ Rn , the Hessian Hf (x) is positive definite, then f is strictly convex.
However, the converse is not true!! Exercise: find a function for which the converse does not
hold.
It can be complicated to check that a function f is convex using Proposition 2.2 when f is a
function of multiple variables (n ≥ 2). The next proposition shows that we can always reduce to
the unidimensional case, by checking that the restriction of f on every line is convex:
Proposition 2.3
A function f : Rn → R is convex if and only if the function

g: R → R
t 7→ f (x + tv)

is convex for all x, v ∈ Rn .

2.2 Jensen’s inequality

Proposition 2.4 (Jensen’s inequality)


Let f : Rn → R be a convex function. Then for all x1 , . . . , xk ∈ Rn and all α1 , . . . , αk ≥ 0
such that ki=1 αi = 1 we have
P

k k
!
X X
f αi xi ≤ αi f (xi ).
i=1 i=1

More generally, if X is a random variable that takes value in Rn we have


  
f E[X] ≤ E f (X) .

Remark 2.2. If f is concave then Proposition 2.4 holds, but with inequalities in the reverse
order.
Example 2.1 (Discrete entropy). Let Z be a random variable that take value in {1, . . . , k} and
write pi = P(Z = i). The entropy of Z is defined as
k
X
H(Z) = − pi log(pi ).
i=1

The entropy of Z is a measure of the uncertainty associated with Z. We apply Jensen’s inequality
to the concave function log:
k k
!
X X
H(Z) = pi log(1/pi ) ≤ log pi /pi = log(k).
i=1 i=1

Notice that H(Z) = log(k) when Z is uniformly distributed over {1, . . . , k}, i.e. P(Z = i) = 1/k
for all i. Conclusion: maximal entropy is achieved for the uniform distribution.

3
2.3 Operations that preserve convexity

Proposition 2.5 (Non-negative linear combination of convex functions)


Let f1 , . . . , fk be convex functions from Rn → R and let α1 , . . . , αk ≥ 0. Then the function f
defined by
k
X
f (x) = αi fi (x)
i=1

is convex. In particular a sum of convex functions is convex.

Proposition 2.6 (Supremum of convex functions)


Let (fi )i∈S is a family of convex functions from Rn → R. Then the function

f (x) = sup fi (x)


i∈S

is convex. In particular, a supremum of affine functions is a convex function.

Proposition 2.7 (Composition with affine function)


Let f : Rn → R be a convex function, A ∈ Rn×m and b ∈ Rn . Then the function g : Rm → R
defined by
g(x) = f (Ax + b)
is convex.

Further reading
See [1] Chapters 2 and 3 for example of properties of convex sets/functions. See also http:
//web.stanford.edu/class/ee364a/lectures.html for nice lecture slides. The book [2] is a
great reference for convex analysis, but is mathematically more involved.

References
[1] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press,
https://web.stanford.edu/˜boyd/cvxbook/, 2004.

[2] R Tyrrell Rockafellar. Convex analysis, volume 28. Princeton university press, 1970.

You might also like