ECEN 629 Convex Optimization for Electrical Engineering

Chapter 3. Convex Functions

Instructor: Dr. Chao TIAN

Texas A&M University-College Station

Boyd, VandenBerghe, Tian Spring 2018 1 / 12

 Definition
f : Rn → R is convex if dom f is a convex set and

f (θx + (1 − θ)y) ≤ θf (x) + (1 − θ)f (y)

for all x, y ∈ dom f , 0 ≤ θ ≤ 1

(y, f (y))
(x, f (x))

• f is concave if −f is convex
• f is strictly convex if dom f is convex and

f (θx + (1 − θ)y) < θf (x) + (1 − θ)f (y)

for x, y ∈ dom f , x 6= y, 0 < θ < 1

Convex functions 3–2

 Examples on R

convex:
• affine: ax + b on R, for any a, b ∈ R
• exponential: eax, for any a ∈ R
• powers: xα on R++, for α ≥ 1 or α ≤ 0
• powers of absolute value: |x|p on R, for p ≥ 1
• negative entropy: x log x on R++

concave:
• affine: ax + b on R, for any a, b ∈ R
• powers: xα on R++, for 0 ≤ α ≤ 1
• logarithm: log x on R++

Convex functions 3–3

 Examples on Rn and Rm×n
affine functions are convex and concave; all norms are convex
examples on Rn
• affine function f (x) = aT x + b
Pn
• norms: kxkp = ( i=1 |xi|p)1/p for p ≥ 1; kxk∞ = maxk |xk |

examples on Rm×n (m × n matrices)

• affine function
m X
X n
f (X) = tr(AT X) + b = Aij Xij + b
i=1 j=1

• spectral (maximum singular value) norm

f (X) = kXk2 = σmax(X) = (λmax(X T X))1/2

Convex functions 3–4

 Restriction of a convex function to a line

f : Rn → R is convex if and only if the function g : R → R,

g(t) = f (x + tv), dom g = {t | x + tv ∈ dom f }

is convex (in t) for any x ∈ dom f , v ∈ Rn

can check convexity of f by checking convexity of functions of one variable

example. f : Sn → R with f (X) = log det X, dom f = Sn++

g(t) = log det(X + tV ) = log det X + log det(I + tX −1/2V X −1/2)

Xn
= log det X + log(1 + tλi)
i=1

where λi are the eigenvalues of X −1/2V X −1/2

g is concave in t (for any choice of X ≻ 0, V ); hence f is concave

Convex functions 3–5

 Extended-value extension

extended-value extension f˜ of f is

f˜(x) = f (x), x ∈ dom f, f˜(x) = ∞, x 6∈ dom f

often simplifies notation; for example, the condition

0≤θ≤1 =⇒ f˜(θx + (1 − θ)y) ≤ θf˜(x) + (1 − θ)f˜(y)

(as an inequality in R ∪ {∞}), means the same as the two conditions

• dom f is convex
• for x, y ∈ dom f ,

0≤θ≤1 =⇒ f (θx + (1 − θ)y) ≤ θf (x) + (1 − θ)f (y)

Convex functions 3–6

 First-order condition

f is differentiable if dom f is open and the gradient

 
∂f (x) ∂f (x) ∂f (x)
∇f (x) = , ,...,
∂x1 ∂x2 ∂xn

exists at each x ∈ dom f

1st-order condition: differentiable f with convex domain is convex iff

f (y) ≥ f (x) + ∇f (x)T (y − x) for all x, y ∈ dom f

f (y)
f (x) + ∇f (x)T (y − x)

(x, f (x))

first-order approximation of f is global underestimator

Convex functions 3–7

 Second-order conditions

f is twice differentiable if dom f is open and the Hessian ∇2f (x) ∈ Sn,

2 ∂ 2f (x)
∇ f (x)ij = , i, j = 1, . . . , n,
∂xi∂xj

exists at each x ∈ dom f

2nd-order conditions: for twice differentiable f with convex domain

• f is convex if and only if

∇2f (x)  0 for all x ∈ dom f

• if ∇2f (x) ≻ 0 for all x ∈ dom f , then f is strictly convex

Convex functions 3–8

 Global Properties and Strict Convexity

From the first order condition f (y ) ≥ f (x ) + ∇f (x )T (y − x )

• From local properties to global properties;
• When ∇f (x )T = 0 then for all y ∈ domf , f (y ) ≥ f (x ).

Boyd, VandenBerghe, Tian Spring 2018 2 / 12

 Global Properties and Strict Convexity

From the first order condition f (y ) ≥ f (x ) + ∇f (x )T (y − x )

• From local properties to global properties;
• When ∇f (x )T = 0 then for all y ∈ domf , f (y ) ≥ f (x ).

Strict convexity
• If and only if domf is convex and for any x , y ∈ domf , x 6= y

f (y ) > f (x ) + ∇f (x )T (y − x ).

Boyd, VandenBerghe, Tian Spring 2018 2 / 12

 Global Properties and Strict Convexity

From the first order condition f (y ) ≥ f (x ) + ∇f (x )T (y − x )

• From local properties to global properties;
• When ∇f (x )T = 0 then for all y ∈ domf , f (y ) ≥ f (x ).

Strict convexity
• If and only if domf is convex and for any x , y ∈ domf , x 6= y

f (y ) > f (x ) + ∇f (x )T (y − x ).

Domain being convex is important

• Consider f (x ) = 1/x 2 with domf = {x ∈ R|x 6= 0}.
• f ′′ (x ) > 0, but it is not convex.

Boyd, VandenBerghe, Tian Spring 2018 2 / 12

 Examples
quadratic function: f (x) = (1/2)xT P x + q T x + r (with P ∈ Sn)

∇f (x) = P x + q, ∇2f (x) = P

convex if P  0
least-squares objective: f (x) = kAx − bk22

∇f (x) = 2AT (Ax − b), ∇2f (x) = 2AT A

convex (for any A)

quadratic-over-linear: f (x, y) = x2/y 2

f (x, y)
  T 1
2 y y
∇2f (x, y) = 3 0
y −x −x 0
2 2
1 0
convex for y > 0 y 0 −2 x

Convex functions 3–9

 Pn
log-sum-exp: f (x) = log k=1 exp xk is convex

1 1
∇2f (x) = diag(z) − zz T
(zk = exp xk )
1T z (1T z)2

to show ∇2f (x)  0, we must verify that v T ∇2f (x)v ≥ 0 for all v:

2
zk ) − ( k v k zk ) 2
P P P
T 2 ( k zk vk )( k
v ∇ f (x)v = ≥0
( k zk ) 2
P

2 2
P P P
since ( k v k zk ) ≤( k zk vk )( k zk ) (from Cauchy-Schwarz inequality)

Qn 1/n n
geometric mean: f (x) = ( k=1 x k ) on R ++ is concave

(similar proof as for log-sum-exp)

Convex functions 3–10

 Epigraph and sublevel set

α-sublevel set of f : Rn → R:

Cα = {x ∈ dom f | f (x) ≤ α}

sublevel sets of convex functions are convex (converse is false)

epigraph of f : Rn → R:

epi f = {(x, t) ∈ Rn+1 | x ∈ dom f, f (x) ≤ t}

epi f

f is convex if and only if epi f is a convex set

Convex functions 3–11

 Sublevel Sets and Convex Functions

Sublevel sets of a convex function are convex:

• If x , y ∈ Cα , then f (x ) ≤ α and f (y ) ≤ α

f (θx + (1 − θ)y ) ≤ α ⇒ θx + (1 − θ)y ∈ Cα .

Boyd, VandenBerghe, Tian Spring 2018 3 / 12

 Sublevel Sets and Convex Functions

Sublevel sets of a convex function are convex:

• If x , y ∈ Cα , then f (x ) ≤ α and f (y ) ≤ α

f (θx + (1 − θ)y ) ≤ α ⇒ θx + (1 − θ)y ∈ Cα .

• Can be used to establish a set is convex. E.g. fix α ∈ [0, 1]

 
 n !1/n n
1
  Y X 
x ∈ Rn+  xi ≥α xi .
 n 
i=1 i=1

Boyd, VandenBerghe, Tian Spring 2018 3 / 12

 Sublevel Sets and Convex Functions

Sublevel sets of a convex function are convex:

• If x , y ∈ Cα , then f (x ) ≤ α and f (y ) ≤ α

f (θx + (1 − θ)y ) ≤ α ⇒ θx + (1 − θ)y ∈ Cα .

• Can be used to establish a set is convex. E.g. fix α ∈ [0, 1]

 
 n !1/n n
1
  Y X 
x ∈ Rn+  xi ≥α xi .
 n 
i=1 i=1

A non-convex function can have all its sublevel sets convex

• Example: f (x ) = −e x .

Boyd, VandenBerghe, Tian Spring 2018 3 / 12

 Jensen’s inequality

basic inequality: if f is convex, then for 0 ≤ θ ≤ 1,

f (θx + (1 − θ)y) ≤ θf (x) + (1 − θ)f (y)

extension: if f is convex, then

f (E z) ≤ E f (z)

for any random variable z

basic inequality is special case with discrete distribution

prob(z = x) = θ, prob(z = y) = 1 − θ

Convex functions 3–12

 Inequalities

Many known inequalities are derived through convexity:

√
• Arithematic-geometric mean inequality ab ≤ (a + b)/2 for a, b ≥ 0.

a+b − log a − log b

   
f (x ) = − log x is convex ⇒ − log ≤
2 2

Boyd, VandenBerghe, Tian Spring 2018 4 / 12

 Inequalities

Many known inequalities are derived through convexity:

√
• Arithematic-geometric mean inequality ab ≤ (a + b)/2 for a, b ≥ 0.

a+b − log a − log b

   
f (x ) = − log x is convex ⇒ − log ≤
2 2

• Hölder’s inequality: for p > 1, 1/p + 1/q = 1 and x , y ∈ Rn

X X 1/p X 1/q
p q
xi yi ≤ |xi | |yi |

Boyd, VandenBerghe, Tian Spring 2018 4 / 12

 Operations that preserve convexity

practical methods for establishing convexity of a function

1. verify definition (often simplified by restricting to a line)

2. for twice differentiable functions, show ∇2f (x)  0

3. show that f is obtained from simple convex functions by operations

that preserve convexity
• nonnegative weighted sum
• composition with affine function
• pointwise maximum and supremum
• composition
• minimization
• perspective

Convex functions 3–13

Positive weighted sum & composition with affine function

nonnegative multiple: αf is convex if f is convex, α ≥ 0

sum: f1 + f2 convex if f1, f2 convex (extends to infinite sums, integrals)

composition with affine function: f (Ax + b) is convex if f is convex

examples

• log barrier for linear inequalities

X
m
f (x) = − log(bi − aTi x), dom f = {x | aTi x < bi, i = 1, . . . , m}
i=1

• (any) norm of affine function: f (x) = kAx + bk

Convex functions 3–14

 Pointwise maximum

if f1, . . . , fm are convex, then f (x) = max{f1(x), . . . , fm(x)} is convex

examples

• piecewise-linear function: f (x) = maxi=1,...,m(aTi x + bi) is convex

• sum of r largest components of x ∈ Rn:

f (x) = x[1] + x[2] + · · · + x[r]

is convex (x[i] is ith largest component of x)

proof:
f (x) = max{xi1 + xi2 + · · · + xir | 1 ≤ i1 < i2 < · · · < ir ≤ n}

Convex functions 3–15

 Proof: Pointwise Maximum is Convex

f (θx + (1 − θ)y ) = max{f1 (θx + (1 − θ)y ), f2 (θx + (1 − θ)y )}

≤ max{θf1 (x ) + (1 − θ)f1 (y ), θf2 (x ) + (1 − θ)f2 (y )}
≤ θ max{f1 (x ), f2 (x )} + (1 − θ) max{f1 (y ), f2 (y )}
= θf (x ) + (1 − θ)f (y ).

Boyd, VandenBerghe, Tian Spring 2018 5 / 12

 Pointwise supremum
if f (x, y) is convex in x for each y ∈ A, then

g(x) = sup f (x, y)

y∈A

is convex
examples
• support function of a set C: SC (x) = supy∈C y T x is convex
• distance to farthest point in a set C:

f (x) = sup kx − yk
y∈C

• maximum eigenvalue of symmetric matrix: for X ∈ Sn,

λmax(X) = sup y T Xy
kyk2 =1

Convex functions 3–16

 Composition with scalar functions

composition of g : Rn → R and h : R → R:

f (x) = h(g(x))

g convex, h convex, h̃ nondecreasing

f is convex if
g concave, h convex, h̃ nonincreasing

• proof (for n = 1, differentiable g, h)

f ′′(x) = h′′(g(x))g ′(x)2 + h′(g(x))g ′′(x)

• note: monotonicity must hold for extended-value extension h̃

examples
• exp g(x) is convex if g is convex
• 1/g(x) is convex if g is concave and positive

Convex functions 3–17

 Vector composition

composition of g : Rn → Rk and h : Rk → R:

f (x) = h(g(x)) = h(g1(x), g2(x), . . . , gk (x))

gi convex, h convex, h̃ nondecreasing in each argument

f is convex if
gi concave, h convex, h̃ nonincreasing in each argument

proof (for n = 1, differentiable g, h)

f ′′(x) = g ′(x)T ∇2h(g(x))g ′(x) + ∇h(g(x))T g ′′(x)

examples
Pm
• i=1 log gi (x) is concave if gi are concave and positive
Pm
• log i=1 exp gi(x) is convex if gi are convex

Convex functions 3–18

 Composition

Extended-value extension h̃ condition is important:

• Example: g(x ) = x 2 and domg = R and h(x ) = 0 with domh = [1, 2]
√ √
h(g(x )) = 0, domf = [− 2, −1] ∪ [1, 2].

Boyd, VandenBerghe, Tian Spring 2018 6 / 12

 Minimization

if f (x, y) is convex in (x, y) and C is a convex set, then

g(x) = inf f (x, y)

y∈C

is convex
examples
• f (x, y) = xT Ax + 2xT By + y T Cy with
 
A B
 0, C≻0
BT C

minimizing over y gives g(x) = inf y f (x, y) = xT (A − BC −1B T )x

g is convex, hence Schur complement A − BC −1B T  0
• distance to a set: dist(x, S) = inf y∈S kx − yk is convex if S is convex

Convex functions 3–19

 Minimization over a Convex Set

Proof: take an arbitraray ǫ > 0

g(θx1 + (1 − θ)x2 ) = inf f (θx1 + (1 − θ)x2 , y )

y ∈C

≤ f (θx1 + (1 − θ)x2 , θy1 + (1 − θ)y2 )

≤ θf (x1 , y1 ) + (1 − θ)f (x2 , y2 )
≤ θg(x1 ) + (1 − θ)g(x2 ) + ǫ.

Boyd, VandenBerghe, Tian Spring 2018 7 / 12

 Perspective

the perspective of a function f : Rn → R is the function g : Rn × R → R,

g(x, t) = tf (x/t), dom g = {(x, t) | x/t ∈ dom f, t > 0}

g is convex if f is convex

examples
• f (x) = xT x is convex; hence g(x, t) = xT x/t is convex for t > 0
• negative logarithm f (x) = − log x is convex; hence relative entropy
g(x, t) = t log t − t log x is convex on R2++
• if f is convex, then

T T


g(x) = (c x + d)f (Ax + b)/(c x + d)

is convex on {x | cT x + d > 0, (Ax + b)/(cT x + d) ∈ dom f }

Convex functions 3–20

 The conjugate function

the conjugate of a function f is

f ∗(y) = sup (y T x − f (x))

x∈dom f

f (x)
xy

(0, −f ∗(y))

• f ∗ is convex (even if f is not)

• will be useful in chapter 5

Convex functions 3–21

examples

• negative logarithm f (x) = − log x

f ∗(y) = sup(xy + log x)

x>0

−1 − log(−y) y < 0
=
∞ otherwise

• strictly convex quadratic f (x) = (1/2)xT Qx with Q ∈ Sn++

f ∗(y) = sup(y T x − (1/2)xT Qx)

x
1 T −1
= y Q y
2

Convex functions 3–22

 Quasiconvex functions

f : Rn → R is quasiconvex if dom f is convex and the sublevel sets

Sα = {x ∈ dom f | f (x) ≤ α}

are convex for all α

a b c

• f is quasiconcave if −f is quasiconvex
• f is quasilinear if it is quasiconvex and quasiconcave

Convex functions 3–23

 Examples
p
• |x| is quasiconvex on R
• ceil(x) = inf{z ∈ Z | z ≥ x} is quasilinear
• log x is quasilinear on R++
• f (x1, x2) = x1x2 is quasiconcave on R2++
• linear-fractional function
aT x + b
f (x) = T , dom f = {x | cT x + d > 0}
c x+d
is quasilinear
• distance ratio
kx − ak2
f (x) = , dom f = {x | kx − ak2 ≤ kx − bk2}
kx − bk2

is quasiconvex

Convex functions 3–24

 Properties
modified Jensen inequality: for quasiconvex f

0≤θ≤1 =⇒ f (θx + (1 − θ)y) ≤ max{f (x), f (y)}

first-order condition: differentiable f with cvx domain is quasiconvex iff

f (y) ≤ f (x) =⇒ ∇f (x)T (y − x) ≤ 0

∇f (x)
x

sums of quasiconvex functions are not necessarily quasiconvex

Convex functions 3–26

 Log-concave and log-convex functions
a positive function f is log-concave if log f is concave:

f (θx + (1 − θ)y) ≥ f (x)θ f (y)1−θ for 0 ≤ θ ≤ 1

f is log-convex if log f is convex

• powers: xa on R++ is log-convex for a ≤ 0, log-concave for a ≥ 0

• many common probability densities are log-concave, e.g., normal:

1 1 T
Σ−1 (x−x̄)
f (x) = p e− 2 (x−x̄)
(2π)n det Σ

• cumulative Gaussian distribution function Φ is log-concave

Z x
1 2
Φ(x) = √ e−u /2 du
2π −∞

Convex functions 3–27

 Properties of log-concave functions

• twice differentiable f with convex domain is log-concave if and only if

f (x)∇2f (x)  ∇f (x)∇f (x)T

for all x ∈ dom f

• product of log-concave functions is log-concave

• sum of log-concave functions is not always log-concave

• integration: if f : Rn × Rm → R is log-concave, then

Z
g(x) = f (x, y) dy

is log-concave (not easy to show)

Convex functions 3–28

 Log-Convex, Convex, Quasiconvex

log-convex ⇒ convex ⇒ quasiconvex

• f is convex ⇒ f is quasiconvex;
• f is log-convex ⇒ f is convex (because h is convex ⇒ e h is convex).

concave ⇒ log-convave ⇒ quasiconcave

• f is log-concave ⇒ f is quasiconcave: log function is monotone;
• f is concave ⇒ f is log-concave (because h is concave ⇒ log h is concave).

Boyd, VandenBerghe, Tian Spring 2018 8 / 12

 Exercise 1

The level set of of a function is given as f (x ) = α. Is the following function

quasi-convex, convex, concave, or quasi-concave?

Boyd, VandenBerghe, Tian Spring 2018 9 / 12

 Exercise 2

Suppose f : R → R is increasing and convex on its domain (a, b). Let g denote
its inverse. Is g(·) a convex function, a concave function, or both, or neither?

Boyd, VandenBerghe, Tian Spring 2018 10 / 12

 Exercise 3

Are the following functions convex, concave, quasiconvex, or quasiconcave?

1. e x − 1 on R;
2. f (x1 , x2 ) = x1 x2 on R2++ .

Boyd, VandenBerghe, Tian Spring 2018 11 / 12

 Exercise 4

Show the following functions are convex

1. T (x , ω) = x1 + x2 cos ω + x3 cos 2ω + ... + xn cos(n − 1)ω, and
Z 2π
f (x ) = − log T (x , ω)dω,
0

on {x ∈ Rn |T (x , ω) > 0, 0 ≤ ω ≤ 2π}.
2. f (x ) = − log det(A0 + x1 A1 + x2 A2 + ... + xn An ) where Ai ∈ Sm , on
{x |A0 + x1 A1 + x2 A2 + ... + xn An ≻ 0}
Pr
3. f (x ) = i=1 |x |[i] where |x |[i] is the i − th largest absolute value in the
components of the vector x .
Pm aiT x +bi
Pm T
ai x +bi
4. f (x ) = − log(− log( i=1
e )) on {x | i=1
e ) < 1}.

Boyd, VandenBerghe, Tian Spring 2018 12 / 12