Theory of Probability Zitcovic PDF

Theory of Probability
Measure theory, classical probability and stochastic analysis
Lecture Notes
by Gordan itkovic
Department of Mathematics, The University of Texas at Austin
Contents
Contents
I Theory of Probability I
Measurable spaces
1.1 Families of Sets . . . . . . . . . . .
1.2 Measurable mappings . . . . . . .
1.3 Products of measurable spaces . .
1.4 Real-valued measurable functions
1.5 Additional Problems . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Measures
2.1 Measure spaces . . . . . . . . . . . . . . . . . .
2.2 Extensions of measures and the coin-toss space
2.3 The Lebesgue measure . . . . . . . . . . . . . .
2.4 Signed measures . . . . . . . . . . . . . . . . .
2.5 Additional Problems . . . . . . . . . . . . . . .
Lebesgue Integration
3.1 The construction of the integral
3.2 First properties of the integral .
3.3 Null sets . . . . . . . . . . . . .
3.4 Additional Problems . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5
5
8
10
13
17
.
.
.
.
.
19
19
24
28
30
33
.
.
.
.
36
36
39
44
46
Lebesgue Spaces and Inequalities

50
4.1 Lebesgue spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2 Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3 Additional problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Theorems of Fubini-Tonelli and Radon-Nikodym

60
5.1 Products of measure spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.2 The Radon-Nikodym Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.3 Additional Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Basic Notions of Probability

6.1 Probability spaces . . . . . . . . . . . . . . . . . . . . . . .
6.2 Distributions of random variables, vectors and elements
6.3 Independence . . . . . . . . . . . . . . . . . . . . . . . . .
6.4 Sums of independent random variables and convolution
6.5 Do independent random variables exist? . . . . . . . . . .
6.6 Additional Problems . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
72
72
74
77
82
84
86
Weak Convergence and Characteristic Functions

7.1 Weak convergence . . . . . . . . . . . . . . .
7.2 Characteristic functions . . . . . . . . . . . .
7.3 Tail behavior . . . . . . . . . . . . . . . . . . .
7.4 The continuity theorem . . . . . . . . . . . . .
7.5 Additional Problems . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
89
89
96
100
102
103
Classical Limit Theorems

8.1 The weak law of large numbers
8.2 An iid-central limit theorem .
8.3 The Lindeberg-Feller Theorem
8.4 Additional Problems . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
106
106
108
110
114
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Conditional Expectation
9.1 The definition and existence of conditional expectation
9.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . .
9.3 Regular conditional distributions . . . . . . . . . . . . .
9.4 Additional Problems . . . . . . . . . . . . . . . . . . . .
10 Discrete Martingales
10.1 Discrete-time filtrations and stochastic processes
10.2 Martingales . . . . . . . . . . . . . . . . . . . . .
10.3 Predictability and martingale transforms . . . .
10.4 Stopping times . . . . . . . . . . . . . . . . . . . .
10.5 Convergence of martingales . . . . . . . . . . . .
10.6 Additional problems . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
11 Uniform Integrability
11.1 Uniform integrability . . . . . . . . . . . . . . . . . .
11.2 First properties of uniformly-integrable martingales
11.3 Backward martingales . . . . . . . . . . . . . . . . .
11.4 Applications of backward martingales . . . . . . . .
11.5 Exchangeability and de Finettis theorem (*) . . . . .
11.6 Additional Problems . . . . . . . . . . . . . . . . . .
Index
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
115
115
118
121
128
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
130
130
131
133
135
137
139
.
.
.
.
.
.
142
142
145
147
149
150
156
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
158
Preface
These notes were written (and are still being heavily edited) to help students with the graduate
courses Theory of Probability I and II offered by the Department of Mathematics, University of
Texas at Austin.
Statements, proofs, or entire sections marked by an asterisk () are not a part of the syllabus
and can be skipped when preparing for midterm, final and prelim exams. .
G ORDAN ITKOVI C
Austin, TX
December 2010.
Part I
Theory of Probability I
Chapter
Measurable spaces
Before we delve into measure theory, let us fix some notation and terminology.
denotes a subset (not necessarily proper).
A set A is said to be countable if there exists an injection (one-to-one mapping) from A
into N. Note that finite sets are also countable. Sets which are not countable are called
uncountable.
For two functions f : B C, g : A B, the composition f g : A C of f and g is given
by (f g)(x) = f (g(x)), for all x A.
{An }nN denotes a sequence. More generally, (A ) denotes a collection indexed by the
set .
1.1
Families of Sets
Definition 1.1 (Order properties) A (countable) family {An }nN of subsets of a non-empty set S is
said to be
1. increasing if An An+1 for all n N,
2. decreasing if An An+1 for all n N,
3. pairwise disjoint if An Am = for m 6= n,
4. a partition of S if {An }nN is pairwise disjoint and n An = S.
We use the notation An A to denote that the sequence {An }nN is increasing and A = n An .
Similarly, An A means that {An }nN is decreasing and A = n An .
CHAPTER 1. MEASURABLE SPACES

Here is a list of some properties that a family S of subsets of a nonempty set S can have:
(A1) S,
(A2) S S,
(A3) A S Ac S,
(A4) A, B S A B S,
(A5) A, B S, A B B \ A S,
(A6) A, B S A B S,
(A7) An S for all n N n An S,
(A8) An S, for all n N and An A implies A S,
(A9) An S, for all n N and {An }nN is pairwise disjoint implies n An S,
Definition 1.2 (Families of sets) A family S of subsets of a non-empty set S is called an
1. algebra if it satisfies (A1),(A3) and (A4),
2. -algebra if it satisfies (A1), (A3) and (A7)
3. -system if it satisfies (A6),
4. -system if it satisfies (A2), (A5) and (A8).
Problem 1.3 Show that:

1. Every -algebra is an algebra.
2. Each algebra is a -system and each -algebra is an algebra and a -system.
3. A family S is a -algebra if and only if it satisfies (A1), (A3), (A6) and (A9).
4. A -system which is a -system is also a -algebra.
5. There are -systems which are not algebras.
6. There are algebras which are not -algebras (Hint: Pick all finite subsets of an infinite set.
That is not an algebra yet, but sets can be added to it so as to become an algebra which is not
a -algebra.)
7. There are -systems which are not -systems.
Definition 1.4 (Generated -algebras) For a family A of subsets of a non-empty set S, the intersection of all -algebras on S that contain A is denoted by (A) and is called the -algebra generated
by A.

Remark 1.5 Since the family 2S of all subsets of S is a -algebra, the concept of a generated algebra is well defined: there is always at least one -algebra containing A - namely 2S . (A)
is itself a -algebra (why?) and it is the smallest (in the sense of set inclusion) -algebra that
contains A. In the same vein, one can define the algebra, the -system and the -system generated
by A. The only important property is that intersections of -algebras, -systems and -systems
are themselves -algebras, -systems and -systems.
Problem 1.6 Show, by means of an example, that the union of a family of algebras (on the same S)
does not need to be an algebra. Repeat for -algebras, -systems and -systems.
Definition 1.7 (Topology) A topology on a set S is a family of subsets of S which contains

and S and is closed under finite intersections and arbitrary (countable or uncountable!) unions. The
elements of are often called the open sets. A set S on which a topology is chosen (i.e., a pair (S, ) of
a set and a topology on it) is called a topological space.
Remark 1.8 Almost all topologies in these notes will be generated by a metric, i.e., a set A S
will be open if and only if for each x A there exists > 0 such that {y S : d(x, y) < } A.
The prime example is R where a set is declared open if it can be represented as a union of open
intervals.
Definition 1.9 (Borel -algebras) If (S, ) is a topological space, then the -algebra ( ), generated
by all open sets, is called the Borel -algebra on (S, ).
Remark 1.10 We often abuse terminology and call S itself a topological space, if the topology
on it is clear from the context. In the same vein, we often speak of the Borel -algebra on a set S.
Example 1.11 Some important -algebras. Let S be a non-empty set:
1. The set S = 2S (also denoted by P(S)) consisting of all subsets of S is a -algebra.
2. At the other extreme, the family S = {, S} is the smallest -algebra on S. It is called the
trivial -algebra on S.
3. The set S of all subsets of S which are either countable or whose complements are countable
is a -algebra. It is called the countable-cocountable -algebra and is the smallest -algebra
on S which contains all singletons, i.e., for which {x} S for all x S.
4. The Borel -algebra on R (generated by all open sets as defined by the Euclidean metric on
R), is denoted by B(R).
Problem 1.12 Show that the B(R) = (A), for any of the following choices of the family A:
1. A = {all open subsets of R },
2. A = {all closed subsets of R },
7

3. A = {all open intervals in R},
4. A = {all closed intervals in R},
5. A = {all left-closed right-open intervals in R},
6. A = {all left-open right-closed intervals in R}, and
7. A = {all open intervals in R with rational end-points}
8. A = {all intervals of the form (, r], where r is rational}.
(Hint: An arbitrary open interval I = (a, b) in R can be written as I = nN [a + n1 , b n1 ]. )
1.2
Measurable mappings
Definition 1.13 (Measurable spaces) A pair (S, S) consisting of a non-empty set S and a -algebra
S of its subsets is called a measurable space.
If (S, S) is a measurable space, and A S, we often say that A is measurable in S.
Definition 1.14 (Pull-backs and push-forwards) For a function f : S T and subsets A S,
B T , we define the
1. push-forward f (A) of A S as
f (A) = {f (x) : x A} T,
2. pull-back f 1 (B) of B T as
f 1 (B) = {x S : f (x) B} S.
It is often the case that the notation is abused and the pull-back of B under f is denoted simply
by {f B}. This notation presupposes, however, that the domain of f is clear from the context.
Problem 1.15 Show that the pull-back operation preserves the elementary set operations, i.e., for
f : S T , and B, {Bn }nN T ,
1. f 1 (T ) = S, f 1 () = ,
2. f 1 (n Bn ) = n f 1 (Bn ),
3. f 1 (n Bn ) = n f 1 (Bn ), and
4. f 1 (B c ) = [f 1 (B)]c .

Give examples showing that the push-forward analogues of the statements (1), (3) and (4) above
are not true.
(Note: The assumption that the families in (2) and (3) above are countable is not necessary.
Uncountable unions or intersections commute with the pull-back, too.)
Definition 1.16 (Measurability) A mapping f : S T , where (S, S) and (T, T ) are measurable
spaces, is said to be (S, T )-measurable if f 1 (B) S for each B T .
Remark 1.17 When T = R, we tacitly assume that the Borel -algebra is defined on T , and we
simply call f measurable. In particular, a function f : R R, which is measurable with respect to
the pair of the Borel -algebras is often called a Borel function.
Proposition 1.18 (A measurability criterion) Let (S, S) and (T, T ) be two measurable spaces, and
let C be a subset of T such that T = (C). If f : S T is a mapping with the property that
f 1 (C) S, for any C C, then f is (S, T )-measurable.
P ROOF Let D be the family of subsets of T defined by
D = {B T : f 1 (B) S}.
By the assumptions of the proposition, we have C D. On the other hand, by Problem 1.15, the
family D has the structure of the -algebra, i.e., D is a -algebra that contains C. Remembering
that T = (C) is the smallest -algebra that contains C, we conclude that T D. Consequently,
f 1 (B) S for all B T .
Problem 1.19 Let (S, S) and (T, T ) be measurable spaces.
1. Suppose that S and T are topological spaces, and that S and T are the corresponding Borel
-algebras. Show that each continuous function f : S T is (S, T )-measurable. (Hint:
Remember that the function f is continuous if the pull-backs of open sets are open.)
2. Let f : S R be a function. Show that f is measurable if and only if
{x S : f (x) q} S, for all rational q.
3. Find an example of (S, S), (T, T ) and a measurable function f : S T such that f (A) =
{f (x) : x A} 6 T for all nonempty A S.
Proposition 1.20 (Compositions of measurable maps) Let (S, S), (T, T ) and (U, U ) be measurable spaces, and let f : S T and g : T U be measurable functions. Then the composition
h = g f : S U , given by h(x) = g(f (x)) is (S, U )-measurable.

P ROOF It is enough to observe that h1 (B) = f 1 (g 1 (B)), for any B U .
Corollary 1.21 (Compositions with a continuous maps) Let (S, S) be a measurable space, T be a
topological space and T the Borel -algebra on T . Let g : T R be a continuous function. Then the
map g f : S R is measurable for each measurable function f : S T .
Definition 1.22 (Generation by a function) Let f : S T be a map from the set S into a measurable space (T, T ). The -algebra generated by f , denoted by (f ), is the intersection of all
-algebras S on S which make f (S, T )-measurable.
The letter will typically be used to denote an abstract index set - we only assume that it is
nonempty, but make no other assumptions about its cardinality.
Definition 1.23 (Generation by several functions) Let (f ) be a family of maps

froma set S
into a measurable space (T, T ). The -algebra generated by (f ) , denoted by (f ) , is the
intersection of all -algebras on S which make each f , , measurable.
Problem 1.24 In the setting of Definitions 1.22 and 1.23, show that
1. for f : S T , we have
(1.1)
(f ) = {f 1 (B) : B T }.
2. for a familty f : S T , , we have

[

1
f
(T
)
,
=
(f
)
(1.2)
where f1 (T ) = {f1 (B) : B T }.

(Note: Note how the right-hand sides differ in (1.1) and (1.2).)
1.3
Products of measurable spaces
Definition 1.25 (Products, choice functions) Let

Q (S ) be a family of sets, parametrized by some
(possibly uncountable) index set . The product S is the set of all functions s : S
(called choice functions) with the property that s() S .
10

Remark 1.26
1. When is finite, each function s : S can be identified with an ordered tuple
(s(1 ), . . . , s(n )), where n is the cardinality (number of elements) of , and 1 , . . . , n is
some ordering of its elements. With this identification, it is clear that our definition of a
product coincides with the well-known definition in the finite case.
2. The celebrated Axiom of Choice in set theory postulates that no matter what the family (S )
is, there exists at least one choice function. In other words, axiom of choice simply asserts
that products of sets are non-empty.
Definition 1.27 (Natural projections) For 0 , the function 0 :

0 (s) = s(0 ), for s
S ,
S0 defined by
is called the (natural) projection onto the coordinate 0 .
Definition 1.28 (Products of measurable spaces) Let {(S

Q , S )} be a family of measurable
spaces. The product (S , S ) is a measurable space ( S , S ), where S is the
smallest -algebra that makes all natural projections ( ) measurable.
Example 1.29 When is finite, the above definition can be made more intuitive. Suppose, just for
simplicity, that = {1, 2}, so that (S1 , S1 )(S2 , S2 ) is a measurable space of the form (S1 S2 , S1
S2 ), where S1 S2 is the smallest -algebra on S1 S2 which makes both 1 and 2 measurable.
The pull-backs under 1 of sets in S1 are given by
1 (B1 ) = {(x, y) S1 S2 : x B1 } = B1 S2 , for B1 S1 .
Similarly
Therefore, by Problem 1.24,
Equivalently (why?)
1 (B2 ) = S1 B2 , for B2 S2 .

S1 S2 = {B1 S2 , S1 B2 : B1 S1 , B2 S2 } .

S1 S2 = {B1 B2 : B1 S1 , B2 S2 } .
In a completely analogous fashion, we can show that, for finitely many measurable spaces
(S1 , S1 ), . . . , (Sn , Sn ), we have
n
O
i=1

Si = {B1 B2 Bn : B1 S1 , B2 S2 , . . . , Bn Sn }
The same goes for countable products. Uncountable products, however, behave very differently.
11

Problem 1.30 We know that the Borel -algebra (based on the usual Euclidean topology) can be
constructed on each Rn . A -algebra on Rn (for n > 1), can also be constructed as a product algebra ni=1 B(R). A third possibility is to consider the mixed case where 1 < m < n is picked
and the -algebra B(Rm ) B(Rnm ) is constructed on Rn (which is now interpreted as a product
of Rm and Rnm ). Show that we get the same -algebra in all three cases.
Q
Problem 1.31 Let (P, P), {(S , S )} be measurable spaces and set S = S , S = S .
Prove that a map f : P S is (P, S)-measurable if and only if the composition f : P S
is (P, S ) measurable for each . (Note: Loosely speaking, this result states that a vectorvalued mapping is measurable if and only if all of its components are measurable.)
Definition
1.32 (Cylinder sets) Let {(S , S )}Q be a family of measurable spaces, and let
Q
( S , S ) be its product. A subset C S is called a cylinder set if there exist
a finite subset {1 , . . . , n } of , as well as a measurable set B S1 S2 Sn such that
Y
C = {s
S : (s(1 ), . . . , s(n )) B}.
A cylinder set for which the set B can be chosen of the form B = B1 Bn , for some B1
S1 , . . . , Bn Sn is called a product cylinder set. In that case
Y
S : (s(1 ) B1 , s(2 ) B2 , . . . , s(n ) Bn }.
C = {s
Problem 1.33
1. Show that the family of product cylinder sets generates the product -algebra.
2. Show that (not-necessarily-product) cylinders are measurable in the product -algebra.
3. Which of the 4 families of sets from Definition 1.2 does the collection of all product cylinders
belong to in general? How about (not-necessarily-product) cylinders?
Example 1.34 The following example will play a major role in probability theory. Hence the name
coin-toss space. Here =
i , Si ) is the discrete two-element space Si = {1, 1},
Q N and for i N, (S
N
S
1
Si = 2 . The product iN Si = {1, 1} can be identified with the set of all sequences s =
(s1 , s2 , . . . ), where si {1, 1}, i N. For each cylinder set C, there exists (why?) n N and a
subset B of {1, 1}n such that
C = {s = (s1 , . . . , sn , sn+1 , . . . ) {1, 1}N : (s1 , . . . , sn ) B}.
The product cylinders are even simpler - they are always of the form C = {1, 1}N or C =
Cn1 ,...,nk ;b1 ,...,bk , where
n
o
Cn1 ,...,nk ;b1 ,...,bk = s = (s1 , s2 , . . . ) {1, 1}N : sn1 = b1 , . . . , snk = bk ,
(1.3)
for some k N, 1 n1 < n2 < < nk N and b1 , b2 , . . . , bk {1, 1}.
12
We know that the -algebra S = iN Si is generated by all projections i : {1, 1}N {1, 1},
i N, where i (s) = si . Equivalently, by Problem 1.33, S is generated by the collection of all
cylinder sets.
Problem 1.35 One can obtain the product -algebra S on {1, 1}N as the Borel -algebra corresponding to a particular topology which makes {1, 1}N compact. Here is how. Start by defining
a mapping d : {1, 1}N {1, 1}N [0, ) by
(1.4)
d(s1 , s2 ) = 2i(s
1 ,s2 )
, where i(s1 , s2 ) = inf{i N : s1i 6= s2i },
for sj = (sj1 , sj2 , . . . ), j = 1, 2.

1. Show that d is a metric on {1, 1}N .
2. Show that {1, 1}N is compact under d. (Hint: Use the diagonal argument.)
3. Show that each cylinder of {1, 1}N is both open and closed under d.
4. Show that each open ball is a cylinder.
5. Show that {1, 1}N is separable, i.e., it admits a countable dense subset.
6. Conclude that S coincides with the Borel -algebra on {1, 1}N under the metric d.
1.4
Real-valued measurable functions
Let L0 (S, S; R) (or, simply, L0 (S; R) or L0 (R) or L0 when the domain (S, S) or the co-domain R are
clear from the context) be the set of all S-measurable functions f : S R. The set of non-negative
measurable functions is denoted by L0+ or L0 ([0, )).
Proposition 1.36 (Measurable functions form a vector space) L0 is a vector space, i.e.
f + g L0 , whenever , R, f, g L0 .
P ROOF Let us define a mapping F : S R2 by F (x) = (f (x), g(x)). By Problem 1.30, the Borel
-algebra on R2 is the same as the product -algebra when we interpret R2 as a product of two
copies of R. Therefore, since its compositions with the coordinate projections are precisely the
functions f and g, Problem 1.31 implies that F is (S, B(R2 ))-measurable.
Consider the function : R2 R given by (x, y) = x + y. It is linear, and, therefore,
continuous. By Corollary 1.21, the composition F : S R is (S, B(R))-measurable, and it only
remains to note that
( F )(x) = (F (x)) = f (x) + g(x), i.e., F = f + g.
In a similar manner (the functions (x, y) 7 max(x, y) and (x, y) xy are continuous from R2
to R - why?) one can prove the following proposition.
13
Proposition 1.37 (Products and maxima preserve measurability) Let f, g be in L0 . Then

1. f g L0 ,
2. max(f, g) and min(f, g) L0 ,
Even though the map x 7 1/x is not defined on the whole R, the following problem is not too
hard:
Problem 1.38 Suppose that f L0 has the property that f (x) 6= 0 for all x S. Then the function
1/f is also in L0 .
For A S, the indicator function 1A is defined by
(
1, x A
1A (x) =
0, x
6 A.
Despite their simplicity, indicators will be extremely useful throughout these notes.
Problem 1.39 Show that for A S, we have A S if and only if 1A L0 .
Remark 1.40 Since it contains the products of pairs of its elements, the set L0 has the structure
of an algebra (not to be confused with the algebra of sets defined above). It is true, however, that
any algebra A of subsets of a non-empty set S, together with the operations of union, intersection
and complement forms a Boolean algebra. Alternatively, it can be given the (algebraic) structure
of a commutative ring with a unit. Indeed, under the operation of symmetric difference, A is an
Abelian group (prove that!). If, in addition, the operation of intersection is introduced in lieu of
multiplication, the resulting structure is, indeed, the one of a commutative ring.
Additionally, a natural partial order given by f g if f (x) g(x), for all x S, can be
introduced on L0 . This order is compatible with the operations of addition and multiplication
and has the additional property that each pair {f, g} L0 admits a least upper bound, i.e., the
element h L0 such that f h, g h and h k, for any other k with the property that f, g k.
Indeed, we simply take h(x) = max(f (x), g(x)). A similar statement can be made for a greatest
lower bound. A vector space with a partial order which satisfies the above properties is called a
vector lattice.
Since a limit of a sequence of real numbers does not necessarily belong to R, it is often nec =
essary to consider functions which are allowed to take the values and . The set R
R {, } is called the extended set of real numbers. Most (but not all) of the algebraic and
In some cases there is no unique way to do that, so
topological structure from R can be lifted to R.
we choose one of them as a matter of convention.
all the arithmetic operations are defined in the usual
1. Arithmetic operations. For x, y R,
way when x, y R. When one or both are in {, }, we use the following convention,
where {+, , , /}:
We define x y = z if all pairs of sequences {xn }nN , {yn }nN in R such that x = limn xn ,
y = limn yn and xn yn is well-defined for all n N, we have
z = lim(xn yn ).
n
14

Otherwise, x y is not defined. This basically means that all intuitively obvious conventions
(such as + = and a0 = for a > 0 hold). In measure theory, however, we do make
one important exception to the above rule. We set
0 = 0 = 0 () = () 0 = 0.
admits a supremum
2. Order. < x < , for all x R. Also, each non-empty subset of R
and an infimum in R in an obvious way.

but a
3. Convergence. It is impossible to extend the usual (Euclidean) metric from R to R,
metric d : R R [0, 1) given by

d (x, y) = |arctan(y) arctan(x)| ,
if we set arctan() = /2 and arctan() = /2. We
extends readily to a metric on R
define convergence (and topology) on R using d . For example, a sequence {xn }nN in R
converges to + if
a) It contains only a finite number of terms equal to ,
b) Every subsequence of {xn }nN whose elements are in R converges to + (in the usual
sense).
for a sequence {xn }nN in the
We define the notions of limit superior and limit inferior on R
following manner:
lim sup xn = inf Sn , where Sn = sup xk ,
n
kn
and
lim inf xn = sup In , where In = inf xk .
n
kn
If you have forgotten how to manipulate limits inferior and superior, here is an exercise to remind
you:
Prove the following statements:
Problem 1.41 Let {xn }nN be a sequence in R.
satisfies a lim supn xn if and only if for any (0, ) there exists n N such that
1. a R
xn a + for n n .
2. lim inf n xn lim supn xn .
3. Define
subsequence of {xn }nN }.

A = {lim xnk : xnk is a convergent (in R)
k
Show that
{lim inf xn , lim sup xn } A [lim inf xn , lim sup xn ].
n
we immediately have the -algebra B(R)

of Borel sets there
Having introduced a topology on R
and the notion of measurability for functions mapping a measurable space (S, S) into R.
is in B(R)
if and only if A \ {, } is Borel in R. Show
Problem 1.42 Show that a subset A R
if and only if the sets f 1 ({}),

that a function f : S R is measurable in the pair (S, B(R))
1
1
iff
f ({}) and f (A) are in S for all A B(R) (equivalently and more succinctly, f L0 (R)
0
{f = }, {f = } S and f 1{f R} L ).
15

is denoted by L0 (S, S; R),
and, as always we leave
The set of all measurable functions f : S R
out S and S when no confusion can arise. The set of extended non-negative measurable functions
Unlike L0 (R), L0 (R)
is not a vector
often plays a role, so we denote it by L0 ([0, ]) or L0+ (R).
space, but it retains all the order structure. Moreover, it is particularly useful because, unlike
L0 (R), it is closed with respect to the limiting operations. More precisely, for a sequence {fn }nN
we define the functions lim supn fn : S [, ] and lim inf n fn : S [, ] by
in L0 (R),
!
(lim sup fn )(x) = lim sup fn (x) = inf
n
and
(lim inf fn )(x) = lim inf fn (x) = sup
n
sup fk (x) ,
kn

inf fk (x) .
kn
Then, we have the following result, where the supremum and infimum of a sequence of functions
are defined pointwise (just like the limits superior and inferior).
Proposition 1.43 (Limiting operations preserve measurability) Let {fn }nN be a sequence in
Then
L0 (R).
1. supn fn , inf n fn L0 (R),
2. lim supn fn , lim inf n fn L0 (R),

for each x S, then f L0 (R),
and
3. if f (x) = limn fn (x) exists in R
} is in S.
4. the set A = {limn fn exists in R
P ROOF
1. We show only the statement for the supremum. It is clear that it is enough to show that the
set {supn fn a} is in S for all a (, ] (why?). This follows, however, directly from
the simple identity
{sup fn a} = n {fn a},
n
and the fact that -algebras are closed with respect to countable intersections.
for each n N.
2. Define gn = supkn fk and use part 1. above to conclude that gn L0 (R)
0
The statement about
Another appeal to part 1. yields that lim supn fn = inf n gn is in L (R).
the limit inferior follows in the same manner.
3. If the limit f (x) = limn fn (x) exists for all x S, then f = lim inf n fn which is measurable
by part 2. above.
4. The statement follows from the fact that A = f 1 ({0}), where

f (x) = arctan lim sup fn (s) arctan lim inf fn (x) .
n
(Note: The unexpected use of the function arctan is really noting to be puzzled by. The only
property needed is its measurability (it is continuous) and monotonicity+bijectivity from
16

[, ] to [/2, /2]. We compose the limits superior and inferior with it so that we dont
run into problems while trying to subtract + from itself.)
1.5
Additional Problems
Problem 1.44 Which of the following are -algebras on R?

1. S = {A R : 0 A}.
2. S = {A R : A is finite}.
3. S = {A R : A is finite, or Ac is finite}.
4. S = {A R : A is countable or Ac is countable}.
5. S = {A R : A is open}.
6. S = {A R : A is open or A is closed}.
Problem 1.45 A partition S is a family P of non-empty subsets of S with the property that each
S belongs to exactly one A P.
1. Show that the number of different algebras on a finite set S is equal to the number of different
partitions of S. (Note: This number for Sn = {1, 2, . . . , n} is called the nth Bell number Bn ,
and no nice closed-form expression for it is known. See below, though.)
2. How many algebras are there on the set S = {1, 2, 3}?
3. Does there exist an algebra with 754 elements?
4. For N N, let an be the number of different algebras on the set {1, 2, . . . , n}. Show that
a1 = 1, a2 = 2, a3 = 5, and that the following recursion holds (where a0 = 1 by definition),
an+1 =
n
X
n
k=0
ak .
5. Show that the exponential generating function for the sequence {an }nN is f (x) = ee
that
X

xn
x
dn ex 1
e
an
= ee 1 or, equivalently, an = dx
.
n
n!
x=0
x 1
, i.e.,
n=0
Problem 1.46 Let (S, S) be a measurable space. For f, g L0 show that the sets {f = g} = {x
S : f (x) = g(x)}, {f < g} = {x S : f (x) < g(x)} are in S.
Problem 1.47 Show that all
1. monotone,
2. convex
functions f : R R are measurable.
17

Problem 1.48 Let (S, S) be a measurable space and let f : S R be a Borel-measurable function.
Show that the graph
Gf = {(x, y) S R : f (x) = y},
of f is a measurable subset in the product space (S R, S B(R)).
18
Chapter
Measures
2.1
Measure spaces
Definition 2.1 (Measure) Let (S, S) be a measurable space. A mapping : S [0, ] is called a
(positive) measure if
1. () = 0, and
P
2. (n An ) = nN (An ), for all pairwise disjoint sequences {An }nN in S.
A triple (S, S, ) consisting of a non-empty set, a -algebra S on it and a measure on S is called a

measure space.
Remark 2.2
1. A mapping whose domain is some nonempty set A of subsets of some set S is sometimes
called a set function.
2. If the requirement 2. in the definition of the measure is weakened so that it is only required
that (A1 An ) = (A1 ) + + (An ), for n N, and pairwise disjoint A1 , . . . , An ,
we say that the mapping is a finitely-additive measure. If we want to stress that a mapping satisfies the original requirement 2. for sequences of sets, we say that is -additive
(countably additive).
Definition 2.3 (Terminology) A measure on the measurable space (S, S) is called

1. a probability measure, if (S) = 1,
2. a finite measure, if (S) < ,
3. a -finite measure, if there exists a sequence {An }nN in S such that n An = S and (An ) <
,
19
CHAPTER 2. MEASURES
4. measure or atom-free, if ({x}) = 0, whenever x S and {x} S.

A set N S is said to be null if (N ) = 0.
Example 2.4 (Examples of measures) Let S be a non-empty set, and let S be a -algebra on S.
1. Measures on countable sets. Suppose that S is a finite or countable set. Then each measure
on S = 2S is of the form
X
p(x),
(A) =
xA
for some function p : S [0, ] (why?). In particular, for a finite set S with N elements,
if p(x) = 1/N then is a probability measure called the uniform measure on S. It has the
property that (A) = #A
#S , where # denotes the cardinality (number of elements).
2. Dirac measure. For x S, we define the set function x on S by

(
1, x A,
x (A) =
0, x 6 A.
It is easy to check that x is indeed a measure on S. Alternatively, x is called the point
mass at x (or an atom on x, or the Dirac function, even though it is not really a function).
Moreover, x is a probability measure and, therefore, a finite and a -finite measure. It is
atom free only if {x} 6 S.
3. Counting Measure. Define a set function : S [0, ] by
(
#A, A is finite,
for A S,
(A) =
,
A is infinite,
where, as above, #A denotes the number of elements in the set A. Again, it is not hard to
check that is a measure - it is called the counting measure. Clearly, is a finite measure if
and only is S is a finite set. could be -finite, though, even without S being finite. Simply
take S = N, S = 2N . In that case (S) = , but for An = {n}, n N, we have (An ) = 1, and
S = n An . Finally, is never atom-free and it is a probability measure only if card S = 1.
Example 2.5 (A finitely-additive set function which is not a measure) Let S = N, and S = 2S .
For A S define (A) = 0 if A is finite and (A) = , otherwise. For A1 , . . . , An S, we have
the following two possibilities:
1. P
Ai is finite, for each i = 1, . . . , n. Then ni=1 Ai is also finite and so 0 = (ni=1 Ai ) =
n
i=1 (Ai ).
P
2. at least one Ai is infinite. Then ni=1 Ai is also infinite and so = (ni=1 Ai ) = ni=1 (Ai ),
because (Ai ) = .
Therefore, is finitely additive.
P On the other hand, take Ai = {i}, for i N. Then (Ai ) = 0, for each i N, and, so,
iN (Ai ) = 0, but (i Ai ) = (N) = .
20
CHAPTER 2. MEASURES
(Note: It is possible to construct very simple-looking finite-additive measures which are not
-additive. For example, there exist {0, 1}-valued finitely-additive measures on all subsets of N,
which are not -additive. Such objects are called ultrafilters and their existence is equivalent to a
certain version of the Axiom of Choice.)
Proposition 2.6 (First properties of measures) Let (S, S, ) be a measure space.

1. For A1 , . . . , An S with Ai Aj = , for i 6= j, we have
n
X
(Ai ) = (ni=1 Ai ).
i=1
(Finite additivity)
2. If A, B S, A B, then
(A) (B).
(Monotonicity of measures)
3. If {An }nN in S is increasing, then
(n An ) = lim (An ) = sup (An ).
n
(Continuity with respect to increasing sequences)

4. If {An }nN in S is decreasing and (A1 ) < , then
(n An ) = lim (An ) = inf (An ).
n
(Continuity with respect to decreasing sequences)

5. For a sequence {An }nN in S, we have
(n An )
(An ).
nN
(Subadditivity w.r.t. general sequences)
P ROOF
1. Note that the sequence A1 , A2 , . . . , An , , , . . . is pairwise disjoint, and so, by -additivity,
(ni=1 Ai ) = (iN Ai ) =
(Ai ) =
n
X
i=1
iN
21
(Ai ) +
i=n+1
() =
n
X
i=1
(Ai ).
CHAPTER 2. MEASURES
2. Write B as a disjoint union A (B \ A) of elements of S. By (1) above,
(B) = (A) + (B \ A) (A).
3. Define B1 = A1 , Bn = An \ An1 for n > 1. Then {Bn }nN is a pairwise disjoint sequence in
S with nk=1 Bk = An for each n N (why?). By -additivity we have
(n An ) = (n Bn ) =
(Bn ) = lim
n
nN
n
X
(Bk ) = lim (nk=1 Bk ) = lim (An ).

n
k=1
4. Consider the increasing sequence {Bn }nN in S given by Bn = A1 \ An . By De Morgan laws,

finiteness of (A1 ) and (3) above, we have
(A1 ) (n An ) = (A1 \ (n An )) = (n Bn ) = lim (Bn ) = lim (A1 \ An )
n
= (A1 ) lim (An ).

n
Subtracting both sides from (A1 ) < produces the statement.

5. We start from the observation that for A1 , A1 S the set A1 A2 can be written as a disjoint
union
A1 A2 = (A1 \ A2 ) (A2 \ A1 ) (A1 A2 ),
so that
On the other hand,
(A1 A2 ) = (A1 \ A2 ) + (A2 \ A1 ) + (A1 A2 ).

(A1 ) + (A2 ) = ((A1 \ A2 ) + (A1 A2 )) + (A2 \ A1 ) + (A1 A2 )
= (A1 \ A2 ) + (A2 \ A1 ) + 2(A1 A2 ),
and so
(A1 ) + (A2 ) (A1 A2 ) = (A1 A2 ) 0.
Induction can be used to show that
(A1 An )
n
X
(Ak ).
k=1
Since all (An ) are nonnegative, we now have

(A1 An ) , for each n N, where =
(An ).
nN
The sequence {Bn }nN given by Bn = nk=1 Ak is increasing, so the continuity of measure
with respect to increasing sequences implies that
(n An ) = (n Bn ) = lim (Bn ) = lim (A1 An ) .
n
22
CHAPTER 2. MEASURES
Remark 2.7 The condition (A1 ) < in the part (4) of Proposition 2.6 cannot be significantly
relaxed. Indeed, let be the counting measure on N, and let An = {n, n + 1, . . . }. Then (An ) =
and, so limn (An ) = . On the other hand, An = , so (n An ) = 0.
In addition to unions and intersections, one can produce other important new sets from sequences of old ones. More specifically, let {An }nN be a sequence of subsets of S. The subset
lim inf n An of S, defined by
lim inf An = n Bn , where Bn = kn Ak ,
n
is called the limit inferior of the sequence An . It is also denoted by limn An or {An , ev.} (ev. stands
for eventually). The reason for this last notation is the following: lim inf n An is the set of all x S
which belong to An for all but finitely many values of the index n.
Similarly, the subset lim supn An of S, defined by
lim sup An = n Bn , where Bn = kn Ak ,
n
is called the limit superior of the sequence An . It is also denoted by limn An or {An , i.o.} (i.o.
stands for infinitely often). In words, lim supn An is the set of all x S which belong An for infinitely
many values of n. Clearly, we have
lim inf An lim sup An .
n
Problem 2.8 Let (S, S, ) be a finite measure space. Show that

(lim inf An ) lim inf (An ) lim sup (An ) (lim sup An ),
for any sequence {An }nN in S. Give an example of a (single) sequence {An }nN for which all
inequalities above are strict.
(Hint: For the second part, a measure space with finite (and small) S will do.)
Proposition 2.9 (Borel-Cantelli Lemma P

I) Let (S, S, ) be a measure space, and let {An }nN be a
sequence of sets in S with the property that nN (An ) < . Then
(lim sup An ) = 0.
n
P ROOF Set Bn = kn Ak , so that {Bn }nN is a decreasing sequence of sets in S with lim supn An =
n Bn . Using the subadditivity of measures of Proposition 2.6, part 5., we get
(2.1)
(Bn )
(An ).
k=n
Since nN (An ) converges, the right-hand side of (2.1) can be made arbitrarily small by choosing large enough n N. Hence (lim supn An ) = 0.
23
CHAPTER 2. MEASURES
2.2
Extensions of measures and the coin-toss space
Example 1.34 of Chapter 1 has introduced a measurable space ({1, 1}N , S), where S is the product -algebra on {1, 1}N . The purpose of the present section is to turn ({1, 1}N , S) into a measure space, i.e., to define a suitable measure on it. It is easy to construct just any measure on
{1, 1}N , but the one we are after is the one which will justify the name coin-toss space.
The intuition we have about tossing a fair coin infinitely many times should help us start with
the definition of the coin-toss measure - denoted by C - on cylinders. Since the coordinate
spaces {1, 1} are particularly simple, each product cylinder is of the form C = {1, 1}N or C =
Cn1 ,...,nk ;b1 ,...,bk , as given by (1.3), for a choice 1 n1 < n2 < . . . , nk N of coordinates and
the corresponding values b1 , . . . , bk {1, 1}. In the language of elementary probability, each
cylinder corresponds to the event when the outcome of the ni -th coin is bi {1, 1}, for k =
1, . . . , n. The measure (probability) of this event can only be given by
(2.2)
C (Cn1 ,...,nk ;b1 ,...,bk ) =
|2
1
2
21 = 2k .
{z
}
k times
The hard part is to extend this definition to all elements of S, and not only cylinders. For example,
in order to state the law of large numbers later on, we will need to be able to compute the measure
of the set
n
n
o
X
N
1
s {1, 1} : lim n
sk = 21 ,
n
k=1
which is clearly not a cylinder.

Problem 1.33 states, however, that cylinders form an algebra and generate the -algebra S.
Luckily, this puts us close to the conditions of the following important theorem of Caratheodory.
The proof does not use unfamiliar methodology, but we omit it because it is quite long and tricky.
Theorem 2.10 (Caratheodorys Extension Theorem) Let S be a non-empty set, let A be an algebra
of its subsets and let : A [0, ] be a set-function with the following properties:
1. () = 0, and
P
2. (A) =
n=1 (An ), if {An }nN is a pairwise-disjoint family in A and A = n An A.
Then, there exists a measure

on (A) with the property that (A) =
(A) for A A.
Remark 2.11 In words, a -additive measure on an algebra A can be extended to a -additive

measure on the -algebra generated by A. It is clear that the -additivity requirement of Theorem
2.10 is necessary, but it is quite surprising that it is actually sufficient.
In order to apply Theorem 2.10 in our situation, we need to check that is indeed a countablyadditive measure on the algebra A of all cylinders. The following problem will help pinpoint the
hard part of the argument:
Problem 2.12 Let A be an algebra on the non-empty set S, and let : A [0, ] be a finite
((S) < ) and finitely-additive set function on S with the following, additional, property:
(2.3)
lim (An ) = 0, whenever An .

n
24
CHAPTER 2. MEASURES
Then is -additive on A, i.e., it satisfies the conditions of Theorem 2.10.
The part about finite additivity is easy (but messy) and we leave it to the reader:
Problem 2.13 Show that the set-function C , defined by (2.2) on the algebra A of cylinders, is
finitely additive.
Lemma 2.14 (Conditions of Caratheodorys theorem) The the set-function C , defined by (2.2)
on the algebra A of cylinders, has the property (2.3).
P ROOF By Problem 1.35, cylinders are closed sets, and so {An }nN is a sequence of closed sets
whose intersection is empty. The same problem states that {1, 1}N is compact, so, by the finiteintersection property1 , we have An1 . . . Ank = , for some finite collection n1 , . . . , nk of indices.
Since {An }nN is decreasing, we must have An = , for all n nk , and, consequently, limn (An ) =
0.
Proposition 2.15 (Existence of the coin-toss measure) There

({1, 1}N , S) with the property that (2.2) holds for all cylinders.
exists
measure
on
P ROOF Thanks to Lemma 2.14, Theorem 2.10 can now be used.

In order to prove uniqueness, we will need the celebrated - Theorem of Eugene Dynkin:
Theorem 2.16 (Dynkins - Theorem) Let P be a -system on a non-empty set S, and let be
a -system which contains P. Then also contains the -algebra (P) generated by P.
P ROOF Using the result of part 4. of Problem 1.3, we only need to prove that (P) (where (P)
denotes the -system generated by P) is a -system. For A S, let GA denote the family of all
subsets of S whose intersections with A are in (P):
GA = {C S : C A (P)}.
Claim 1: GA is a -system for A (P).
Since A (P), clearly S GA .
For an increasing family {Cn }nN in GA we have (n Cn ) A = n (Cn A). Each Cn A is
in , and the family {Cn A}nN is increasing, so (n Cn ) A .
1
The finite-intersection property refers to the following fact, familiar from real analysis: If a family of closed sets of a
compact topological space has empty intersection, then it admits a finite subfamily with an empty intersection.
25
CHAPTER 2. MEASURES
Finally, for C1 , C2 G with C1 C2 , we have
(C2 \ C1 ) A = (C2 A) \ (C1 A) ,
because C1 A C2 A.
Since P is a -system, for any A P, we have P GA . Therefore, (P) GA , because GA is a
-system. In other words, for A P and B (P), we have A B (P).
That means, however, that P GB , for any B (P). Using the fact that GB is a -system we
must also have (P) GB , for any B (P), i.e., A B (P), for all A, B (P), which shows
that (P) is -system.
Proposition 2.17 (Measures which agree on a -system) Let (S, S) be a measurable space, and
let P be a -system which generates S. Suppose that 1 and 2 are two measures on S with the
property that 1 (S) = 2 (S) < and
1 (A) = 2 (A), for all A P.
Then 1 = 2 , i.e., 1 (A) = 2 (A), for all A S.
P ROOF Let L be the family of all subsets A of S for which 1 (A) = 2 (A). Clearly P L, but
L is, potentially, bigger. In fact, it follows easily from the elementary properties of measures (see
Proposition 2.6) and the fact that 1 (S) = 2 (S) < that it necessarily has the structure of a system. By Theorem 2.16 (the - Theorem), L contains the -algebra generated by P, i.e., S L.
On the other hand, by definition, L S and so 1 = 2 .
Remark 2.18 It seems that the structure of a -system is defined so that it would exactly describe
the structure of the family of all sets on which two measures (with the same total mass) agree. The
structure of the -system corresponds to the minimal assumption that allows Proposition 2.17 to
hold.
Proposition 2.19 (Uniqueness of the coin-toss measure) The measure C is the unique measure
on ({1, 1}N , S) with the property that (2.2) holds for all cylinders.
P ROOF The existence is the content of Proposition 2.15. To prove uniqueness, it suffices to note
that algebras are -systems and use Proposition 2.17.
Problem 2.20 Define D1 , D2 {1, 1}N by
1. D1 = {s {1, 1}N : lim supn sn = 1},
2. D2 = {s {1, 1}N : N N, sN = sN +1 = sN +2 }.
Show that D1 , D2 S and compute (D1 ), (D2 ).
26
CHAPTER 2. MEASURES
Our next task is to probe the structure of the -algebra S on {1, 1}N a little bit more and show
N
that S 6= 2{1,1} . It is interesting that such a result (which deals exclusively with the structure of
S) requires a use of a measure in its proof.
Example 2.21 ((*) A non-measurable subset of {1, 1}N ) Since -algebras are closed under countable set operations, and since the product -algebra S for the coin-toss space {1, 1}N is generated
by sets obtained by restricting finite collections of coordinates, one is tempted to think that S contains all subsets of {1, 1}N . That is not the case. We will use the axiom of choice, together with
the fact that a measure C can be defined on the whole of {1, 1}N , to show to construct an
example of a non-measurable set.
Let us start by constructing a relation on {1, 1}N in the following way: we set s1 s2 if
and only if there exists n N such that s1k = s2k , for k n (here, as always, si = (si1 , si2 , . . . ),
i = 1, 2). In words, s1 and s2 are related if they only differ in a finite number of coordinates. It is
easy to check that is an equivalence relation and that it splits {1, 1}N into disjoint equivalence
classes. One of the many equivalent forms of the axiom of choice states that there exists a subset
N of {1, 1}N which contains exactly one element from each of the equivalence classes.
Let us suppose that N is an element in S and see if we can reach a contradiction. Let F denote
the set of all finite subsets of N. For each nonempty n = {n1 , . . . , nk } F , let us define the
mapping Tn : {1, 1}N {1, 1}N in the following manner:
(
sl ,
l n,
(Tn (s))l =
sl , l 6 n.
In words, Tn flips the signs of the elements of its argument on the positions corresponding to n.
We define T = Id, i.e., T (s) = s.
Since n is finite, Tf preserves the -equivalence class of each element. Consequently (and
using the fact that N contains exactly one element from each equivalence class) the sets N and
Tn (N ) = {Tn (s) : s N } are disjoint. Similarly and more generally, the sets Tn (N ) and Tn (N )
are also disjoint whenever n 6= n . On the other hand, each s {1, 1}N is equivalent to some
N , i.e., it can be obtained from s
by flipping a finite number of coordinates. Therefore, the
s
family
N = {Tn (N ) : n F }
forms a partition of {1, 1}N .

The mapping Tn has several other nice properties. First of all, it is immediate that it is involutory, i.e., Tn Tn = Id. To show that it is (S, S)-measurable, we need to prove that its composition
with each projection map k : S {1, 1} is measurable. This follows immediately from the fact
that for k N
(
Ck;1 ,
k 6 n,
1
(Tn k ) ({1}) =
Ck;1 , k n,
where. for i {1, 1}, Ck;i = {s {1, 1}N : sk = i} - a cylinder. If we combine the involutivity
and measurability of Tn , we immediately conclude that Tn (A) S for each A S. In particular,
N S.
In addition to preserving measurability, the map Tn also preserves the measure2 the in C , i.e.,
C (Tn (A)) = C (A), for all A S. To prove that, let us pick n F and consider the set-function
2
Actually, we say that a map f from a measure space (S, S, S ) to the measure space (T, T , T ) is measure preserving if it is measurable and S (f 1 (A)) = T (A), for all A T . The involutivity of the map Tn implies that this
general definition agrees with our usage in this example.
27
CHAPTER 2. MEASURES
n : S [0, 1] given by
n (A) = C (Tn (A)).
It is a simple matter to show that n is, in fact, a measure on (S, S) with n (S) = 1. Moreover,
thanks to the simple form (2.2) of the action of the measure C on cylinders, it is clear that n = C
on the algebra of all cylinders. It suffices to invoke Proposition 2.17 to conclude that n = C on
the entire S, i.e., that Tn preserves C .
The above properties of the maps Tn , n F can imply the following: N is a partition of S into
countably many measurable subsets of equal measure. Such a partition {N1 , N2 , . . . } cannot exist,
however. Indeed if it did, one of the following two cases would occur:
P
P
1. (N1 ) = 0. In that case (S) = (k Nk ) = n (Nk ) = n 0 = 0 6= 1 = (S).
P
P
6 1 = (S).
2. (N1 ) = > 0. In that case (S) = (k Nk ) = n (Nk ) = n = =
Therefore, the set N cannot be measurable in S.
(Note: Somewhat heavier set-theoretic machinery can be used to prove that most of the subsets
of S are not in S, in the sense that the cardinality of the set S is strictly smaller than the cardinality
of the set 2S of all subsets of S)
2.3
The Lebesgue measure
As we shall see, the coin-toss space can be used as a sort of a universal measure space in probability theory. We use it here to construct the Lebesgue measure on [0, 1]. We start with the notion
somewhat dual to the already introduced notion of the pull-back in Definition 1.14. We leave it
as an exercise for the reader to show that the set function f () from Definition 2.22 is indeed a
measure.
Definition 2.22 (Push-forwards) Let (S, S, ) be a measure space and let (T, T ) be a measurable
space. The measure f () on (T, T ), defined by
f (B) = (f 1 (B)), for B T ,
is called the push-forward of the measure by f .
Let f : S [0, 1] be the mapping given by

f (s) =
X
k=1
1+sk
2
2k , s {1, 1}N .
The idea is to use f to establish a correspondence between all real numbers in [0, 1] and their
expansions in the binary system, with the coding 1 7 0 and 1 7 1. It is interesting to note
that f is not one-to-one3 , as it, for example, maps s1 = (1, 1, 1, . . . ) and s2 = (1, 1, 1, . . . )
into the same value - namely 12 . Let us show, first, that the map f is continuous in the metric d
3
The reason for this is, poetically speaking, that [0, 1] is not the Cantor set.
28
CHAPTER 2. MEASURES
defined by part (1.4) of Problem 1.33. Indeed, we pick s1 and s2 in {1, 1}N and remember that
for d(s1 , s2 ) = 2n , the first n 1 coordinates of s1 and s2 coincide. Therefore,
|f (s1 ) f (s2 )|
2k = 2n+1 = 2d(s1 , s2 ).
k=n
Hence, the map f is Lipschitz and, therefore, continuous.

The continuity of f (together with the fact that S is the Borel -algebra for the topology induced
by the metric d) implies that f : ({1, 1}N , S) ([0, 1], B([0, 1])) is a measurable mapping. Therefore, the push-forward = f () is well defined on ([0, 1], B([0, 1])), and we call it the Lebesgue
measure on [0, 1].
Proposition 2.23 (Intuitive properties of the Lebesgue measure) The Lebesgue measure on
([0, 1], B([0, 1])) satisfies
(2.4)
([a, b)) = b a, ({a}) = 0 for 0 a < b 1.
P ROOF
n
1. Consider a, b of the form b = 2kn and b = k+1
2n , for n N and k < 2 . For such a, b we
have f 1 ([a, b)) = C1,...,n;c1 ,c2 ,...,cn , where c1 c2 . . . cn is the base-2 expansion of k (after the
recoding 1 7 0, 1 7 1). By the very definition of and the form (2.2) of the action of the
coin-toss measure C on cylinders, we have

k
[a, b) = C f 1 [a, b) = C (C1,...,n;c1 ,c2 ,...,cn ) = 2n = k+1
2n 2n .
Therefore, (2.4) holds for a, b of the form b = 2kn and b = 2ln , for n N, k < 2n and l = k + 1.
Using (finite) additivity of , we immediately conclude that (2.4) holds for all k, l, i.e., that
it holds for all dyadic rationals. A general a (0, 1] can be approximated by an increasing
sequence {qn }nN of dyadic rationals from the left, and the continuity of measures with
respect to decreasing sequences implies that

[a, p) = n [qn , p) = lim [qn , p) = lim(p qn ) = (p a),
n
whenever a (0, 1] and p is a dyadic rational. In order to remove the dyadicity requirement
from the right limit, we approximate it from the left by a sequence {pn }nN of dyadic rationals with pn > a, and use the continuity with respect to increasing sequences to get, for
a < b (0, 1),

[a, b) = n [a, pn ) = lim [a, pn ) = lim(pn a) = (b a).
n
The Lebesgue measure has another important property:

Problem 2.24 Show that the Lebesgue measure is translation invariant. More precisely, for B
B([0, 1]) and x [0, 1), we have
29
CHAPTER 2. MEASURES
1. B +1 x = {b + x (mod 1) : b B} is in B([0, 1]) and
2. (B +1 x) = (B),
(
a,
a 1,
. Geometrically, the set x +1 B is
a 1, a > 1
obtained from B by translating it to the right by x and then shifting the part that is sticking out
by 1 to the left.) (Hint: Use Proposition 2.17 for the second part.)
where, for a [0, 2), we define a (mod 1) =
Finally, the notion of the Lebesgue measure is just as useful on the entire R, as on its compact
subset [0, 1]. For a general B B(R), we can define the Lebesgue measure of B by measuring its
intersections with all intervals of the form [n, n + 1), and adding them together, i.e.,
(B) =
n=

B [n, n + 1) n .
Note how we are overloading the notation and using the letter for both the Lebesgue measure
on [0, 1] and the Lebesgue measure on R.
It is a quite tedious, but does not require any new tools, to show that many of the properties
of on [0, 1] transfer to on R:
Problem 2.25 Let be the Lebesgue measure on (R, B(R)). Show that
1. ([a, b)) = b a, ({a}) = 0 for a < b,
2. is -finite but not finite,
3. (B + x) = (B), for all B B(R) and x R, where B + x = {b + x : b B}.
Remark 2.26 The existence of the Lebesgue measure allows to show quickly that the converse of
the implication in the Borel-Cantelli Lemma does not hold without additional conditions, even if
is a probability measure. Indeed, let = be the Lebesgue measure on [0, 1].
Set An = (0, n1 ], for n N so that
lim sup An =
n
\[
Ak =
n kn
\
n
An = ,
which implies that (lim supn An ) = 0. On the other hand

X
X
1
(An ) =
n = .
nN
nN
We will see later that the converse does hold if the family of sets {An }nN satisfy the additional
condition of independence.
2.4
Signed measures
In addition to (positive) measures, it is sometimes useful to know a few things about measure-like
(and not in [0, ]).
set functions which take values in R
30
CHAPTER 2. MEASURES
Definition 2.27 (Signed measures) Let (S, S) be a measurable space. A mapping : S

(, ] is called a signed measure (or real measure) if
1. () = 0, and
2. for any pairwise
P disjoint sequence {An }nN in S the series
(n An ) = n (An ).
n (An )
is summable and
The notion of convergence here is applied to sequences that may take the value , so we need
to be precise about how it is defined. Remember that a+ = max(a, 0) and a = max(a, 0).
Definition 2.28
for sequences) A sequence {an }nN
P(Summability
P in (, ] is said to be
summable if nN aP
that case, the sum of the series nN an is the (well-defined)
n < . InP
extended real number nN a+

n
nN an (, ].
Remark 2.29
1. Simply put, a series with elements in (, ] is summable if the sum of the sub-series of
its negative elements is finite. The problem we are trying to avoid is, of course, the one
involving . It is easy to show that in the case of a real-valued series, the notion of
summability coincides with the notion of absolute convergence.
2. The fact that we are allowing the signed measure to take the value and not the value
is entirely arbitrary. A completely parallel theory can be built with the opposite choice.
What cannot be dealt with in a really meaningful way is the case when both and are
allowed.
Definition 2.30 (Finite measurable partitions) A finite collection {B1 , . . . , Bn } of measurable

subsets of A S is said to be a finite measurable partition of A if
Bi Bj = for i 6= j, and A = i Bi .
The family of all finite measurable partitions of the set A is denoted by P[0,) (A).
Definition 2.31 (Total variation) For A S and a signed measure on S, we define the number
|| (A) [0, ], called the total variation of on A by
|| (A) =
sup
n
X
{D1 ,D2 ,...,Dn }P[0,) (A) k=1
31
|(Dk )| ,
CHAPTER 2. MEASURES
where the supremum is taken over all finite measurable partitions D1 , . . . , Dn , n N of S. The number
|| (S) [0, ] is called the total variation (norm) of .
The central result about signed measures is the following:
Theorem 2.32 (Hahn-Jordan decomposition) Let (S, S) be a measure space, and let be a signed
measure on S. Then there exist two (positive) measures + and such that
1. is finite,
2. (A) = + (A) (A),
3. || (A) = + (A) + (A),
Measures + and with the above properties are unique. Moreover, there exists a set D S such
that + (A) = (A Dc ) and (A) = (A D) for all A S.
P ROOF (*) Call a set B S negative if (C) 0, for all C S, C B. Let P be the collection of
all negative sets - it is nonempty because P. Set
= inf{(B) : B P},
and let {Bn }nN be a sequence of negative sets with (Bn ) . We define D = n Bn and note
that D is a negative set with (D) = (why?). In particular, > .
Our first order of business is to show that Dc is a positive set, i.e. that (E) 0 for all E Dc .
Suppose, to the contrary, that (B) < 0, for some B S, B Dc . The set B cannot be a negative
set - otherwise D B would be a negative set with (D B) = (D) + (B) = + (B) < .
Therefore, there exists a measurable subset E1 of B with (E1 ) > 0, i.e., the set
E1 = {E B : E S, (E) > 0}
is non-empty. Pick k1 N such that
1
k1
sup{(E) : E E1 },
and an almost-maximal set E1 E1 with

1
k1
(E1 ) >
1
k1 +1 .
We set B1 = B \ E1 and observe that, since 0 > (E1 ) > , we have

(B1 ) = (B) (E1 ) < 0,
and so (B1 ) < 0. Replacing B by B1 , the above discussion can be repeated and a constant k2 and
the an almost maximal E2 with k12 (E2 ) > k21+1 can be constructed. Continuing in the same
manner, we obtain the sequence {En }nN of pairwise disjoint subsets of B with k1n (En ) >
1
kn +1 .
32
CHAPTER 2. MEASURES
Given that (B) < 0, it cannot have subsets of measure . Therefore, (n En ) < and
X
X
1
<
(En ) = (n En ) < ,
kn +1
nN
nN
and so kn 0, as n .
Let F S be a subset of B \ n En . Then, it a subset of B \ nk=1 Ek , and, therefore, by construction, (F ) k1n . The fact that kn now implies that (F ) 0, which, in turn, implies that
B\n En is a negative set. The set D is, however, the maximal negative set, and so (B\n En ) = 0.
On the other hand,
X
(B0 \ n En ) = (B0 )
(En ) > (B0 ) > 0,
n
Dc
a contradiction. Therefore,
is a positive set.
Having split S into a disjoint union of a positive and a negative set, we define
+ (A) = (A Dc ) and (A) = (A D),
so that both + and are (positive measures) with finite and = + .

Finally, we need to show that || = + + . Take A S and {B1 , . . . , Bn } P[0,) (A). Then
n
X
k=1
|(B)| =
n
n

X
X
+
(Bk ) (Bk )
+ (Bk ) + (Bk ) = + (A) + (A).
k=1
k=1
To show that the obtained upper bound is tight, we consider the partition {A D, A Dc } of A
for which we have
|(A D)| + |(A Dc )| = (A D) + + (A Dc ) = + (A) + (A).
2.5
Additional Problems
Problem 2.33 (Local separation by constants) Let (S,S, ) be a measure space and let the function f, g L0 (S, S, ) satisfy {x S : f (x) < g(x)} > 0. Prove or construct a counterexample
for the following statement:

There exist constants a, b R such that {x S : f (x) a 0.
Problem 2.34 (A pseudometric on sets) Let (S, S, ) be a finite measure space. For A, B S define
d(A, B) = (A B),
where denotes the symmetric difference: A B = (A \ B) (B \ A). Show that d is a pseudometric4 on S, and for A S describe the set of all B S with d(A, B) = 0.
4
Let X be a nonempty set. A function d : X X [0, ) is called a pseudo metric if
1. d(x, y) + d(y, x) d(x, z), for all x, y, z X,

2. d(x, y) = d(y, x), for all x, y X, and
3. d(x, x) = 0, for all x X.
Note how the only difference between a metric and a pseudometric is that for a metric d(x, y) = 0 implies x = y, while
no such requirement is imposed on a pseudometric.
33
CHAPTER 2. MEASURES
Problem 2.35 (Complete measure spaces) A measure space (S, S, ) is called complete if all subsets of null sets are themselves in S. For a (possibly incomplete) measure space (S, S, ) we define
the completion (S, S , ) in the following way:
S = {A N : A S and N N for some N S with (N ) = 0}.
For B S with representation B = A N we set (B) = (A).
1. Show that S is a -algebra.
2. Show that the definition (B) = (A) above does not depend on the choice of the decom = (A) if B = A N
is another decomposition of B
position B = A N , i.e., that (A)
of a null set in S.
into a set A in S and a subset N
3. Show that is a measure on (S, S ) and that (S, S , ) is a complete measure space with
the property that (A) = (A), for A S.
Problem 2.36 (The Cantor set) The Cantor set is defined as the collection of all real numbers x in
[0, 1] with the representation
x=
n=1
cn 3n , where cn {0, 2}.
Show that it is Borel-measurable and compute its Lebesgue measure.

Problem 2.37 (The uniform measure on a circle) Let S 1 be the unit circle, and let f : [0, 1) S 1
be the winding map

f (x) = cos(2x), sin(2x) , x [0, 1).
1. Show that the map f is (B([0, 1)), S 1 )-measurable, where S 1 denotes the Borel -algebra on
S 1 (with the topology inherited from R2 ).
2. For (0, 2), let R denote the (counter-clockwise) rotation of R2 with center (0, 0) and
angle . Show that R (A) = {R (x) : x A} is in S 1 if and only if A S 1 .
3. Let 1 be the push-forward of the Lebesgue measure by the map f . Show that 1 is
rotation-invariant, i.e., that 1 (A) = 1 R (A) .
(Note: The measure 1 is called the uniform measure (or the uniform distribution on S 1 .)
Problem 2.38 (Asymptotic densities) We say that the subset A of N admits asymptotic density if
the limit
#(A {1, 2, . . . , n})
,
d(A) = lim
n
n
exists (remember that # denotes the number of elements of a set). Let D be the collection of all
subsets of N which admit asymptotic density.
1. Is D an algebra? A -algebra?
2. Is the map A 7 d(A) finitely-additive on D? A measure?
34
CHAPTER 2. MEASURES
Problem 2.39 (A subset of the coin-toss space) An element in {1, 1}N (i.e., a sequence s with
s = (s1 , s2 , . . . ) where sn {1, 1} for all n N) is said to be eventually periodic if there exists
N0 , K N such that sn = sn+K for all n N0 . Let P {1, 1}N be the collection of all eventuallyperiod sequences. Show that P is measurable in the product -algebra S and compute C (P ).
Problem 2.40 (Regular measures) The measure space (S, S, ), where (S, d) is a metric space and
S is a -algebra on S which contains the Borel -algebra B(d) on S is called regular if for each
A S and each > 0 there exist a closed set C and an open set O such that C A O and
(O \ C) < .
1. Suppose that (S, S, ) is a regular measure space, and let (S, B(d), |B(d) ) be the measure
space obtained from (S, S, ) by restricting themeasure onto the -algebra of Borel sets.
Show that S B(d) , where S, B(d) , (|B(d) ) is the completion of (S, B(d), |B(d) ) (in the
sense of Problem 2.35).
2. Suppose that (S, d) is a metric space and that is a finite measure on B(d). Show that
(S, B(d), ) is a regular measure space.
(Hint: Consider a collection A of subsets A of S such that for each > 0 there exists a closed
set C and an open set O with C A O and (O \ C) < . Argue that A is a -algebra.
Then show that each closed set can be written as an intersection of open sets; use (but prove,
first) the fact that the map
x 7 d(x, C) = inf{d(x, y) : y C},
is continuous on S for any nonempty C S. )
3. Show that (S, B(d), ) is regular if is not necessarily finite, but has the property that (A) <
whenever A B(d) is bounded, i.e., when sup{d(x, y) : x, y A} < . (Hint: Pick a
point x0 S and, for n N, define the family {Rn }nN of subsets of S as follows:
R1 = {x S : d(x, x0 ) < 2}, and
Rn = {x S : n 1 < d(x, x0 ) < n + 1}, for n > 1,

as well as a sequence {n }nN of set functions on B(d), given by n (A) = (A Rn ), for
A B(d). Under the right circumstances, even countable unions of closed sets are closed).
(Note: It follows now from
the fact that Lebesgue measure of any ball is finite, that the Lebesgue
measure on Rn , B(Rn ) is regular.)
35
Chapter
Lebesgue Integration
3.1
The construction of the integral
Unless expressly specified otherwise, we pick and fix a measure space (S, S, ) and assume that
all functions under consideration are defined there.
Definition 3.1 (Simple functions) A function f L0 (S, S, ) is said to be simple if it takes only
a finite number of values.
The collection of all simple functions is denoted by LSimp,0 (more precisely by LSimp,0 (S, S, ))
Simp,0
and the family of non-negative simple functions by L+
. Clearly, a simple function f : S R
admits a (not necessarily unique) representation
(3.1)
f=
n
X
k 1Ak ,
k=1
for 1 , . . . , n R and A1 , . . . , An S. Such a representation is called the simple-function representation of f .

When the sets Ak , k = 1, . . . , n are intervals in R, the graph of the simple function f looks like
a collection of steps (of heights 1 , . . . , n ). For that reason, the simple functions are sometimes
referred to as step functions.
The Lebesgue integral is very easy to define for non-negative simple functions and this definition allows for further generalizations. In fact, the progression of events you will see in this section
is typical for measure theory: you start with indicator functions, move on to non-negative simple
functions, then to general non-negative measurable functions, and finally to (not-necessarily-nonnegative) measurable functions. This approach is so common, that it has a name - the Standard
Machine.
36
CHAPTER 3. LEBESGUE INTEGRATION
Definition 3.2 (Lebesgue

integration for simple functions) For f LSimp,0
we define the
+
R
(Lebesgue) integral f d of f with respect to by
Z
where f =
Pn
k=1 k 1Ak
f d =
n
X
k=1
k (Ak ) [0, ],
is a simple-function representation of f ,
Problem 3.3 Show that P

the Lebesgue integral is well-defined for simple functions, i.e., that the
value of the expression nk=1 k (Ak ) does not depend on the choice of the simple-function representation of f .
Remark 3.4
R
1. It is important to note that f d can equal + even if fRnever takes the value +. It is
enough to pick f = 1A where (A) = + - indeed, then f d = 1(A) = , but f only
takes values in the set {0, 1}. This is one of the reasons we start with non-negative functions.
Otherwise, we would need to deal with the (unsolvable) problem of computing . On
the other hand, such examples cannot be constructed when is a finite measure. Indeed, it
R
is easy to show that when (S) < , we have f d < for all f LSimp,0
.
+
2. One can think of the (simple) Lebesgue integral as a generalization of the notion of (finite)
additivity
of measures. Indeed, if the simple-function representation of f is given by f =
Pn
the equality of the values of the integrals
k=1 1Ak , for pairwise disjoint A1 , . . . , An , thenP
n
for two representations f = 1nk=1 Ak and f =
k=1 1Ak is a simple restatement of finite
additivity. When A1 , . . . , An are not disjoint, then the finite additivity gives way to finite
subadditivity
n
X
(nk=1 Ak )
(Ak ),
k=1
but the integral f d takes into account those x which are covered by more than one Ak ,
k = 1, . . . , n. Take, for example, n = 2 and A1 A2 = C. Then
f = 1A1 + 1A2 = 1A1 \C + 21C + 1A2 \C ,
and so
f d = (A1 \ C) + (A2 \ C) + 2(C) = (A1 ) + (A2 ) + (C).
It is easy to see that LSimp,0

is a convex cone, i.e., that itR is closed under finite linear combinations
+
with non-negative coefficients. The integral map f 7 f d preserves this structure:
Problem 3.5 For f1 , f2 LSimp,0
and 1 , 2 0 we have
+
R
R
1. if f1 (x) f2 (x) for all x S then f1 d f2 d, and
R
R
R
2. (1 f1 + 2 f2 ) d = 1 f1 d + 2 f2 d.
37

Having defined the integral for f LSimp,0
, we turn to general non-negative measurable func+
tions. In fact, at no extra cost we can consider a slightly
larger set consisting of all measurable
[0, ]-valued functions which we denote by L0+ [0, ] . While there is no obvious advantage
at this point of integrating a function which takes the value +, it will become clear soon how
convenient it really is.

0
Definition 3.6 (Lebesgue integral
R for nonnegative functions) For a function f L+ [0, ] ,
we define the Lebesgue integral f d of f by

Z
Z
Simp,0
g d : g L+
, g(x) f (x), x S [0, ].
f d = sup
Remark
3.7 While there is no question that the expression above defines uniquely the number
R
f d, one can wonder if it matches the previously given definition of the Lebesgue integral for
simple functions. A simple argument based on the monotonicity property of part 1. of Problem
3.5 can be used to show that this is, indeed, the case.
R
Problem 3.8 Show that f d = if there existsRa measurable set A with (A) > 0 such that
f (x) = for x A. On the other hand, show that f d = 0 for f of the form
(
, x A,
f (x) = 1A (x) =
0,
x 6= A,
whenever (A) = 0. (Note: Relate this to our convention that 0 = 0 = 0.)
Finally, we are ready to define the integral for general measurable functions. Each f L0 can be
written as a difference of two functions in L0+ in many ways. There exists a decomposition which
is, in a sense, minimal. We define
f + = max(f, 0), f = max(f, 0),
so that f = f + f (and both f + and f are measurable). The minimality we mentioned above
is reflected in the fact that for each x S, at most one of f + and f is non-zero.
Definition 3.9 (Integrable functions) A function f L0 is said to be integrable if
Z
Z
+
f d < and
f d < .
The collection of all integrable functions in L0 is denoted by L1 . The family of integrable
functions is tailor-made for the following definition:
38
Definition 3.10 (The Lebesgue integral) For f L1 , we define the Lebesgue integral
f by
Z
Z
Z
+
f d = f d f d.
f d of
Remark 3.11
1. We have seen so far two cases in which an integral for a function f L0 can be defined:
when f 0 or when f L1 . It is possible to combine the two and define the Lebesgue
integral for all functions f L0 with f L1 . The set of all such functions is denoted by
L01 and we set
Z
Z
Z
+
f d = f d f d (, ], for f L01 .
Note that no problems of the form arise here, and also note that, like L0+ , L01 is only
a convex cone, and not a vector space. While the notation L0 and L1 is quite standard, the
one we use for L01 is not.
R
R
2. For A S and f L01 we usually write A f d for f 1A d.
Problem 3.12 Show that the Lebesgue integral remains a monotone operation in L01 . More pre01 and g L0 are such that g(x) f (x), for all x S, then g L01 and
cisely,
show
R
R that if f L
g d f d.
3.2
First properties of the integral
The wider the generality to which a definition applies, the harder it is to prove theorems about it.
Simp,0
Linearity of the integral is a trivial matter for functions in L+
, but you will see how much we
need to work to get it for L0+ . In fact, it seems that the easiest route towards linearity is through
two important results: an approximation theorem and a convergence theorem. Before that, we
need to pick some low-hanging fruit:

Problem 3.13 Show that for f1 , f2 L0+ [0, ] and [0, ] we have
R
R
1. if f1 (x) f2 (x) for all x S then f1 d f2 d.
R
R
2. f d = f d.

Theorem 3.14 (Monotone convergence theorem) Let {fn }nN be a sequence in L0+ [0, ] with
the property that
f1 (x) f2 (x) . . . for all x S.
Then
lim
n
fn d =

where f (x) = limn fn (x) L0+ [0, ] , for x S.
39
f d,

P ROOF RThe (monotonicity) property (1) ofRProblem 3.13
that
R above implies immediately
R
R the sequence fn d is non-decreasing and that fn d f d. Therefore, limn fn d f d. To
R
show the opposite inequality, we deal
with
the
case
f d < Rand pick > 0 and g LSimp,0
+
R
R
with g(x) f (x), for all x S and g d f d (the case f d = is similar and left to
the reader). For 0 < c < 1, define the (measurable) sets {An }nN by
An = {fn cg}, n N.
By the increase of the sequence {fn }nN , the sets {An }nN also increase. Moreover, since the
function cg satisfies cg(x) g(x) f (x) for all x S and cg(x) < f (x) when f (x) > 0, the
increasing convergence fn f implies that n An = S. By non-negativity of fn and monotonicity,
Z
Z
Z
fn d fn 1An d c g1An d,
and so
sup
n
Let g =
Pk
i=1 i 1Bi
fn d c sup
n
g1An d.
be a simple-function representation of g. Then

Z
g1An d =
Z X
k
i=1
i 1Bi An d =
k
X
i=1
i (Bi An ).
Since An S, we have An Bi Bi , i = 1, . . . , k, and the continuity of measure implies that

(An Bi ) (Bi ). Therefore,
Z
Consequently,
lim
n
g1An d
fn d = sup
n
k
X
i (Bi ) =
i=1
fn d c
and the proof is completed when we let c 1.
g d.
g d, for all c (0, 1),
Remark 3.15
1. The monotone convergence theorem is a testament to the incredible robustness of the Lebesgue
integral. This stability with respect to limiting operations is one of the reasons why it is a
de-facto industry standard.
2. The monotonicity condition in the monotone convergence theorem cannot be dropped.
Take, for example S = [0, 1], S = B([0, 1]), and = (the Lebesgue measure), and define
fn = n1(0,n1 ] , for n N.
Then fn (0) = 0 for all n N and fn (x) = 0 for n > x1 and x > 0. In either case fn (x) 0.
On the other hand
Z

fn d = n (0, n1 ] = 1,
40

so that
lim
n
fn d = 1 > 0 =
lim fn d.
n
We will see later that the while the equality of the limit of the integrals and the integral of the
limit will not hold in general, they will always be ordered in a specific way, if the functions
{fn }nN are non-negative (that will be the content of Fatous lemma below).

Proposition 3.16 (Approximation by simple functions) For each f L0+ [0, ] there exists a
Simp,0
sequence {gn }nN L+
such that
1. gn (x) gn+1 (x), for all n N and all x S,
2. gn (x) f (x) for all x S,
3. f (x) = limn gn (x), for all x S, and
4. the convergence gn f is uniform on each set of the form {f M }, M > 0, and, in particular,
on the whole S if f is bounded.
P ROOF For n N, let Ank , k = 1, . . . , n2n be a collection of subsets of S given by

1 k1 k
k
f
<
}
=
f
,
)
, k = 1, . . . , n2n .
[
Ank = { k1
n
n
n
n
2
2
2
2
Note that the sets Ank , k = 1, . . . , n2n are disjoint and that the measurability of f implies that
Ank S for k = 1, . . . , n2n . Define the function gn LSimp,0
by
+
n
gn =
n2
X
k1
n
2 n 1A k
k=1
+ n1{f n} .
The statements 1., 2., and 4. follow immediately from the following three simple observations:
gn (x) f (x) for all x S,
gn (x) = n if f (x) = , and
gn (x) > f (x) 2n when f (x) < n.
Finally, we leave it to the reader to check the simple fact that {gn }nN is non-decreasing.
Problem 3.17 Show, by means of an example, that the sequence {gn }nN would not necessarily be
monotone if we defined it in the following way:
2
gn =
n
X
k1
n 1{f [ k1 , k )}
n n
k=1
41
+ n1{f n} .
Proposition 3.18 (Linearity

of the integral for non-negative functions) For 1 , 2 0 and

0
f1 , f2 L+ [0, ] we have
Z
Z
Z
(1 f1 + 2 f2 ) d = 1 f1 d + 2 f2 d.
P ROOF Thanks to Problem 3.13 it is enough to prove the statement for 1 = 2 = 1. Let {gn1 }nN
Simp,0
and {gn2 }nN be sequences in L+
which approximate f 1 and f 2 in the sense of Proposition 3.16.
The sequence {gn }nN given by gn = gn1 + gn2 , n N, has the following properties:
gn LSimp,0
for n N,
+
gn (x) is a nondecreasing sequence for each x S,
gn (x) f1 (x) + f2 (x), for all x S.
Therefore, we can apply the linearity of integration for the simple functions and the monotone
convergence theorem (Theorem 3.14) to conclude that
Z
Z
Z
Z
Z
Z
1
2
1
2
gn d + gn d = f1 d + f2 d.
(f1 + f2 ) d = lim (gn + gn ) d = lim
n

Corollary 3.19 (Countable additivity of the integral) Let {fn }nN be a sequence in L0+ [0, ] .
Then
Z X
XZ
fn d =
fn d.
nN
nN
P ROOF Apply the monotone convergence theorem to the partial sums gn = f1 + + fn , and use
linearity of integration.
Once we have established a battery of properties for non-negative functions, an extension to L1 is
not hard. We leave it to the reader to prove all the statements in the following problem:
Problem 3.20 The family L1 of integrable functions has the following properties:
R
1. f L1 iff |f | d < ,
2. L1 is a vector space,
R
R
3. f d |f | d, for f L1 .
R
R
R
4. |f + g| d |f | d + |g| d, for all f, g L1 .
42

We conclude the present section with two results, which, together with the monotone convergence
theorem, play the central role in the Lebesgue integration theory.

Theorem 3.21 (Fatous lemma) Let {fn }nN be a sequence in L0+ [0, ] . Then
Z
Z
lim inf fn d lim inf fn d.
n

P ROOF Set gn (x) = inf kn fk (x), so that gn L0+ [0, ] and gn (x) is a non-decreasing sequence
for each x S. The monotone convergence theorem and the fact that lim inf fn (x) = supn gn (x) =
limn gn (x), for all x S, imply that
Z
Z
gn d lim inf d.
n
On the other hand, gn (x) fk (x) for all k n, and so

Z
Z
fk d.
gn d inf
kn
Therefore,
lim
n
gn d lim inf
n kn
fk d = lim inf
n
fk d.
Remark 3.22
1. The inequality in the Fatous lemma does not have to be equality, even if the limit limn fn (x)
exists for all x S. You can use the sequence {fn }nN of Remark 3.15 to see that.
2. Like the monotone convergence theorem, Fatous lemma requires that all function {fn }nN
be non-negative. This requirement is necessary - to see that, simply consider the sequence
{fn }nN , where {fn }nN is the sequence of Remark 3.15 above.
3. The strength of Fatous lemma comes from the fact that, apart from non-negativity, it requires no special properties for the sequence {fn }nN . Its conclusion is not as strong as that
of the monotone convergence theorem, but it proves
to be very useful in various settings beR
cause it gives an upper bound (namely lim inf n fn d) on the integral of the non-negative
function lim inf fn .
Theorem 3.23 (Dominated convergence theorem) Let {fn }nN be a sequence in L0 with the
property that there exists g L1 such that |fn (x)| g(x), for all x X and all n N. If
f (x) = limn fn (x) for all x S, then f L1 and
Z
Z
f d = lim fn d.
n
43

P ROOF The condition |fn (x)| g(x), for all x X and all n N implies that g(x) 0, for all
x S. Since fn+ g, fn g and g L1 , we immediately have fn L1 , for all n N. The limiting
function f inherits the same properties f + g and f g from {fn }nN so f L1 , too.
Clearly g(x) + fn (x) 0 for all n N and all x S, so we can apply Fatous lemma to get
Z
Z
Z
Z
g d + lim inf fn d = lim inf (g + fn ) d lim inf (g + fn ) d
n
n
n
Z
Z
Z
= (g + f ) d = g d + f d.
In the same way (since g(x) fn (x) 0, for all x S, as well), we have
Z
Z
Z
Z
g d lim sup fn d = lim inf (g fn ) d lim inf (g fn ) d
n
n
n
Z
Z
Z
= (g f ) d = g d f d.
Therefore
lim sup
n
and, consequently,
f d = limn
fn d
f d lim inf
n
fn d,
fn d.
Remark 3.24 The dominated convergence theorem combines the lack of monotonicity requirements of Fatous lemma and the strong conclusion of the monotone convergence theorem. The
price to be paid is the uniform boundedness requirement. There is a way to relax this requirement
a little bit (using the concept of uniform integrability), but not too much. Still, it is an unexpectedly
useful theorem.
3.3
Null sets
An important property - inherited directly from the underlying measure - is that it is blind to sets
of measure zero. To make this statement precise, we need to introduce some language:
Definition 3.25 (Null sets) Let (S, S, ) be a measure space.

1. N S is said to be a null set if (N ) = 0.
is called a null function if there exists a null set N such that f (x) = 0
2. A function f : S R
for x N c .
3. Two functions f, g are said to be equal almost everywhere - denoted by f = g, a.e. - if f g
is a null function, i.e., if there exists a null set N such that f (x) = g(x) for all x N c .
Remark 3.26
1. In addition to almost-everywhere equality, one can talk about the almost-everywhere version of any relation between functions which can be defined on points. For example, we
write f g, a.e. if f (x) g(x) for all x S, except, maybe, for x in some null set N .
44

2. One can also define the a.e. equality of sets: we say that A = B, a.e., for A, B S if 1A = 1B ,
a.e. It is not hard to show (do it!) that A = B a.e., if and only if (A B) = 0 (Remember
that denotes the symmetric difference: A B = (A \ B) (B \ A)).
3. When a property (equality of functions, e.g.) holds almost everywhere, the set where it fails
to hold is not necessarily null. Indeed, there is no guarantee that it is measurable at all. What
is true is that it is contained in a measurable (and null) set. Any such (measurable) null set is
often referred to as the exceptional set.
Problem 3.27 Prove the following statements:
1. The almost-everywhere equality is an equivalence relation between functions.
2. The family {A S : (A) = 0 or (Ac ) = 0} is a -algebra (the so-called -trivial algebra).
The blindness property of the Lebesgue integral we referred to above can now be stated formally:
Proposition 3.28 (The blindness property of the Lebesgue integral) Suppose that f = g,
a.e,. for some f, g L0+ . Then
Z
Z
f d =
g d.
c
P ROOF Let N be an Rexceptional set
R for f = g, a.e., i.e., f = g on N and (NR ) = 0. Then
f 1N c = g1N c , and so R f 1N c d = g1N c d. On
R the other hand f 1N 1N and 1N d = 0,
so, by monotonicity, f 1N d = 0. Similarly g1N d = 0. It remains to use the additivity of
integration to conclude that
Z
Z
Z
Z
Z
Z
c
c
f d = f 1N d + f 1N d = g1N d + g1N d = g d.
A statement which can be seen as a converse of Proposition 3.28 also holds:

R
Problem 3.29 Let f L0+ be such that f d = 0. Show that f = 0, a.e. (Hint: What is the
negation of the statement f = 0. a.e. for f L0+ ?)
The monotone convergence theorem and the dominated convergence theorem both require the
sequence {fn }nN functions to converge for each x S. A slightly weaker notion of convergence
is required, though:
Definition 3.30 (Almost-everywhere convergence) A sequence of functions {fn }nN is said to

converge almost everywhere to the function f , if there exists a null set N such that
fn (x) f (x) for all x N c .
45

Remark 3.31 If we want to emphasize that fn (x) f (x) for all x S, we say that {fn }nN
converges to f everywhere.
Proposition 3.32 (Monotone

(almost-everywhere) convergence theorem) Let {fn }nN be a se
quence in L0+ [0, ] with the property that
fn fn+1 a.e., for all n N.
Then
lim
n
if f L0+ and fn f , a.e.
fn d =
f d,
P ROOF There are + 1 a.e.-statements we need to deal with: one for each n N in fn fn+1 ,
a.e., and an extra one when we assume that fn f , a.e. Each of them comes with an exceptional
set; more precisely, let {An }nN be such that fn (x) fn+1 (x) for x Acn and let B be such that
fn (x) f (x) for x B c . Define A S by A = (n An ) B and note that A is a null set. Moreover,
consider the functions f, {fn }nN defined by f = f 1Ac , fn = fn 1Ac . Thanks to the definition of
the set A, fn (x) fn+1 (x), for all n N and x S; hence fn f, everywhere.
R Therefore,
R the
monotone convergence theorem (Theorem

3.14)
thatR fn d f d.
R
R can be used to conclude
R
Finally, Proposition 3.28 implies that fn d = fn d for n N and f d = f d.
Problem 3.33 State and prove a version of the dominated convergence theorem where the almosteverywhere convergence is used. Is it necessary for all {fn }nN to be dominated by g for all x S,
or only almost everywhere?
Remark 3.34 There is a subtlety that needs to be pointed out. If a sequence {fn }nN of measurable functions converges to the function f everywhere, then f is necessarily a measurable function
(see Proposition 1.43). However, if fn f only almost everywhere, there is no guarantee that
f is measurable. There is, however, always a measurable function which is equal to f almost
everywhere; you can take lim inf n fn , for example.
3.4
Additional Problems
Problem 3.35 (The monotone-class theorem) Prove the following result, known as the monotoneclass theorem (remember that an a means that an is a non-decreasing sequence and an a)
Let H be a class of bounded functions from S into R satisfying the following conditions
1. H is a vector space,
2. the constant function 1 is in H, and
3. if {fn }nN is a sequence of non-negative functions in H such that fn (x) f (x), for all
x S and f is bounded, then f H.
Then, if H contains the indicator 1A of every set A in some -system P, then H necessarily
contains every bounded (P)-measurable function on S.
46

(Hint: Use Theorems 3.16 and 2.16)
Problem 3.36 (A form of continuity for Lebesgue integration) Let (S, S, ) be a measure space,
1
and suppose that
R f L . Show that for each > 0 there exists > 0 such that if A S and
(A) < , then A f d < .
Problem 3.37 (Sums as integrals) Consider the measurable space (N, 2N , ), where is the counting measure.
1. For a function f : N [0, ], show that
Z
f d =
f (n).
n=1
2. Use the monotone convergence theorem to show the following special case of Fubinis theorem
X
X
X
akn =
akn ,
k=1 n=1
n=1 k=1
whenever {akn : k, n N} is a double sequence in [0, ].
3. Show that f : N R is in L1 if and only if the series
f (n),
n=1
converges absolutely.
Problem 3.38 (A criterion for integrability) Let (S, S, ) be a finite measure space. For f L0+ ,
show that f L1 if and only if
X
({f n}) < .
nN
Problem
3.39 (A limit of integrals) Let (S, S, ) be a measure space, and suppose f L1+ is such
R
that f d = c > 0. Show that the limit
Z

lim n log 1 + (f /n) d
n
exists in [0, ] for each > 0 and compute its value.

(Hint: Prove and use the inequality log(1 + x ) x, valid for x 0 and 1.)
Problem 3.40 (Integrals converge but the functionsRdont . . . ) Construct an sequence {fn }nN of
continuous functions fn : [0, 1] [0, 1] such that fn d 0, but the sequence {fn (x)}nN is
divergent for each x [0, 1].
Problem 3.41 (. . . or they do, but are not dominated)
Construct an sequence {fn }nN of continuR
ous functions fn : [0, 1] [0, ) such that fn d 0, and fn (x) 0 for all x, but f 6 L1 , where
f (x) = supn fn (x).
47

Problem 3.42 (Functions measurable in the generated -algebra) Let S 6= be a set and let f :
S R be a function. Prove that a function g : S R is measurable with respect to the pair
((f ), B(R)) if and only if there exists a Borel function h : R R such that g = h f .
Problem 3.43 (A change-of-variables formula) Let (S, S, ) and (T, T , ) be measurable spaces,
and let F : S T be a measurable function with the property that = F (i.e., is the pushforward of through F ). Show that for every f L0+ (T, T ) or f L1 (T, T ), we have
Z
Z
f d = (f F ) d.
Problem 3.44 (The Riemann Integral) A finite collection = {t0 , . . . , tn }, where a = t0 < t1 <
< tn = b and n N, is called a partition of the interval [a, b]. The set of all partitions of [a, b] is
denoted by P ([a, b]).
For a bounded function f : [a, b] R and = {t0 , . . . , tn } P ([a, b]), we define its upper and
lower Darboux sums U (f, ) and L(f, ) by
!
n
X
U (f, ) =
sup f (t) (tk tk1 )
k=1
and
L(f, ) =
n
X
k=1
t(tk1 ,tk ]
inf
t(tk1 ,tk ]
f (t) (tk tk1 ).
A function f : [a, b] R is said to be Riemann integrable if it is bounded and

sup
L(f, ) =
P ([a,b])
inf
P ([a,b])
U (f, ).
In that case the common value of the supremum and the infimum above is called the Riemann
Rb
integral of the function f - denoted by (R) a f (x) dx.
1. Suppose that a bounded Borel-measurable function f : [a, b] R is Riemann-integrable.
Show that
Z
Z b
f d = (R)
f (x) dx.
[a,b]
2. Find an example of a bounded an Borel-measurable function f : [a, b] R which is not

Riemann-integrable.
3. Show that every continuous function is Riemann integrable.
4. It can be shown that for a bounded Borel-measurable function f : [a, b] R the following
criterion holds (and you can use it without proof):
f is Riemann integrable if and only if there exists a Borel set D [a, b] with (D) = 0 such that f
is continuous at x, for each x [a, b] \ D. Show that
all monotone functions are Riemann-integrable,
f g is Riemann integrable if f : [c, d] R is Riemann integrable and g : [a, b] [c, d]

is continuous,
48

products of Riemann-integrable functions are Riemann-integrable.
5. Let ([a, b], B([a, b]) , ) be the completion of ([a, b], B([a, b]), ). Show that each Riemannintegrable function on [a, b] is B([a, b]) -measurable.
(Hint: Pick a sequence {n }nN in P ([a, b]) so that n n+1 and U (f, n ) L(f, n )
0. Using those partitions and the function f , define two sequences of Borel-measurable
R
functions {f n }nN and {f n }nN so that f n f , f n f , f f f , and (f f ) d = 0.
Conclude that f agrees with a Borel measurable function on a complement of a subset of the
set {f 6= f } which has Lebesgue measure 0. )
49
Chapter
Lebesgue Spaces and Inequalities

4.1
Lebesgue spaces
We have seen how the family of all functions f R L1 forms a vector space and how the map
f 7 ||f ||L1 , from L1 to [0, ) defined by ||f ||L1 = |f | d has the following properties
1. f = 0 implies ||f ||L1 = 0, for f L1 ,
2. ||f + g||L1 ||f ||L1 + ||g||L1 , for f, g L1 ,

3. ||f ||L1 = || ||f ||L1 , for R and f L1 .
Any map from a vector space into [0, ) with the properties 1., 2., and 3. above is called a pseudo
norm. A pair (V, || ||) where V is a vector space and || || is a pseudo norm on V is called a
pseudo-normed space.
If a pseudo norm happens to satisfy the (stronger) axiom
1. f = 0 if and only if ||f ||L1 = 0, for f L1 ,
instead of 1., it is called a norm, and the pair (V, || ||) is called a normed space.
The pseudo-norm || ||L1 is, in general, not a norm. Indeed, by Problem 3.29, we have ||f ||L1 =
0 iff f = 0, a.e., and unless is the only null-set, there are functions different from the constant
function 0 with this property.
Remark 4.1 There is a relatively simple procedure one can use to turn a pseudo-normed space
(V, || ||) into a normed one. Declare two elements x, y in V equivalent (denoted by x y) if
||y x|| = 0, and let V be the quotient space V / (the set of all equivalence classes). It is easy
to show that ||x|| = ||y|| whenever x y, so the pseudo-norm || || can be seen as defined on V .
Moreover, it follows directly from the properties of the pseudo norm that (V , || ||) is, in fact a
normed space. Idea is, of course, bundle together the elements of V which differ by such a small
amount that || || cannot detect it.
This construction can be applied to the case of the pseudo-norm || ||L1 on L1 , and the resulting
normed space is denoted by L1 . The normed space L1 has properties similar to those of L1 , but
its elements are not functions anymore - they are equivalence classes of measurable functions.
Such a point of view is very useful in analysis, but it sometimes leads to confusion in probability
(especially when one works with stochastic processes with infinite time-index sets). Therefore, we
will stick to L1 and deal with the fact that it is only a pseudo-normed space.
50
CHAPTER 4. LEBESGUE SPACES AND INEQUALITIES

A pseudo-norm || || on a vector space can be used to define a pseudo metric (pseudo-distance
function) on V by the following simple prescription:
d(x, y) = ||y x||, x, y V.
Just like a pseudo norm, a pseudo metric has most of the properties of a metric
1. d(x, y) [0, ), for x, y V ,
2. d(x, y) + d(y, z) d(x, z), for x, y, z V ,
3. d(x, y) = d(y, x), x, y V ,
4. x = y implies d(x, y) = 0, for x, y V .
The missing axiom is the stronger version of 4. given by
4. x = y if and only if d(x, y) = 0, for x, y V .
Luckily, a pseudo metric is sufficient for the notion of convergence, where we say that a sequence
{xn }nN in V converges towards x V if d(xn , x) 0, as n . If we apply it to our original
example (L1 , || ||L1 ), we have the following definition:
Definition 4.2 (Convergence in L1 ) For a sequence {fn }nN in L1 , we say that {fn }nN converges
to f in L1 if
||fn f ||L1 0.
To get some intuition about convergence in L1 , here is a problem:
Problem 4.3 Show that the conclusion of the dominated convergence theorem (Theorem 3.23) can
be replaced by fn f in L1 . Does the original conclusion follow from the new one?
The only problem that arises when one defines convergence using a pseudo metric (as opposed
to a bona-fide metric) is that limits are not unique. This is, however, merely an inconvenience and
one gets used to it quite readily:
Problem 4.4 Suppose that {fn }nN converges to f in L1 . Show that {fn }nN also converges to
g L1 if and only if f = g, a.e.
In addition to the space L1 , one can introduce many other vector spaces of similar flavor. For
p [1, ), let Lp denote the family of all functions f L0 such that |f |p L1 .
Problem 4.5 Show that there exists a constant C > 0 (depending on p, but independent of a, b)
such that (a + b)p C(ap + bp ), p (0, ) and for all a, b 0. Deduce that Lp is a vector space for
all p (0, ).
We will see soon that the map || ||Lp , defined by
||f ||Lp =
Z
|f | d
51
1/p
, f Lp ,

is a pseudo norm on Lp . The hard part of the proof - showing that ||f + g||Lp ||f ||Lp + ||g||Lp will
be a direct consequence of an important inequality of Minkowski which will be proved below.
Finally, there is a nice way to extend the definition of Lp to p = .
is called an essential supremum of the
Definition 4.6 (Essential supremum) A number a R
function f L0 - and is denoted by a = esssup f - if
1. ({f > a}) = 0
2. ({f > b}) > 0 for any b < a.
A function f L0 with esssup f < is said to be essentially bounded from above. When
esssup |f | < , we say that f is essentially bounded.
Remark 4.7 Even though the function f may take values larger than a, it does so only on a null set.
It can happen that a function is unbounded, but that its essential supremum exists in R. Indeed,
take (S, S, ) = ([0, 1], B([0, 1]), ), and define
(
n, x = n1 for some n N,
f (x) =
0, otherwise.
Then esssup f = 0, since ({f > 0}) = ({1, 1/2, 1/3, . . . }) = 0, but supx[0,1] f (x) = .
Let L denote the family of all essentially bounded functions in L0 . Define ||f ||L = esssup |f |,
for f L .
Problem 4.8 Show that L is a vector space, and that || ||L is a pseudo-norm on L .
The convergence in Lp for p > 1 is defined similarly to the L1 -convergence:
Definition 4.9 (Convergence in Lp ) Let p [1, ]. We say that a sequence {fn }nN in Lp converges in Lp to f Lp if
||fn f ||Lp 0, as n .
Problem 4.10 Show that {fn }nN L converges to f L in L if and only if there exist
functions {fn }nN , f in L0 such that
1. fn = fn , a.e., and f = f , a.e, and
2. fn f uniformly (we say that gn g uniformly if supx |gn (x) g(x)| 0, as n ).
52
4.2
Inequalities
Definition 4.11 (Conjugate exponents) We say that p, q [1, ] are conjugate exponents if
1
1
p + q = 1.
Lemma 4.12 (Youngs inequality) For all x, y 0 and conjugate exponents p, q [1, ) we have
xp
p
(4.1)
yq
q
xy.
The equality holds if and only if xp = y q .
P ROOF If x = 0 or y = 0, the inequality trivially holds so we assume that x > 0 and y > 0. The
function log is strictly concave on (0, ) and p1 + 1q = 1, so
log( p1 + 1q )
1
p
log() + 1q log(),
for all , > 0, with equality if and only if = . If we substitute = xp and = y q , and
exponentiate both sides, we get
xp
p
yq
q
exp( p1 log(xp ) + 1q log(y q )) = xy,
with equality if and only if xp = y q .

Remark 4.13 If you do not want to be fancy, you can prove Youngs inequality by locating the
maximum of the function x 7 xy p1 xp using nothing more than elementary calculus.
Proposition 4.14 (Hlders inequality) Let p, q [0, ] be conjugate exponents. For f Lp and
g Lq , we have
Z
|f g| d ||f ||Lp ||g||Lq .
(4.2)
The equality holds if and only if there exist constants , 0 with + > 0 such that |f |p = |g|q ,
a.e.
P ROOF We assume that 1 < p, g < and leave the (easier) extreme cases to the reader. Clearly,
we can also assume that ||f ||Lp > 0 and ||q||Lq > 0 - otherwise, the inequality is trivially satisfied.
We define f = |f | /||f ||Lp and g = |g| /||g||Lq , so that ||f||Lp = ||
g ||Lq = 1.
Plugging f for x and g for y in Youngs inequality (Lemma 4.12 above) and integrating, we get
Z
Z
Z
q
p
1
1
(4.3)
f d + q g d fg d,
p
53

and consequently,
Z
(4.4)
fg d 1,
R
R
p
because fp d = ||f||Lp = 1, and gq d = ||
g ||Lp = 1 and p1 + 1q = 1. Hlders inequality (4.2)
now follows by multiplying both sides of (4.4) by ||f ||Lp ||g||Lq .
If the equality in (4.2) holds, then it also holds a.e. in the Youngs inequality (4.3). Therefore,
the equality will hold if and only if ||g||qLq |f |p = ||f ||pLp |g|q , a.e. The reader will check that if a pair
of constants , as in the statement exists, then (||g||qLq , ||f ||pLp ) must be proportional to it.
For p = q = 2 we get the following well-known special case:
Corollary 4.15 (Cauchy-Schwarz inequality) For f, g L2 , we have

Z
|f g| d ||f ||L2 ||g||L2 .
Corollary 4.16 (Minkowskis inequality) For f, g Lp , p [1, ], we have

(4.5)
||f + g||Lp ||f ||Lp + ||g||Lp .
P ROOF Like above, we assume p < and leave the case p = to the reader. Moreover, we
assume that ||f + g||Lp > 0 - otherwise, the inequality trivially holds. Note, first that for conjugate
exponents p, q we have q(p 1) = p. Therefore, Hlders inequality implies that
Z
|f | |f + g|
p1
d ||f ||Lp ||(f + g)
p1
||Lq = ||f ||Lp
p/q
Z
|f + g|
q(p1)
1/q
= ||f ||Lp ||f + g||Lp ,

and, similarly,
Z
p/q
|g| |f + g|p1 d ||g||Lp ||f + g||Lp .
Therefore,
||f +
g||pLp
|f + g| d
|f | |f + g|
p1

||f ||Lp + ||g||Lp ||f + g||p1
Lp ,
and if we divide through by ||f + g||p1

Lp > 0, we get (4.5).
54
d +
|g| |f + g|p1 d
Corollary 4.17 (Lp is pseudo-normed) (Lp , || ||Lp ) is a pseudo-normed space for each p [1, ].
A pseudo-metric space (X, d) is said to be complete if each Cauchy sequence converges. A sequence {xn }nN is called a Cauchy sequence if
> 0, N N, m, n N d(xn , xm ) < .
A pseudo-normed space (V, || ||) is called a pseudo-Banach space if is it complete for the metric
induced by || ||. If || || is, additionally, a norm, (V, || ||) is said to be a Banach space.
Problem 4.18 Let {xn }nN be Cauchy sequence in a pseudo-metric space (X, d), and let {xnk }kN
be a subsequence of {xn }nN which converges to x X. Show that xn x.
Proposition 4.19 (Lp is pseudo-Banach) Lp is a pseudo-Banach space, for p [1, ].
P ROOF We assume p [1, ) and leave the case p = to the reader. Let {fn }nN be a Cauchy
sequence in Lp . Thanks to the Cauchy property, there exists a subsequence of {fnk }kN such that
||fnk+1 fnk ||Lp < 2k , for all k N.

Pk1
We define the sequence {gk }kN in L0+ by gk = |fn1 | + i=1
fni+1 fni , as well as the function
g = limk gk L0 ([0, ]). The monotone-convergence theorem implies that
Z
Z
p
g d = lim gnp d,
n
and, by Minkowskis inequality, we have

Z
gkp d
||gk ||pLp
||fn1 ||Lp +
k1
X
i=1
||fnk+1 fnk ||Lp
p
(||fn1 ||Lp + 1)p , k N.
p
Therefore, g p d (1 + ||fn1 ||Lp )p < , and, in particular,
P g L and g < , a.e. It follows
immediately from the absolute convergence of the series i=1 fnk+1 fnk that
fnk (x) = fn1 (x) +
k1
X
i=1
(fnk+1 (x) fnk (x)),
converges in R, for almost all x. Hence, the function f = lim inf k fnk is in Lp since |f | g, a.e.
Since |f | g and |fnk | g, forRall k N, we have |f fnk |p 2 |g|p L1 , so the dominated
convergence theorem implies that |fnk f |p d 0, i.e., fnk f in Lp . Finally, we invoke the
result of Problem 4.18 to conclude that fn f in Lp .
The following result is a simple consequence of the (proof of) Proposition 4.19.
55
Corollary 4.20 (Lp -convergent a.e.-convergent, after passing to subsequence) For p

[1, ], let {fn }nN be a sequence in Lp such that fn f in Lp . Then there exists a subsequence
{fnk }kN of {fn }nN such that fnk f , a.e.
We have seen above how the concavity of the function log was used in the proof of Youngs
inequality (Lemma 4.12). A generalization of the definition of convexity, called Jensens inequality,
is one of the most powerful tools in measure theory. Recall that L01 denotes the set of all f L0
with f L1 .
Proposition 4.21 (Jensens inequality) Suppose that (S) = 1 ( is a probability measure) and
that : R R is a convex function. For a function f L1 we have (f ) L01 and
Z
Z
(f ) d ( f d).
Before we give a proof, we need a lemma about convex functions:
Lemma 4.22 (Convex functions as suprema of sequences of affine functions) Let : R R

be a convex function. Then, there exists two sequences {an }nN and {bn }nN of real numbers such that
(x) = sup(an x + bn ).
nN
P ROOF (*) For x R, we define the left and right derivative
and
+
x
of at x by

(x) = sup 1 ((x) (x )),
x
>0
and
+
(x) = inf 1 ((x + ) (x)).
>0
x
Convexity of the function implies that the difference quotient

7 1 ((x + ) (x)), 6= 0,
is a non-decreasing function (why?), and so both
Moreover, we always have
(4.6)
1
((x)
(x ))
and
+
x
are, in fact, limits as > 0.

+
(x)
(x) 1 ((x + ) (x)),
x
x
for all x and all , > 0.

56

Let {qn }nN be an enumeration of rational numbers in R. For each n N we pick an

+
[ x (qn ), x
(qn )] and set bn = (qn ) an qn , so that the line x 7 an x + bn passes through
(qn , (qn )) and has a slope which is between the left and the right derivative of at qn .
Let us first show that (x) an x + bn for all x R and all n N. We pick x R and assume
that x qn (the case x < qn is analogous). If x = qn then (x) = an x + bn by construction, and we
are done. When = x qn > 0, relation (4.6) implies that

n)
(x qn ) + (qn ) = (x).
an x + bn = an (x qn ) + (qn ) (x)(q
xqn
Conversely, suppose that (x) > supn (an x + bn ), for some x R. Both functions and , where
(x) = supn (an x+bn ) are convex (why?), and, therefore, continuous. It follows from (x) > (x),
that there exists a rational number qn such that (qn ) > (qn ). This is a contradiction, though,
since (qn ) an qn + bn = (qn ).
P ROOF (P ROOF OF P ROPOSITION 4.21) Let us first show that ((f )) L1 . By Lemma 4.22, there
exists sequences {an }nN and {bn }nN such that (x) = supnN (an x + bn ). In particular, (f (x))
a1 f (x) + b1 , for all x S. Therefore,

(f (x)) (a1 f (x) + b1 ) |a1 | |f (x)| + |b1 | L1 .
R
R
R
Next, we have (f ) d an f + bn d = an f d + bn , for all n N. Therefore,
Z
Z
Z
(f ) d sup(an f d + bn ) = ( f d).
n
Problem 4.23 State and prove a generalization of Jensens inequality when is defined only on
an interval I of R, but ({f 6 I}) = 0.
Problem 4.24 Use Jensens inequality on an appropriately chosen measure space to prove the
arithmetic-geometric inequality
a1 ++an
n a1 . . . an , for a1 , . . . , an 0.
n
The following inequality is know as Markovs inequality in probability theory, but not much
wider than that. In analysis it is known as Chebyshevs inequality.
Proposition 4.25 (Markovs inequality) For f L0+ and > 0 we have

Z
({f }) 1 f d.
P ROOF Consider the function g L0+ defined by
(
g(x) = 1{f } =
,
0,
f (x) [, )
f (x) [0, ).
Then f (x) g(x) for all x S, and so

Z
Z
f d g d = ({f }).
57
4.3
Additional problems
Problem 4.26 (Projections onto a convex set) A subset K of a vector space is said to be convex if
x + (1 )y K, whenever x, y K and [0, 1]. Let K be a closed and convex subset of L2 ,
and let g be an element of its complement L2 \ K. Prove that
1. There exists an element f K such that ||g f ||L2 ||g f ||L2 , for all f K.
R
2. (f f )(g f ) d 0, for all f K.
(Hint: Pick a sequence {fn }nN in K with ||fn g||L2 inf f K ||f g||L2 and show that it
is Cauchy. Use (but prove first) the parallelogram identity 2||h||2L2 + 2||k||2L2 = ||h + k||2L2 +
||h k||2L2 , for h, k L2 .)
Problem 4.27 (Egorovs theorem) Suppose that is a finite measure, and let {fn }nN be a sequence in L0 which converges a.e. to f L0 . Prove that for each > 0 there exists E S
with (E c ) < such that
lim esssup |fn 1E f 1E | = 0.
n
(Hint: Define An,k = mn {|fm f | k1 }, show that for each k N, there exists nk N such that
(Ank ,k ) < /2k , and set E = k Acnk ,k .)
Problem 4.28 (Relationships between different Lp spaces)
1. Show that for p, q [1, ), we have
||f ||Lp ||f ||Lq (S)r ,
where r = 1/p 1/q. Conclude that Lq Lp , for p q if (S) < .
2. For p0 [1, ), construct an example of a measure space (S, S, ) and a function f L0
such that f Lp if and only if p = p0 .
3. Suppose that f Lr L , for some r [1, ). Show that f Lp for all p [r, ) and
||f ||L = lim ||f ||Lp .
p
Problem 4.29 (Convergence in measure) A sequence {fn }nN in L0 is said to converge in measure toward f L0 if

> 0, |fn f | 0 as n .
Assume that (S) < (parts marked by () are true without this assumption).
1. Show that the mapping
d(f, g) =
|f g|
1+|f g|
d, f, g L0 ,
defines a pseudo metric on L0 and that convergence in d is equivalent to convergence in

measure.
2. Show that fn f , a.e., implies that fn f in measure. Give an example which shows that
the assumption (S) < is necessary.
58

3. Give an example of a sequence which converges in measure, but not a.e.
4. () For f L0 and a sequence {fn }nN in L0 , suppose that
X

|fn f | < , for all > 0.
nN
Show that fn f , a.e.

5. () Show that each sequence convergent in measure has a subsequence which converges a.e.
6. () Show that each sequence convergent in Lp , p [1, ) converges in measure.
7. For p [1, ), find an example of a sequence which converges in measure, but not in Lp .
8. Let {fn }nN be a sequence in L0 with the property that any of its subsequences admits a
(further) subsequence which converges a.e. to f L0 . Show that fn f in measure.
9. Let : R2 R be a continuous function, and let {fn }nN and {gn }nN be two sequences in
L0 . If f, g L0 are such that fn f and gn g in measure, then
(fn , gn ) (f, g) in measure.
(Note: The useful examples include (x, y) = x + y, (x, y) = xy, etc.)
59
Chapter
Theorems of Fubini-Tonelli and

Radon-Nikodym
5.1
Products of measure spaces
We have seen in Chapter 2 that it is possible to define products of arbitrary collections of measurable spaces - one generates the -algebra on the product by all finite-dimensional cylinders. The
purpose of the present section is to extend that construction to products of measure spaces, i.e., to
define products of measures.
Let us first consider the case of two measure spaces (S, S, S ) and (T, T , T ). If the measures
are stripped, the product S T is endowed with the product -algebra S T = ({A B : A
S, B T }). The family P = {A B : A S, B T } serves as a good starting point towards the
creation of the product measure S T . Indeed, if we interpret of the elements in P as rectangles
of sorts, it is natural to define
(S T )(A B) = S (A)T (B).
The family P is a -system (why?), but not necessarily an algebra, so we cannot use Theorem 2.10
(Caratheodorys extension theorem) to define an extension of S T to the whole S T . It it not
hard, however, to enlarge P a little bit, so that the resulting set is an algebra, but that the measure
S T can still be defined there in a natural way. Indeed, consider the smallest algebra that
contains P. It is easy to see that it must contain the family S defined by
A = {nk=1 Ak Bk : n N, Ak S, Bk T , k = 1, . . . , n}.
Problem 5.1 Show that A is, in fact, an algebra and that each element C A can be written in the
form
C = nk=1 Ak Bk ,
for n N, Ak S, Bk T , k = 1, . . . , n, such that A1 B1 , . . . , An Bn are pairwise disjoint.
The problem above allows us to extend the definition of the set function S T to the entire A
by
n
X
(S T )(C) =
S (Ak )T (Bk ),
k=1
60
CHAPTER 5. THEOREMS OF FUBINI-TONELLI AND RADON-NIKODYM

where C = nk=1 Ak Bk for n N, Ak S, Bk T , k = 1, . . . , n is a representation of C with
pairwise disjoint A1 B1 , . . . , An Bn .
At this point, we could attempt to show that the so-defined set function is -additive on A
and extend it using the Caratheodory extension theorem. This is indeed possible - under the
additional assumption of -finiteness - but we will establish the existence of product measures as
a side-effect in the proof of Fubinis theorem below.
Lemma 5.2 (Sections of measurable sets are measurable) Let C be an S T -measurable subset
of S T . For each x S the section Cx = {y T : (x, y) C} is measurable in T .
P ROOF In the spirit of most of the measurability arguments seen so far in these notes, let C denote
the family of all C S T such that Cx is T -measurable for each x S. Clearly, the rectangles
A B, A S, B T are in A because their sections are either equal to or B, for each x S.
Remember that the set of all rectangles generates S T . The proof of the theorem will, therefore,
be complete once it is established that C is a -algebra. This easy exercise is left to the reader.
Problem 5.3 Show that an analogous result holds for measurable functions, i.e., show that if f :
is a S T -measurable function, then the function x 7 f (x, y0 ) is S-measurable for
ST R
each y0 T , and the function y 7 f (x0 , y) is T -measurable for each y0 T .
Proposition 5.4 (A simple Cavallieris principle) Let S and T be finite measures. For C
S T , define the functions C : T [0, ) and C : S [0, ) by
C (y) = S (Cy ), C (x) = T (Cx ).
Then,
1. C L0+ (T ),
2. C L0+ (S),
R
R
3. C dT = C dS .
P ROOF Note that, by Problem 5.3, the function x 7 1C (x, y) is S-measurable for each y T .
Therefore,
Z
(5.1)
1C (, y) dS = S (Cy ) = C (y),
and the function C is well-defined.
Let C denote the family of all sets in S T such that (1), (2) and (3) in the statement of the
proposition hold. First, observe that C contains all rectangles A B, A S, B T , i.e., it contains
a -system which generates S T . So, by the - Theorem (Theorem 2.16), it will be enough to
show that C is a -system. We leave the details to the reader (Hint: Use representation (5.1) and
the monotone convergence theorem. Where is the finiteness of the measures used?)
61
Proposition 5.5 (Simple Cavallieri holds for -finite measures) The conclusion of Proposition
5.4 continues to hold if we assume that S and T are only -finite.
P ROOF (*) Thanks to -finiteness, there exists pairwise disjoint sequences {An }nN and {Bn }nN
in S and T , respectively, such that n An = S, m Bm = T and S (An ) < and S (Bm ) < , for
all m, n N.
For m, n N, define the set-functions nS and m
T on S and T respectively by
nS (A) = S (An A), m
T (B) = T (Bm B).
n
m
It is easy to check that
N are finite measures on S and T , respectively.
Pall nS and T , m, n
P
n
n
Moreover, S (A) = n=1 S (A), T (B) = m=1 m
T (B). In particular, if we set C (y) = S (Cy )
m
m
and C (x) = T (Cx ), for all x S and y S, we have
C (y) = S (Cy ) =
C (x) = T (Cx ) =
nS (Cy ) =
n=1
nC (y), and
n=1
m
T (Cx ) =
m=1
m
C
(x),
m=1
for all x S, y T .
We can apply the conclusion of Proposition 5.4 to all pairs (S, S, nS ) and (T, T , m
T ), m, n N,
of finite measure spaces to conclude that all elements of the sums above are measurable functions
and that so are C and C .
P
P
i
Similarly, the sequences of non-negative functions ni=1 nC (y) and m
i=1 C (x) are non-decreasing
and converge to C and C . Therefore, by the monotone convergence theorem,
Z
C dT = lim
n
n Z
X
iC
dT , and
i=1
C dS = lim
n Z
X
i
C
dS .
i=1
m dn , by Proposition 5.4. Summing over all n N

On the other hand, we have nC dm
C
S
T =
we have
Z
Z
Z
X
m
m
n
m
C dT =
C dS = C
dS ,
nN
where the last equality follows from the fact (see Problem 5.7 below) that
Z
XZ
f dnS = f dS ,
nN
for all f L0+ . Another summation - this time over m N - completes the proof.
Remark 5.6 The argument of the proof above uncovers the fact that integration is a bilinear operation, i.e., that the mapping
Z
(f, )
is linear in both arguments.
62
f d,

Problem 5.7 Let {An }nN be a measurable partition of S, and let the measure n be defined by
n (A) = (A An ) for all A S. Show that for f L0+ , we have
Z
XZ
f dn .
f d =
nN
Proposition 5.8 (Finite products of measure spaces) Let (Si , Si , i ), i = 1, . . . , n be finite measure spaces. There exists a unique measure - denoted by 1 n - on the product space
(S1 Sn , S1 Sn ) with the property that
(1 n )(A1 An ) = 1 (A1 ) . . . n (An ),
for all Ai Si , i = 1, . . . , n. Such a measure is necessarily -finite.
P ROOF To simplify the notation, we assume that n = 2 - the general case is very similar. For
C S1 S2 , we define
Z
(1 2 )(C) =
C d2 , where C (y) = 1 (Cy ) and Cy = {x S1 : (x, y) C}.
S2
It follows from Proposition 5.5 that 1 2 is well-defined as a map from S1 S2 to [0, ]. Also,
it is clear that (1 2 )(A B) = 1 (A)2 (B), for all A S1 , B S2 . It remains to show that
1 2 is a measure. We start with a pairwise disjoint sequence {Cn }nN in S1 S2 . For y S2 ,
the sequence {(Cn )y }nN is also pairwise disjoint, and so, with C = n Cn , we have
X
X
C (y) = 1 (Cy ) =
2 (Cn )y =
Cn (y), y S2 .
nN
nN
Therefore, by the monotone convergence theorem (see Problem 3.37 for details) we have
Z
XZ
X
(1 2 )(C) =
C d2 =
(1 2 )(Cn ).
Cn d =
S2
nN S2
nN
Finally, let {An }nN , {Bn }nN be sequences in S1 and S2 (respectively) such that 1 (An ) <
and 2 (Bn ) < for all n N and n An = S1 , n Bn = S2 . Define {Cn }nN as an enumeration of
the countable family {Ai Bj : i, j N} in S1 S2 . Then (1 2 )(Cn ) < and all n N and
n Cn = S1 S2 . Therefore, 1 2 is -finite.
The measure 1 n is called the product measure, and the measure space (S1 Sn , S1
Sn , 1 n ) the product (measure space) of measure spaces (S1 , S1 , 1 ), . . . , (Sn , Sn , n ).
Now that we know that product measures exist, we can state and prove the important theorem
which, when applied to integrable functions bears the name of Fubini, and when applied to nonnegative functions, of Tonelli. We state it for both cases simultaneously (i.e., on L01 ) in the case of
a product of two measure spaces. An analogous theorem for finite products can be readily derived
from it. When
the variable or the underlying measure
space of integration needs to be specified,
R
R
we write S f (x) (dx) for the Lebesgue integral f d.
63
Theorem 5.9 (Fubini, Tonelli) Let (S, S, S ) and (T, T , T ) be two -finite measure spaces. For
f L01 (S T ) we have

Z Z
Z Z
f (x, y) T (dy) S (dx) =
f (x, y) S (dx) T (dy)
S
T
S
(5.2)
ZT
= f d(S T ).
P ROOF All the hard work has already been done. We simply need to crank the Standard Machine.
Let H denote the family of all functions in L0+ (ST ) with the property that (5.2) holds. Proposition
5.5 implies that H contains the indicators of all elements of S T . Linearity of all components
of (5.2) implies that H contains all simple functions in L0+ , and the approximation theorem 3.16
implies that the whole L0+ is in H. Finally, the extension to L01 follows by additivity.
Since f is always in L01 , we have the following corollary
Corollary 5.10 (An integrability criterion) For f L0 (S T ), we have

Z Z
01
f (x, y) T (dy) S (dx) < .
f L (S T ) if and only if
T
Example 5.11 (-finiteness cannot be left out . . . ) The assumption of -finiteness cannot be left
out of the statement of Theorem 5.9. Indeed, let (S, S, ) = ([0, 1], B([0, 1]), ) and (T, T , ) =
([0, 1], 2[0,1] , ), where is the counting measure on 2[0,1] , so that (T, T , ) fails the -finite property.
Define f L0 (S T ) (why it is product-measurable?) by
(
1, x = y,
f (x, y) =
0, x 6= y.
Then
and so
Z
Z Z
T
On the other hand,
and so
f (x, y) (dx) = ({y}) = 0,

S
f (x, y) (dx) (dy) =

S
Z
Z Z
S
0 (dy) = 0.
[0,1]
f (x, y) (dy) = ({x}) = 1,

T
f (x, y) (dy) (dx) =

T
64
1 (dx) = 1.
[0,1]

Example 5.12 (. . . and neither can product-integrability) The integrability of either f + or f for
f L0 (ST ) is (essentially) necessary for validity of Fubinis theorem, even if all iterated integrals
exist. Here is what can go wrong. Let (S, S, ) = (T, T , ) = (N, 2N , ), where is the counting
measure. Define the function f : N N R by
m = n,
1,
f (n, m) = 1, m = n + 1,
0,
otherwise
Then
f (n, m) (dm) =
T
and so
Z Z
S
On the other hand,

Z
f (n, m) (dn) =
S
f (n, m) (dm) (dn) = 0.

T
f (n, m) =
nN
i.e.,
Therefore,
f (n, m) = 0 + + 0 + 1 + (1) + 0 + = 0,
mN
Z
Z Z
T
1 + 0 + = 1,
0 + + 0 + (1) + 1 + 0 + . . . ,
m=1
m > 1,
f (n, m) (dn) = 1{m=1} .
f (n, m) (dn) (dm) =

S
1{m=1} (dm) = 1.
If you think that using the counting measure is cheating, convince yourself that it is not hard to
transfer this example to the setup where (S, S, ) = (T, T , ) = ([0, 1], B([0, 1]), ).
The existence of the product measure gives us an easy access to the Lebesgue measure on
higher-dimensional Euclidean spaces. Just as on R measures the length of sets, the Lebesgue
measure on R2 will measure area, the one on R3 volume, etc. Its properties are collected in
the following problem:
Problem 5.13 For n N, show the following statements:
1. There exists a unique measure (note the notation overload) on B(Rn ) with the property
that

[a1 , b1 ) [an , bn ) = (b1 a1 ) . . . (bn an ),
for all a1 < b1 , . . . , an < bn in R.
2. The measure on Rn is invariant with respect to all isometries1 of Rn .

Note: Feel free to use the following two facts without proof:
a) A function f : Rn Rn is an isometry if and only if there exists x0 Rn and an

orthogonal linear transformation O : Rn Rn such that f (x) = x0 + Ox.
An isometry of Rn is a map f : Rn Rn with the property that d(x, y) = d(f (x), f (y)) for all x, y Rn . It can be
shown that the isometries of R3 are precisely translations, rotations, reflections and compositions thereof.
65

b) Let O be an orthogonal linear transformation. Then R1 and OR1 have the same Lebesgue
measure, where R1 denotes the unit rectangle [0, 1) [0, 1). (the least painful way
to prove this fact is by using the change-of-variable formula for the Riemann integral).
5.2
The Radon-Nikodym Theorem
We start the discussion of the Radon-Nikodym theorem with a simple observation:

Problem 5.14 (Integral as a measure) For a function f L0 ([0, ]), we define the set-function
: S [0, ] by
Z
(A) =
f d.
(5.3)
A
1. Show that is a measure.

2. Show that (A) = 0 implies (A) = 0, for all A S.
3. Show that the following two properties are equivalent
(A) = 0 if and only if (A) = 0, A S, and
f > 0, a.e.
Definition 5.15 (Absolute continuity, etc.) Let , be measures on the measurable space (S, S).
We say that
1. is absolutely continuous with respect to - denoted by - if (A) = 0, whenever
(A) = 0, A S.
2. and are equivalent if and , i.e., if (A) = 0 (A) = 0, for all A S,
3. and are (mutually) singular - denoted by - if there exists D S such that (D) = 0
and (Dc ) = 0.
Problem 5.16 Let and be measures with finite and . Show that for each > 0 there
exists > 0 such that for each A S, we have (A) (A) . Show that the assumption
that is finite is necessary.
Problem 5.14 states that the prescription (5.3) defines a measure on S which is absolutely continuous with respect to . What is surprising is that the converse also holds under the assumption
of -finiteness: all absolutely continuous measures on S are of that form. That statement (and
more) is the topic of this section. Since there is more than one measure in circulation, we use the
convention that a.e. always uses the notion of the null set as defined by the measure .
Theorem 5.17 (The Lebesgue decomposition) Let (S, S) be a measurable space and let and be
two -finite measures on S. Then there exists a unique decomposition = a + s , where
66
1. a ,
2. s .
Furthermore, there exists an a.e.-unique function f L0+ such that
Z
a (A) =
f d.
A
P ROOF (*)
- Uniqueness. Suppose that a1 + s1 = = a2 + s2 are two decompositions satisfying (1) and (2)
in the statement. Let D1 and D2 be as in the definition of mutual singularity applied to the pairs
, s1 and , s2 , respectively. Set D = D1 D2 , and note that (D) = 0 and s1 (Dc ) = s2 (Dc ) = 0.
For any A S, we have (A D) = 0 and so, thanks to absolute continuity,
a1 (A D) = a2 (A D) = 0 and, consequently, s1 (A D) = s2 (A D) = (A D).
By singularity,
s1 (A Dc ) = s2 (A Dc ) = 0, and, consequently, a1 (A Dc ) = a2 (A Dc ) = (A Dc ).
Finally,
a1 (A) = a1 (A D) + a1 (A Dc ) = a2 (A D) + a2 (A Dc ) = 2 (A),
and, similarly, s1 = s2 .
R
To establish the uniqueness of the function f with the property that a (A) = A f d for all
A S, we assume that there are two such functions, f1 and f2 , say. Define the sequence {Bn }nN
by
Bn = {f1 f2 } Cn ,
where {Cn }nN is a pairwise-disjoint sequence in S with the property that (Cn ) < , for all
n N and n Cn = S. Then, with gn = f1 1Bn f2 1Bn L1+ we have
Z
Z
Z
f2 d = a (Bn ) a (Bn ) = 0.
f1 d
gn d =
Bn
Bn
By Problem 3.29, we have gn = 0, a.e., i.e., f1 = f2 , a.e., on Bn , for all n N, and so f1 = f2 , a.e.,
on {f1 f2 }. A similar argument can be used to show that f1 = f2 , a.e., on {f1 < f2 }, as well.
- Existence. By decomposing S into a countable measurable partition whose elements have

finite and measures, we may (and do) assume that both and
R are finite. Let R denote the
set of all functions f L0 ([0, ]) with the property that (A) A f d for all A S. The reader
will have no difficulty showing that
R
R
R
1. if f1 , f2 R, then f = max(f1 , f2 ) R (Hint: A f d = A{f1 f2 } f1 d + A{f1 <f2 } f2 d.)
2. if {fn }nN R and fn f , then f R.
Let {fn }nN be a sequence in R such that

Z
o
nZ
f d : f R .
fn d sup
67

Thanks to (1) above, we can replace fn by max(f1 , . . . , fn ), n N, and, thus, assume that the
sequence {fn (x)}nN is non-decreasing for each x X. Part (2), on the other hand, implies that
for f = limn fn , we have f R so that
Z
Z
(5.4)
f d g d for all g R.
In words,R f is a maximal candidate-Radon-Nikodym derivative. We define the measure a by
a (A) = A f d, so that a . Set s = a and note that s takes values in [0, ] thanks to
the fact that f R.
In order to show that s , we define a sequence {n }nN of set functions n : S (, ]
by n = s n1 . Thanks to the assumption (S) < , n is a well-defined signed measure, for
each n N. Therefore, according to the Hahn-Jordan decomposition (see Theorem 2.32), there
exists a sequence {Dn }nN in S such that
(5.5)
s (A Dn ) n1 (A Dn ) and s (A Dnc ) n1 (A Dnc ),
for all A S and n N. Moreover, with D = n Dn , we have

s (Dc ) s (Dnc ) n1 (Dnc ) n1 (S),
for all n N. Consequently, s (Dc ) = 0, so it will suffice to show that (D) = 0. For that, we
define a sequence {fn }nN in L0+ by fn = f + n1 1Dn and note that
Z
fn d = a (A) + n1 (Dn A) a (A) + s (Dn A) (A).
A
Therefore, fn R and thanks to the maximal property (5.4) of f , we conclude that fn = f , a.e.,
i.e. (Dn ) = 0, and, immediately, (D) = 0, as required.
Corollary 5.18 (Radon-Nikodym) Let and be -finite measures on (S, S) with . Then
there exists f L0+ such that
Z
(A) =
f d, for all A S.
(5.6)
A
For any other g L0+ with the same property, we have f = g, a.e.
Any function f for which (5.6) holds is called the Radon-Nikodym derivative of with respect
d
d
to and is denoted by f = d
, a.e. The Radon-Nikodym derivative f = d
is defined only
up to a.e.-equivalence, and there is no canonical way of picking a representative defined for all
x S. For that reason, we usually say that a function f L0+ is a version of the Radon-Nikodym
derivative of with respect to if (5.6) holds. Moreover, to stress the fact that we are talking about
a whole class of functions instead of just one, we usually write
d
d
L0+ and not
L0+ .
d
d
68

We often neglect this fact notationally, and write statements such as If f L0+ and f = d
d then
. . . . What we really mean is that the statement holds regardless of the particular representative f
d
d
of the Radon-Nikodym derivative we choose. Also, when we write d
= d
, we mean that they
are equal as elements of L0+ , i.e., that there exists f L0+ , which is both a version of
d
.
version of d
d
d
and a
Problem 5.19 Let , and be -finite measures on (S, S). Show that
1. If and , then + and
d
d
d( + )
+
=
.
d d
d
2. If and f L0+ , then
Z
f d =
g d where g = f
d
.
d
3. If and , then and

d
d d
=
.
d d
d
(Note: Make sure to pay attention to the fact that different measure give rise to different
families of null sets, and, hence, to different notions of almost everywhere.)
4. If , then
d
=
d
d
d
1
Problem 5.20 Let 1 , 2 , 1 , 2 be -finite measures with 1 and 1 , as well as 2 and 2 , defined
on the same measurable space. If 1 1 and 2 2 , show that 1 2 1 2 .
Example 5.21 Just like in the statement of Fubinis theorem, the assumption of -finiteness cannot
be omitted. Indeed, take (S, S) = ([0, 1], B([0, 1])) and consider the Lebesgue measure and
the
R
0
counting measure on (S, S). Clearly, , but there is no f L+ such that (A) = A f d.
Indeed, suppose that such f exists and set Dn = {x S : f (x) > 1/n}, for n N, so that
Dn {f > 0} = {f 6= 0}. Then
Z
Z
1
1
1 (Dn ) =
f d
n d = n #Dn ,
Dn
Dn
and so #Dn n. Consequently, the set {f > 0} = n Dn is countable. This leads to a contradiction
since the Lebesgue measure does not charge countable sets, and so
Z
Z
1 = ([0, 1]) = f d =
f d = ({f > 0}) = 0.
{f >0}
69
5.3
Additional Problems
Problem 5.22 (Area under the graph of a function) For f R L0+ , let H = {(x, r) S [0, ) :
f (x) r} be the region under the graph of f . Show that f d = ( )(H).
R
(Note: This equality is consistent with our intuition that the value of the integral f d corresponds to the area of the region under the graph of f .)
Problem 5.23 (A layered representation) Let be a measure on B([0, )) such that N (u) = ([0, u)) <
, for all u R. Let (S, S, ) be a -finite measure space. For f L0+ (S), show that
R
R
1. N f d = [0,) ({f > u}) (du).
R
R
2. for p > 0, we have f p d = p [0,) up1 ({f > u}) (du), where is the Lebesgue measure.
Problem 5.24 (A useful integral)

R
1. Show that 0 sinx x dx = . (Hint: Find a function below sinx x which is easier to integrate.)
(
exy sin(x), 0 x a, 0 y,
2. For a > 0, let f : R2 R be given by f (x, y) =
0,
otherwise.
1
2
2
Show that f L (R , B(R ), ), where denotes the Lebesgue measure on R2 .
Ra
R yeay
R eay
3. Establish the equality 0 sinx x dx = 2 cos(a) 0 1+y
dy.
2 dy sin(a) 0
1+y 2
R

R a sin(x)
a
2

4. Conclude that for a > 0, 0 sin(x)

dx
x
2 a , so that lima 0
x dx = 2 .
Problem 5.25 (The Cantor measure) Let ({1, 1}N , B({1, 1}N ), C ) be the coin-toss space. Define the mapping f : {1, 1}N [0, 1] by
X
f (s) =
(1 + sn )3n , for s = (s1 , s2 , . . . ).
nN
Let be the push-forward of C by the map f . It is called the Cantor measure.

1. Let d be the metric on {1, 1}N (as given by the equation (1.4) in Lecture 1). Show that for
= log3 (2) and s1 , s2 {1, 1}N , we have

d(s1 , s2 ) f (s2 ) f (s1 ) 3d(s1 , s2 ) .
2. Show that is atom-free, i.e., that ({x}) = 0, for all x [0, 1],
3. For a measure on the -algebra of Borel sets of a topological space X, the support of is
collection of all x X with the property that (O) > 0 for each open set O with x O.
Describe the support of . (Hint: Guess what it is and prove that your guess is correct. Use
the result in (1).)
4. Prove that .
(Note: The Cantor measure is an example of a singular measure. It has no atoms, but is still
singular with respect to the Lebesgue measure.)
70

Problem 5.26 (Joint measurability)
1. Give an example of a function f : [0, 1] [0, 1] [0, 1] such that x 7 f (x, y) and y 7 f (x, y)
are B([0, 1])-measurable functions for each y [0, 1] and x [0, 1], respectively, but that f is
not B([0, 1] [0, 1])-measurable.
(Hint: You can use the fact that there exists a subset of [0, 1] which is not Borel measurable.)
2. Let (S, S) be a measurable space. A function f : SR R is called a Caratheodory function

if
x 7 f (x, y) is S-measurable for each y R, and
y 7 f (x, y) is continuous for each x R.
Show that Caratheodory functions are S B(R)-measurable.

(Hint: Express a Caratheodory
P
function as limit of a sequence of the form fn = kZ gn,k (x)hn,k (r), n N.)
71
Chapter
Basic Notions of Probability

6.1
Probability spaces
A mathematical setup behind a probabilistic model consists of a sample space , a family of

events and a probability P. One thinks of as being the set of all possible outcomes of a given
random phenomenon, and the occurrence of a particular elementary outcome as depending on factors not fully known to the modeler. The family F is taken to be some collection of
subsets of , and for each A F, the number P[A] is interpreted as the likelihood that some A
occurs. Using the basic intuition that P[A B] = P[A] + P[B], whenever A and B are disjoint
(mutually exclusive) events, we conclude P should have all the properties of a finitely-additive
measure. Moreover, a natural choice of normalization dictates that the likelihood of the certain
event be equal to 1. For reasons that are outside the scope of these notes, a leap of faith is often made and P is required to be -additive. All in all, we can single out probability spaces as a
sub-class of measure spaces:
Definition 6.1 (Probability space) A probability space is a triple (, F, P), where is a nonempty set, F is a -algebra on and P is a probability measure on F.
In many (but certainly not all) aspects, probability theory is a part of measure theory. For
historical reasons and because of a different interpretation, some of the terminology/notation
changes when one talks about measure-theoretic concepts in probability. Here is a list of what is
different, and what stays the same:
1. We will always assume - often without explicit mention - that a probability space (, F, P)
is given and fixed.
2. Continuity of measure is called continuity of probability, and, unlike the general case, does
not require and additional assumptions in the case of a decreasing sequence (that is, of
course, because P[] = 1 < .)
3. A measurable function f : R is called a random variable. Typically, the sample space
is too large and clumsy for analysis, so we often focus our attention to real-valued functions X on (random variables are usually denoted by capital letters such as X, Y Z, etc.).
72
CHAPTER 6. BASIC NOTIONS OF PROBABILITY

If contains the information about the state of all parts of the model, X() will typically correspond to a single aspect of it. Therefore X 1 ([a, b]) is the set of all elementary
outcomes with for which X() [a, b]. If we want to be able to compute the probability P[X 1 ([a, b])], the set X 1 ([a, b]) better be an event, i.e., X 1 ([a, b]) F. Hence the
measurability requirement.
Sometimes, it will be more convenient for random variables to take values in the extended
of real numbers. In that case we talk about extended random variables or R-valued
set R
random variables.
etc. to denote the set of all random
4. We use the measure-theoretic notation L0 ,L0+ , L0 (R),
variables, non-negative random variables, extended random variables, etc.
5. Let (S, S) be a measurable space. An (F, S)-measurable map X : S is called a random
element (of S).
Random variables are random elements, but there are other important examples. IfQ(S, S) =
(Rn , B(Rn )), we talk about random vectors. More generally, if S = RN and S = n B(R),
the map X : S is called a discrete-time stochastic process. Sometimes, the object of
interest is a set (the area covered by a wildfire, e.g.) and then S is a collection of subsets of
Rn . There are many more examples.
6. The class of null-sets in F still plays the same role as it did in measure theory, but now we
use the acronym a.s. (which stands for almost surely) instead of the measure-theoretic a.e.
7. The Lebesgue integral with respect to the probability P is now called expectation and is
denoted by E, so that we write
Z
Z
E[X] instead of
X dP, or
X() P[d].
For p [1, ], the Lp spaces are defined just like before, and have the property that Lq Lp ,
when p q.
8. The notion of a.e.-convergence is now re-baptized as a.s. convergence, while convergence
a.s.
in measure is now called convergence in probability. We write Xn X if the sequence
P
{Xn }nN of random variables converges to a random variable X, a.s. Similarly, Xn X

refers to convergence in probability. The notion of convergence in Lp , for p [1, ] is
Lp
exactly the same as before. We write Xn X if {Xn }nN converges to X in Lp .
9. Since the constant random variable X() = 1, for is integrable, a special case of the
dominated convergence theorem, known as the bounded convergence theorem holds in
probability spaces:
Theorem 6.2 (Bounded convergence) Let {Xn }nN be a sequence of random variables such
that there exists M 0 such that |Xn | M , a.s., and Xn X, a.s., then
E[Xn ] E[X].
73

10. The relationship between various forms of convergence can now be represented diagramatically as
LDQQQ
DD QQQ
yy
DD QQQ
yy
DD
QQQ
y
y
D"
QQQ
|yy
( q
p
o
a.s.E
L
mL
m
EE
m
z
z
mm
EE
zz mmm
EE
EE zzmzmmmmm
" }zvmm
where 1 p q < and an arrow A B means that A implies B, but that B does not
imply A in general.
6.2
Distributions of random variables, vectors and elements
As we have already mentioned, typically too big to be of direct use. Luckily, if we are only
interested in a single random variable, all the useful probabilistic information about it is contained
in the probabilities of the form P[X B], for B B(R). Btw, it is standard to write P[X B]
instead of the more precise P[{X B}] or P[{ : X() B}]. Similarly, we will write
P[Xn Bn , i.o] instead of P[{Xn Bn } i.o.] and P[Xn Bn , ev.] instead of P[{Xn Bn } ev.]
The map B 7 P[X B] is, however, nothing but the push-forward of the measure P by the
map X onto B(R):
Definition 6.3 (Distribution of a random variable) The distribution of the random variable X
is the probability measure X on B(R), defined by
(B) = P[X 1 (B)],
that is the push-forward of the measure P by the map X.
In addition to be able to recover the information about various probabilities related to X from
X , one can evaluate any possible integral involving a function of X by integrating that function
against X (compare the statement to Problem 5.23):
Problem 6.4 Let g : R R be a Borel function. Then g X L01 (, F, P) if and only if g
L01 (R, B(R), X ) and, in that case,
Z
E[g(X)] = g dX .
In particular,
E[X] =
xX (dx).
R
Taken in isolation from everything else, two random variables X and Y for which X = Y are the
same from the probabilistic point of view. In that case we say that X and Y are equally distributed
(d)
random variables and write X = Y . On the other hand, if we are interested in their relationship
with a third random variable Z, it can happen that X and Y have the same distribution, but that
74

their relationship to Z is very different. It is the notion of joint distribution that comes to rescue.
For a random vector X = (X1 , . . . , Xn ), the measure X on B(Rn ) given by
X (B) = P[X B],
is called the distribution of the random vector X. Clearly, the measure X contains the information about the distributions of the individual components X1 , . . . , Xn , because
X1 (A) = P[X1 A] = P[X1 A, X2 R, . . . , Xn R] = X (A R R).
When X1 , . . . , Xn are viewed as components in the random vector X, their distributions are sometimes referred to as marginal distributions.
Example 6.5 Let = {1, 2, 3, 4}, F = 2 , with P characterized by P[{}] = 41 , for = 1, . . . , 4.
The map X : R, given by X(1) = X(3) = 0, X(2) = X(4) = 1, is a random variable and its
distribution is the measure 21 0 + 21 1 on B(R) (check that formally!), where a denotes the Dirac
measure on B(R), concentrated on {a}.
Similarly, the maps Y : R and Z : R, given by Y (1) = Y (2) = 0, Y (3) = Y (4) = 1,
and Z() = 1 X() are random variables with the same distribution as X. The joint distributions of the random vectors (X, Y ) and (X, Z) are very different, though. The pair (X, Y ) takes 4
different values (0, 0), (0, 1), (1, 0), (1, 1), each with probability 41 , so that the distribution of (X, Y )
is given by

(X,Y ) =
1
4
(0,0) + (0,1) + (1,0) + (1,1) .
On the other hand, it is impossible for X and Z to take the same value at the same time. In
fact, there are only two values that the pair (X, Z) can take - (0, 1) and (1, 0). They happen with
probability 12 each, so

(X,Z) =
1
2
(0,1) + (1,0) .
We will see later that the difference between (X, Y ) and (X, Z) is best understood if we analyze
the way the component random variables depend on each other. In the first case, even if the value
of X is revealed, Y can still take the values 0 or 1 with equal probabilities. In the second case, as
soon as we know X, we know Z.
More generally, if X : S, is a random element with values in the measurable space (S, S),
the distribution of X is the measure X on S, defined by X (B) = P[X B] = P[X 1 (B)], for
B S.
Sometimes it is easier to work with a real-valued function FX defined by
FX (x) = P[X x],
which we call the (cumulative) distribution function (cdf for short), of the random variable X.
The following properties of FX are easily derived by using continuity of probability from above
and from below:
Proposition 6.6 (Preperties of the cdf) Let X be a random variable, and let FX be its distribution
function. Then,
1. FX is non-decreasing and takes values in [0, 1],
75
2. FX is right continuous,
3. limx FX (x) = 1 and limx FX (x) = 0.
Remark 6.7 A notion of a (cumulative) distribution function can be defined for random vectors,
too, but it is not used as often as the single-component case, so we do not write about it here.
The case when X is absolutely continuous with respect to the Lebesgue measure is especially
important:
Definition 6.8 (Absolute continuity and pdfs) A random variable X with the property that
X , where is the Lebesgue measure on B(R), is said to be absolutely continuous. In that case,
X
any Radon-Nikodym derivative d
d is called the probability density function (pdf) of X, and is
denoted by fX . Similarly, a random vector X = (X1 , . . . , Xn ) is said to be absolutely continuous if
X
X , where is the Lebesgue measure on B(Rn ), and the Radon-Nikodym derivative d
d , denoted
by fX is called the probability density function (pdf) of X.
Problem 6.9
1. Let X = (X1 , . . . , Xn ) be an absolutely-continuous random vector. Show that Xk is also
absolutely continuous, and that its pdf is given by
Z
Z
f (1 , . . . , k1 , x, k+1 , . . . , n ) d1 . . . dk1 dk+1 . . . dn .
fXk (x) =
...
R
R
| {z }
n 1 integrals
(Note: Note the fXk (x) is defined only for almost all x R; that is because fX is defined only
up to null sets in B(Rn ).)
2. Let X be an absolutely-continuous random variable. Show that the random vector (X, X)
is not absolutely continuous, even though both of its components are .
Problem 6.10 Let X = (X1 , . . . , Xn ) be an absolutely-continuous random vector with density
fX , and let g : Rn R be a Borel-measurable function with gfX L01 (Rn , B(Rn ), ). Show that
g(X) L01 (, F, P) and that
Z
Z
Z
E[g(X)] = gfX d =
. . . g(1 , . . . , n )fX (1 , . . . , n ) d1 . . . dn .
R
Definition 6.11 (Discrete random variables) A random variable X is said to be discrete if there
exists a countable set B B(R) such that X (B) = 1.
76

Problem 6.12 Show that a sum of two discrete random variables is discrete, but that a sum of two
absolutely-continuous random variables does not need to be absolutely continuous.
Definition 6.13 (Singular distributions) A distribution which has no atoms and is singular with
respect to the Lebesgue measure is called singular.
Example 6.14 (A measure which is neither absolutely continuous nor discrete) By Problem 5.25,
there exists a measure on [0, 1], with the following properties
1. has no atoms, i.e., ({x}) = 0, for all x
[0, 1],
1.0
2. and (the Lebesgue measure) are mutually singular
0.8
3. is supported by the Cantor set.
0.6
We set (, F, P) = ([0, 1], B([0, 1]), ), and define

the random variable X : R, by X() = .
It is clear that the distribution X of X has the
property that
X (B) = (B [0, 1]),
0.4
0.2
-0.5
0.5
1.0
1.5
Cdf of the Cantor distribution.
so that X is an example of a random variable

with a singular distribution.
6.3
Independence
The point at which probability departs from measure theory is when independence is introduced.
As seen in Example 6.5, two random variables can depend on each other in different ways. One
extreme (the case of X and Y ) corresponds to the case when the dependence is very weak - the
distribution of Y stays the same when the value of X is revealed:
Definition 6.15 (Independence of two random variables) Two random variables X and Y are
said to be independent if
P[{X A} {Y B}] = P[X A] P[Y B] for all A, B B(R).
It turns out that independence of random variables is a special case of the more-general notion
of independence between families of sets.
77
Definition 6.16 (Independence of families of sets) Families A1 , . . . , An of elements in F are

said to be
1. independent if
(6.1)
P[Ai1 Ai2 Aik ] = P[Ai1 ] P[Ai2 ] P[Aik ],
for all k = 1, . . . , n, 1 i1 < i2 < < ik n, and all Ail Ail , l = 1, . . . , k,

2. pairwise independent if
P[Ai1 Ai2 ] = P[Ai1 ] P[Ai2 ],
for all 1 i1 < i2 n, and all Ai1 Ai1 , Ai2 Ai2 .
Problem 6.17
1. Show, by means of an example, that the notion of independence would change if we asked
for the product condition (6.1) to hold only for k = n and i1 = 1, . . . , ik = n.
2. Show that, however, if Ai , for all i = 1, . . . , n, it is enough to test (6.1) for k = n and
i1 = 1, . . . , ik = n to conclude independence of Ai , i = 1, . . . , n.
Problem 6.18 Show that random variables X and Y are independent if and only if the -algebras
(X) and (Y ) are independent.
Definition 6.19 (Independence of random variables and events) Random variables X1 ,. . . , Xn

are said to be independent if the -algebras (X1 ), . . . , (Xn ) are independent.
Events A1 , . . . , An are called independent if the families Ai = {Ai }, i = 1, . . . , n, are independent.
When only two families of sets are compared, there is no difference between pairwise independence and independence. For 3 or more, the difference is non-trivial:
Example 6.20 (Pairwise independence without independence) Let X1 , X2 and X3 be independent random variables, each with the coin-toss distribution, i.e., P[Xi = 1] = P[Xi = 1] = 21 ,
for i = 1, 2, 3. It is not hard to construct a probability space where such random variables may be
defined explicitly: let = {1, 2, 3, 4, 5, 6, 7, 8}, F = 2 , and let P be characterized by P[{}] = 81 ,
for all . Define
(
1,
i
Xi () =
1, otherwise
where 1 = {1, 3, 5, 7}, 2 = {2, 3, 6, 7} and 3 = {5, 6, 7, 8}. It is easy to check that X1 , X2 and
X3 are independent (Xi is the i-th bit in the binary representation of ).
78

With X1 , X2 and X3 defined, we set
Y1 = X2 X3 , Y2 = X1 X3 and Y3 = X1 X2 ,
so that Yi has a coin-toss distribution, for each i = 1, 2, 3. Let us show that Y1 and Y2 (and then, by
symmetry, Y1 and Y3 , as well as Y2 and Y2 ) are independent:
P[Y1 = 1, Y2 = 1] = P[X2 = X3 , X1 = X3 ] = P[X1 = X2 = X3 ]
= P[X1 = X2 = X3 = 1] + P[X1 = X2 = X3 = 1] =
1
8
1
8
1
4
= P[Y1 = 1] P[Y2 = 1].
We dont need to check the other possibilities, such as Y1 = 1, Y2 = 1, to conclude that Y1 and Y2
are independent (see Problem 6.21 below).
On the other hand, Y1 , Y2 and Y3 are not independent:
P[Y1 = 1, Y2 = 1, Y3 = 1] = P[X2 = X3 , X1 = X3 , X1 = X2 ] = P[X1 = X2 = X3 ]
=
1
4
6=
1
8
= P[Y1 = 1] P[Y2 = 1] P[Y3 = 1].
Problem 6.21 Show that if A1 , . . . , An are independent, then so are the families Ai = {Ai , Aci },
i = 1, . . . , n.
A more general statement is also true (and very useful):
Proposition 6.22 (Independent -systems imply independent generated -algebras) Let

Pi , i = 1, . . . , n be independent -systems. Then the -algebras (Pi ), i = 1, . . . , n are also
independent.
P ROOF Let F1 denote the set of all C F such that

P[C Ai2 Aik ] = P[C] P[Ai2 ] P[Aik ],
for all k = 2, . . . , n, 1 < i2 < < ik n, and all Ail Pil , l = 2, . . . , k. It is easy to see that Fi
is a -system which contains the -system Pi , and so, by the - Theorem, it also contains (P1 ).
Consequently (P1 ), P2 , . . . Pn are independent families.
A re-play of whole procedure with the families P2 , (P1 ), P3 , . . . Pn , yields that the families
(P1 ), (P2 ), P3 , . . . Pn are also independent. Following the same pattern allows us to conclude
after n steps that (P1 ), (P2 ), . . . (Pn ) are independent.
Remark 6.23 All notions of independence above extend to infinite families of objects (random
variables, families of sets) by requiring that every finite sub-family be independent.
The result of Proposition 6.22 can be used to help us check independence of random variables:
Problem 6.24 Let X1 , . . . , Xn be random variables.
1. Show that X1 , . . . , Xn are independent if and only if
X = X1 Xn ,
where X = (X1 , . . . , Xn ).
79

2. Show that X1 , . . . , Xn are independent if and only if
P[X1 x1 , . . . , Xn xn ] = P[X1 x1 ] P[Xn xn ],
for all x1 , . . . , xn R. (Hint: Note that the family {{Xi x} : x R} does not include , so
that part (2) of Problem 6.17 cannot be applied directly.)
3. Suppose that the random vector X = (X1 , . . . , Xn ) is absolutely continuous. Then X1 , . . . , Xn
are independent if and only if
fX (x1 , . . . , xn ) = fX1 (x1 ) fXn (xn ), -a.e.,
where denotes the Lebesgue measure on B(Rn ).
4. Suppose that X1 , . . . Xn are discrete with P[Xk Ck ] = 1, for countable subsets C1 , . . . , Ck
of R. Show that X1 , . . . , Xn are independent if and only if
P[X1 = x1 , . . . , Xn = xn ] = P[X1 = x1 ] P[Xn = xn ],
for all xi Ci , i = 1, . . . , n.
Problem 6.25 Let X1 , . . . , Xn be independent random variables. Show that the random vector
X = (X1 , . . . , Xn ) is absolutely continuous if and only if each Xi , i = 1, . . . , n is an absolutelycontinuous random variable.
The usefulness of Proposition 6.22 is not exhausted, yet.
Problem 6.26
1. Let Fij i = 1, . . . , n, j = 1, . . . , mi , be an independent collection of -algebras on . Show
that the -algebras G1 , . . . , Gn , where Gi = (Fi1 , . . . , Fimi ), are independent.
(Hint: j=1,...,mi Fij is a -system which generates Gi .)
2. Let Xij i = 1, . . . , n, j = 1, . . . , mi , be an independent random variables, and let fi : Rmi

R, i = 1, . . . , n, be Borel functions. Then the random variables Yi = fi (X1 , . . . , Xmi ), i =
1, . . . , n are independent.
Problem 6.27
1. Let X1 , . . . , Xn be random variables. Show that X1 , . . . , Xn are independent if and only if
n
Y
E[fi (Xi )] = E[
i=1
n
Y
fi (Xi )],
i=1
for all n-tuples (f1 , . . . , fn ) of bounded continuous real functions. (Hint: Approximate!)
2. Let {Xni }nN , i = 1, . . . , m be sequences of random variables such that Xn1 , . . . , Xnm are indea.s.
pendent for each n N. If Xni X i , i = 1, . . . , m, for some X 1 , . . . , X m L0 , show that
X 1 , . . . , X m are independent.
The idea independent means multiply applies not only to probabilities, but also to random
variables:
80
Proposition 6.28 (Expectation of a function of independent components) Let X, Y be independent random variables, and let h : R2 [0, ) be a measurable function. Then

Z Z
h(x, y) X (dx) Y (dy).
E[h(X, Y )] =
R
P ROOF By independence and part (1) of Problem 6.24, the distribution of the random vector
(X, Y ) is given by X Y , where X is the distribution of X and Y is the distribution of Y .
Using Fubinis theorem, we get

Z
Z Z
E[h(X, Y )] = h d(X,Y ) =
h(x, y) X (dx) Y (dy).
R
Proposition 6.29 (Independent means multiply) Let X1 , X2 , . . . , Xn be independent random

variables with Xi L1 , for i = 1, . . . , n. Then
Q
1. ni=1 Xi = X1 Xn L1 , and
2. E[X1 Xn ] = E[X1 ] . . . E[Xn ].
The product formula 2. remains true if we assume that Xi L0+ (instead of L1 ), for i = 1, . . . , n.
P ROOF Using the fact that X1 and X2 Xn are independent random variables (use part (2) of
Problem 6.26), we can assume without loss of generality that n = 2.
Focusing first on the case X1 , X2 L0+ , we apply Proposition 6.28 with h(x, y) = xy to conclude that

Z Z
E[X1 X2 ] =
x1 x2 X1 (dx1 ) X2 (dx2 )
R
ZR
=
x2 E[X1 ] X2 (dx2 ) = E[X1 ]E[X2 ].
R
For the case X1 , X2 L1 , we split X1 X2 = X1+ X2+ X1+ X2 X1 X2+ + X1 X2 and apply the
above conclusion to the 4 pairs X1+ X2+ , X1+ X2 , X1 X2+ and X1 X2 .
Problem 6.30 (Conditions for independent-means-multiply) Proposition 6.29 in states that for
independent X and Y , we have
(6.2)
E[XY ] = E[X]E[Y ],
whenever X, Y L1 or X, Y L0+ . Give an example which shows that (6.2) is no longer necessarily true if X L0+ and Y L1 . (Hint: Build your example so that E[(XY )+ ] = E[(XY ) ] = .
Use ([0, 1], B([0, 1]), ) and take Y () = 1[0,1/2] () 1(1/2,0] (). Then show that any random variable X with the property that X() = X(1 ) is independent of Y . )
81

Problem 6.31 Two random variables X, Y are said to be uncorrelated, if X, Y L2 and Cov(X, Y ) =
0, where Cov(X, Y ) = E[(X E[X])(Y E[Y ])].
1. Show that for X, Y L2 , the expression for Cov(X, Y ) is well defined.
2. Show that independent random variables in L2 are uncorrelated.
3. Show that there exist uncorrelated random variables which are not independent.
6.4
Sums of independent random variables and convolution
Proposition 6.32 (Convolution as the distribution of a sum) Let X and Y be independent random variables, and let Z = X + Y be their sum. Then the distribution Z of Z has the following
representation:
Z
Z (B) =
X (B y) Y (dy), for B B(R),
where B y = {b y : b B}.
P ROOF We can view Z as a function f (x, y) = x + y applied to the random vector (X, Y ), and
so, we have E[g(Z)] = E[h(X, Y )], where h(x, y) = g(x + y). In particular, for g(z) = 1B (z),
Proposition 6.28 implies that
Z Z
Z (B) = E[g(Z)] =
1{x+yB} X (dx) Y (dy)
R R

Z Z
Z
=
1{xBy} X (dx) Y (dy) =
X (B y) Y (dy).
R
One often sees the expression
f (x) dF (x),
R
R
as notation for the integral f d, where F (x) = ((, x)]. The reason for this is that such
integrals - called the Lebesgue-Stieltjes integrals - have a theory parallel to that of the Riemann
integral and the correspondence between dF (x) and d is parallel to the correspondence between
dx and d.
Corollary 6.33 (Cdf of a sum as a convolution) Let X, Y be independent random variables, and
let Z be their sum. Then
Z
FZ () =
FX ( y) dFY (y).
R
82
Definition 6.34 (Convolution of probability measures) Let 1 and 2 be two probability measures on B(R). The convolution of 1 and 2 is the probability measure 1 2 on B(R), given
by
Z
(1 2 )(B) =
1 (B ) 2 (d), for B B(R),
where B = {x : x B} B(R).
Problem 6.35 Show that the is a commutative and associative operation on the set of all probability measures on B(R). (Hint: Use Proposition 6.32).
It is interesting
to see how convolution mixes
R with absolute continuity. To simplify the notation,
R
we write A f (x) dx instead of more precise A f (x) (dx) for the (Lebesgue) integral with respect
R
we write b f (x) dx.
to the Lebesgue measure on R. When A = [a, b] R,
a
Proposition 6.36 (Convolution inherets absolute continuity from either component) Let X
and Y be independent random variables, and suppose that X is absolutely continuous. Then their
sum Z = X + Y is also absolutely continuous and its density fZ is given by
Z
fZ (z) =
fX (z y) Y (dy).
R
R
P ROOF Define f (z) = R fX (z y) Y (dy), for some density fX of X (remember, the density
function is defined only -a.e.). The function f is measurable (why?) so it will be enough (why?)
to show that
h
i Z
P Z [a, b] =
f (z) dz, for all < a < b < .
[a,b]
We start with the right-hand side of (6.3) and use Fubinis theorem to get
Z

Z
Z
f (z) dz =
1[a,b] (z)
fX (z y) Y (dy) dz
[a,b]
R
R
(6.3)

Z Z
=
1[a,b] (z)fX (z y) dz Y (dy)
R
By the translation-invariance property of the Lebesgue measure, we have

Z
Z
h
i

1[a,b] (z)fX (z y) dz =
1[ay,by] (z)fX (z) dz = P Z [a y, b y] = X [a, b] y .
R
Therefore, by (6.3) and Proposition 6.32, we have

Z
Z

h
i
f (z) dz =
X [a, b] y Y (dy) = Z [a, b] = P Z [a, b] .
[a,b]
83
Definition 6.37 (Convolution of functions in L1 ) The convolution of functions f and g in

L1 (R) is the function f g L1 (R) given by
Z
(f g)(z) =
f (z x)g(x) dx.
R
Problem 6.38
1. Use the reasoning from the proof of Proposition 6.36 to show that the convolution is welldefined operation on L1 (R).
2. Show that if X and Y are independent absolutely-continuous random variables, then X + Y
is also absolutely continuous with density which is the convolution of densities of X and Y .
6.5
Do independent random variables exist?
We leave the most basic of the questions about independence for last: do independent random
variable exist? We need a definition and two auxiliary results, first.
Definition 6.39 (Uniform distribution on (a, b)) A random variable X is said to be uniformly
distributed on (a, b), for a < b R, if it is absolutely continuous with density
fX (x) =
1
ba 1(a,b) (x).
Our first result states a uniform random variable on (0, 1) can be transformed deterministically
into any a random variable of prescribed distribution.
Proposition 6.40 (Transforming a uniform distribution) Let be a measure on B(R) with

(R) = 1. Then, there exists a function H : (0, 1) R such that the distribution of the random
variable X = H (U ) is , whenever U is a uniform random variable on (0, 1).
P ROOF Let F be the cdf corresponding to , i.e.,

F (x) = ((, x]).
The function F is non-decreasing, so it almost has an inverse: define
H (y) = inf{x R : F (x) y}.
Since limx F (x) = 1 and limx F (x) = 0, H (y) is well-defined and finite for all y (0, 1).
Moreover, thanks to right-continuity and non-decrease of F , we have
H (y) x y F (x), for all x R, y (0, 1).
84

Therefore
P[H (U ) x] = P[U F (x)] = F (x), for all x R,
and the statement of the Proposition follows.
Remark 6.41 Proposition 6.40 is a basis for a technique used to simulate random variables. There
are efficient algorithms for producing simulated values which resemble the uniform distribution
in (0, 1) (so-called pseudo-random numbers). If a simulated value drawn with distribution is
needed, one can simply apply the function H to a pseudo-random number.
Our next auxiliary result tells us how to construct a sequence of independent uniforms:
Proposition 6.42 (A large-enough probability space exists) There exists a probability space
(, F, P), and on it a sequence {Xn }nN of random variables such that
1. Xn has the uniform distribution on (0, 1), for each n N, and
2. the sequence {Xn }nN is independent.
P ROOF Set (, F, P) = ({1, 1}N , S, C ) - the coin-toss space with the product -algebra and the
coin-toss measure. Let a : N N N be a bijection, i.e., (aij )i,jN is an arrangement of all natural
numbers into a double array. For i, j N, we define the map ij : {1, 1}, by
ij (s) = saij ,
i.e., ij is the natural projection onto the aij -th coordinate. It is straightforward to show that, under
P, the collection (ij )i,jN is independent; indeed, it is enough to check the equality
P[i1 j1 = 1, . . . , in jn = 1] = P[i1 j1 = 1] P[in jn = 1],
for all n N and all different (i1 , j1 ), . . . , (in , jn ) N N.
At this point, we use the construction from Section 2.3 of Lecture 2, to construct an independent copy of a uniformly-distributed random variable from each row of (ij )i,jN . We set
(6.4)
Xi =

X
1+
ij
j=1
2j , i N.
By second parts of Problems 6.26 and 6.27, we conclude that the sequence {Xi }iN is independent.
Moreover, thanks to (6.4), Xi is uniform on (0, 1), for each i N.
Proposition 6.43 (Arbitrary independent sequences exist) Let {n }nN be a sequence of probability measures on B(R). Then, there exists a probability space (, F, P), and a sequence {Xn }nN of
random variables defined there such that
1. Xn = n , and
2. {Xn }nN is independent.
85

P ROOF Start with the sequence of Proposition 6.42 and apply the function Hn to Xn for each
n N, where Hn is as in the proof of Proposition 6.40.
An important special case covered by Proposition 6.43 is the following:
Definition 6.44 (Independent and identically distributed sequences) A sequence {Xn }nN of
random variables is said to be independent and identically distributed (iid) if {Xn }nN is independent and all Xn have the same distribution.
Corollary 6.45 (Iid sequences exist) Given a probability measure on R, there exist a probability
space supporting an iid sequence {Xn }nN such that Xn = .
6.6
Additional Problems
Problem 6.46 (The standard normal distribution) An absolutely continuous random variable X
is said to have the standard normal distribution - denoted by X N (0, 1) - if it admits a density
of the form
1
f (x) = exp(x2 /2), x R
2
For a r.v. with such a distribution we write X N (0, 1).
R
R
1. Show that R f (x) dx = 1. (Hint: Consider the double integral R2 f (x)f (y) dx dy and pass
to polar coordinates. )
2. For X N (0, 1), show that E[|X|n ] < for all n N. Then compute the nth moment E[X n ],
for n N.
3. A random variable with the same distribution as X 2 , where X N (0, 1), is said to have the
2 -distribution. Find an explicit expression for the density of the 2 -distribution.
4. Let Y have the 2 -distribution. Show that there exists a constant 0 > 0 such that E[exp(Y )] <
for < 0 and E[exp(Y )] = + for 0 . (Note: For a random variable Y L0+ , the
quantity E[exp(Y )] is called the exponential moment of order .)
5. Let 0 > 0 be a fixed, but arbitrary constant. Find an example of a random variable X 0
with the property that E[X ] < for 0 and E[X ] = + for > 0 . (Hint: This is not
the same situation as in (4) - this time the critical case 0 is included in a different alternative.
Try X = exp(Y ), where P[Y N] = 1. )
Problem 6.47 (The memory-less property of the exponential distribution) A random variable
is said to have exponential distribution with parameter > 0 - denoted by X Exp() - if its
distribution function FX is given by
FX (x) = 0 for x < 0, and FX (x) = 1 exp(x), for x 0.
86

1. Compute E[X ], for (1, ). Combine your result with the result of part (3) of Problem
6.46 to show that
( 21 ) = ,
where is the Gamma function.
2. Remember that the conditional probability P[A|B] of A, given B, for A, B F, P[B] > 0 is
given by
P[A|B] = P[A B]/P[B].
Compute P[X x2 |X x1 ], for x2 > x1 > 0 and compare it to P[X (x2 x1 )].
(Note: This can be interpreted as follows: the knowledge that the bulb stayed functional until
x1 does not change the probability that it will not explode in the next x2 x1 units of time;
bulbs have no memory.)
Conversely, suppose that Y is a random variable with the property that P[Y > 0] = 1 and
P[Y > y] > 0 for all y > 0. Assume further that
(6.5)
P[Y y2 |Y y1 ] = P[Y y2 y1 ], for all y2 > y1 > 0.
Show that Y Exp() for some > 0.

(Hint: You can use the following fact: let : (0, ) R be a Borel-measurable function
such that (y) + (z) = (y + z) for all y, z > 0. Then there exists a constant R such that
(y) = y for all y > 0.)
Problem 6.48 (The second Borel-Cantelli Lemma)
1. Let {Xn }nN be a sequence in L0+ . Show that there exists a sequence of positive constants
{cn }nN with the property that
Xn
0, a.s.
cn
(Hint: Use the Borel-Cantelli lemma.)
P
2. (The first) Borel-Cantelli lemma states that nN P[An ] < implies P[An , i.o.] = 0. There
are simple examples showing that the converse does not hold in general. Show that it does
hold if the events {An }nN are assumed to be independent. More precisely, show that, for
an independent sequence {An }nN , we have
X
P[An ] = implies P[An , i.o.] = 1.
nN
This is often known as the second Borel-Cantelli lemma. (Hint: Use the inequality (1 x)
ex , x R.)
3. Let {Xn }nN be an iid (independent and identically distributed) sequence of coin tosses, i.e.,
independent random variables with P[Xn = T ] = P[Xn = H] = 1/2 for all n N (if you are
uncomfortable with T and H, feel free to replace them with 1 and 1). A tail-run of size k
is a finite sequence of at least k consecutive T s starting from some index n N. Show that
for almost every (i.e., almost surely) the sequence {Xn ()}nN will contain infinitely
many tail runs of size k. Conclude that, for almost every , the sequence Xn () will contain
infinitely many tail runs of every length.
87

4. Let {Xn }nN be an iid sequence in L0 . Show that
E[|X1 |] < if and only if P[|Xn | n, i.o.] = 0.
P
P
(Hint: Use (but first prove) the fact that 1 + n1 P[X n] E[X] n1 P[X n], for
X L0+ .)
88
Chapter
Weak Convergence and Characteristic

Functions
7.1
Weak convergence
In addition to the modes of convergence we introduced so far (a.s.-convergence, convergence in

probability and Lp -convergence), there is another one, called convergence in distribution. Unlike
the other three, whether a sequence of random variables (elements) converges in distribution or
not depends only on their distributions. In addition to its intrinsic mathematical interest, convergence in distribution (or, equivalently, the weak convergence) is precisely the kind of convergence
we encounter in the central limit theorem.
We take the abstraction level up a notch and consider sequences of probability measures on
(S, S), where (S, d) is a metric space and S = B(d) is the Borel -algebra there. In fact, it will
always be assumed that S is a metric space and S is the Borel -algebra on it, throughout this
chapter.
Definition 7.1 (Weak convergence of probability measures) Let {n }nN be a sequence of probability measures on (S, S). We say that n converges weakly to a probability measure on (S, S) w
and write n - if
Z
Z
f dn
f d,
for all f Cb (S), where Cb (S) denotes the set of all continuous and bounded functions f : S R.
Remark 7.2 It would be more in tune with standard mathematical terminology to use the term
weak- convergence instead of weak convergence. For historical reasons, however, we omit the .
Definition 7.3 (Convergence in distribution) A sequence {Xn }nN of random variables (eleD
ments) is said to converge in distribution to the random variable (element) X, denoted by Xn X,

w
if Xn X .
89
CHAPTER 7. WEAK CONVERGENCE AND CHARACTERISTIC FUNCTIONS

Our first result states that weak limits are unique and for that we need a simple result, first:
Problem 7.4 Let F be a closed set in S. Show that for any > 0 there exists a Lipschitz and
bounded function fF ; : S R such that
1. 0 fF ; (x) 1, for all x R,
2. fF ; (x) = 1 for x F , and
3. fF ; (x) = 0 for d(x, F ) , where d(x, F ) = inf{d(x, y) : y F }.
(Hint: Show first that the function x 7 d(x, F ) is Lipschitz. Then argue that fF ; (x) = h(d(x, F ))
has the required properties for a well-chosen function h : [0, ) [0, 1]. )
Proposition 7.5 (Weak limits are unique) Suppose that {n }nN is a sequence of probability meaw
w
sures on (S, S) such that n and n . Then = .
P ROOF By the very definition of weak convergence, we have
Z
Z
Z
(7.1)
f d = lim f dn = f d ,
n
for all f Cb (S). Let F be a closed set, and let {fk }kN be as in Problem 7.4, with fk = fF ;
corresponding to = 1/k. If we set Fk = {x S : d(x, F ) 1/k}, then Fk is a closed set (why?)
and we have 1F fk 1Fk . By (7.1), we have
Z
Z
(F ) fk d = fk d (Fk ),
and, similarly, (F ) (Fk ), for all k N. Since Fk F (why?), we have (Fk ) (F ) and
(Fk ) (F ), and it follows that (F ) = (F ).
It remains to note that the family of all closed sets is a -system which generates the -algebra
S to conclude that = .
We have seen in the proof of Proposition 7.5 that an operational characterization of weak convergence is needed. Here is a useful one. We start with a lemma; remember that A denotes the
topological boundary A = Cl A \ Int A of a set A S.
Problem 7.6 Let (F ) be a partition of S into (possibly uncountably many) measurable subsets.
Show that for any probability measure on S, (F ) = 0, for all but countably many (Hint:
For n N, define n = { : (F ) n1 }. Argue that n has at most n elements.)
Definition 7.7 (-continuity sets) A set A S with the property that (A) = 0, is called a continuity set
90
Theorem 7.8 (Portmanteau Theorem) Let , {n }nN be probability measures on S. Then, the
following are equivalent:
w
1. n ,
R
R
2. f dn f d, for all bounded, Lipschitz continuous f : S R,
3. lim supn n (F ) (F ), for all closed F S,

4. lim inf n n (G) (G), for all open G S,
5. limn n (A) = (A), for all -continuity sets A S.

Remark 7.9 Before a proof is given, here is a way to remember whether closed sets go together
with the lim inf or the lim sup: take a convergent sequence {xn }nN in S, with xn x. If n is
the Dirac measure concentrated on xn , and the Dirac measure concentrated on x, then clearly
R
R
w
n (since f dn = f (xn ) f (x) = f d). Let F be a closed set. It can happen that xn 6 F
for all n N, but x F (think of x on the boundary of F ). Then n (F ) = 0, but (F ) = 1 and so
lim sup n (F ) = 0 < 1 = (F ).
P ROOF (P ROOF OF T HEOREM 7.8)
(1) (2): trivial.
(2) (3): given a closed set F and let Fk = {x S : d(x, F ) 1/k}, fk = fF ;1/k , k N, be as
in the proof of Proposition 7.5. Since 1F fk 1Fk and the functions fk are Lipschitz continuous,
we have
Z
Z
Z
lim sup n (F ) = lim sup 1F dn lim fk dn = fk d (Fk ),
n
for all k N. Letting k - as the proof of Proposition 7.5 - yields (3).

(3) (4): follows directly by taking complements.
(4) (1): Pick f Cb (S) and (possibly after applying a linear transformation to it) assume
R
R1
that 0 < f (x), for all x S. Then, by Problem 5.23, we have f d = 0 (f > t) dt, for any
probability measure on B(R). The set {f > t} S is open, so by (3), lim inf n n (f > t) (f > t),
for all t. Therefore, by Fatous lemma,
Z 1
Z 1
Z 1
Z
(f > t) dt
lim inf n (f > t) dt
n (f > t) dt
lim inf f dn = lim inf
n
n
n
0
0
0
Z
= f d.
We get the other inequality by f .
f d lim supn
f dn , by repeating the procedure with f replaced
(3), (4) (5): Let A be a -continuity set, let Int A be its interior and Cl A its closure. Then,
since Int A is open and Cl A is closed, we have
(Int A) lim inf n (Int A) lim inf n (A) lim sup n (A)
n
lim sup n (Cl A) (Cl A).

n
91

Since 0 = (A) = (Cl A \ Int A) = (Cl A) (Int A), we conclude that all inequalities above
are, in fact, equalities so that (A) = limn n (A)
(5) (3): For x S, consider the family {BF (r) : r 0}, where
BF (r) = {x S : d(x, F ) r},
of closed sets.
Claim: There exists a countable subset R of [0, ) such that BF (r) is a -continuity set for all
r [0, ) \ R.
P ROOF For r 0 define CF (r) = {x S : d(x, F ) = r}, so that {CF (r) : r 0} forms a
measurable partition of S. Therefore, by Problem, 7.6, there exists a countable set R [0, )
such that (CF (r)) = 0 for r [0, ) \ R. It is not hard to see that BF (r) CF (r) (btw, the
inclusion may be strict), for each r 0. Therefore, (BF (r)) = 0, for all r [0, ) \ R.
The above claim implies that there exists a sequence rk [0, ) \ R such that rk 0. By (5) and
the Claim above, we have n (BF (rk )) (BF (rk )) for all k N. Hence, for k N,
(BF (rk )) = lim n (BF (rk )) lim sup n (F ).
n
By continuity of measure we have (BF (rk )) (F ), as k , and so (F ) lim supn n (F ).

As we will soon see, it is sometimes easy to prove that n (A) (A) for all A in some subset
of B(R). Our next result has something to say about cases when that is enough to establish weak
convergence:
Proposition 7.10 (Weak-convergence test families) Let I be a collection of open subsets of S such
that
1. I is a -system,
2. Each open set in S can be represented as a finite or countable union of elements of I.
w
If n (I) (I), for each I I, then n .

P ROOF For I1 , I2 I, we have I1 I2 I, and so
(I1 I2 ) = (I1 ) + (I2 ) (I1 I2 ) = lim n (I1 ) + lim n (I2 ) lim n (I1 I2 )
n
n
n

= lim n (I1 ) + n (I2 ) n (I1 I2 ) = lim n (I1 I2 ),
n
so that I1 I2 I, i.e., I is closed under finite unions, too.

For an open set G, let G = k Ik be a representation of G as a union of a countable family in I.
By continuity of measure, for each > 0 there exists K N such that (G) (K
k=1 Ik ) + . Since
K
k=1 IK I, we have
K
(G) + = (K
k=1 Ik ) = lim n (k=1 Ik ) lim inf n (G).
n
Since > 0 was arbitrary, we get (G) lim inf n n (G).

92

Remark 7.11 We could have required that I be closed under finite unions right from the start.
Starting from a -system, however, is much more useful for applications.
Corollary 7.12 (Weak convergence using cdfs) Suppose that S = R, and let n be a family of
probability measures on B(R). Let F (x) = ((, x]) and Fn (x) = n ((, x]), x R be the
corresponding cdfs. Then, the following two statements are equivalent:
1. Fn (x) F (x) for all x such that F is continuous at x, and
w
2. n .
P ROOF (2) (1): Let C be the set of all x such that F is continuous at x; eqivalently, C = {x
R : ({x}) = 0}. The sets (, x] are -continuity sets for x C, so the Portmanteau theorem
(Theorem 7.8) implies that Fn (x) = n ((, x]) ((, x]) = F (x), for all x C.
(1) (2): The set C c is at most countable (why?) and so the family
I = {(a, b) : a < b, a, b C},
w
satisfies the the conditions of Proposition 7.10. To show that n , it will be enough to show
that n (I) (I), for all a, b I. Since ((a, b)) = F (b) F (a), where F (b) = limxb F (x), it
will be enough to show that
Fn (x) F (x),
for all x C. Since Fn (x) Fn (x), we have lim supn Fn (x) lim Fn (x) = F (x). To prove
the other inequality, we pick > 0, and, using the continuity of F at x, find > 0 such that
x C and F (x ) > F (x) . Since Fn (x ) F (x ), there exists n0 N such that
Fn (x ) > F (x) 2 for n n0 , and, by increase of Fn , Fn (x) > F (x) 2, for n n0 .
Consequently lim inf n Fn (x) F (x) 2 and the statement follows.
One of the (many) reasons why weak convergence is so important, is the fact that it possesses nice
compactness properties. The central result here is the theorem of Prohorov which is, in a sense, an
analogue of the Arzel-Ascoli compactness theorem for families of measures. The statement we
give here is not the most general possible, but it will serve all our purposes.
Definition 7.13 (Tightness, weak compactness) A subset M of probability measures on S is said

to be
1. tight, if for each > 0 there exists a compact set K such that
sup (K c ) .
2. relatively (sequentially) weakly compact if any sequence {n }nN in M admits a weaklyconvergent subsequence {nk }kN .
93
Theorem 7.14 (Prohorov) Suppose that the metric space (S, d) is complete and separable, and let M
be a set of probability measures on S. Then M is relatively weakly compact if and only if it is tight.
P ROOF (Note: In addition to the fact that the stated version of the theorem is not the most general
available, we only give the proof for the so-called Hellys selection theorem, i.e., the special case
S = R. The general case is technically more involved, but the key ideas are similar.)
(Tight relatively weakly compact): Suppose that M is tight, and let {n }nN be a sequence in
M. Let Q be a countable and dense subset of R, and let {qk }kN be an enumeration of Q. Since
all {n }nN are probability measures, the sequence {Fn (q1 )}nN , where Fn (x) = n ((, x]) is
bounded. Consequently, it admits a convergent subsequence; we denote its indices by n1,k , k N.
The sequence {Fn1,k (q2 )}kN is also bounded, so we can extract a further subsequence - lets denote
it by n2,k , k N, so that Fn2,k (q2 ) converges as k . Repeating this procedure for each element
of Q, we arrive to a sequence of increasing sequences of integers ni,k , k N, i N with the
property that ni+1,k , k N is a subsequence of ni,k , k N and that Fni,k (qj ) converges for each
j i. Therefore, the diagonal sequence mk = nk,k , is a subsequence of each ni,k , k N, i N, and
can define a function F : Q [0, 1] by
F (q) = lim Fmk (q).
k
Each Fn is non-decreasing and so if F . As a matter of fact the right-continuous version

F (x) =
inf
q<x,qQ
F (q),
is non-decreasing and right-continuous (why?), with values in [0, 1].

Our next task is to show that Fmk (x) F (x), for each x CF , where CF is the set of all points
where F is continuous. We pick x CF , > 0 and q1 , q2 Q, y R such that q1 < q2 < x < y and
F (x) < F (q1 ) F (q2 ) F (x) F (y) < F (x) + .
Since Fmk (q2 ) F (q2 ) F (q1 ) and Fmk (s) F (s) F (s) (why is F (s) F (s)?), we have,
for large enough k N
F (x) < Fmk (q2 ) Fmk (x) Fmk (s) < F (x) + ,
which implies that Fmk (x) F (x).
It remains to show - thanks to Corollary 7.12 - that F (x) = ((, x]), for some probability
measure on B(R). For that, in turn, it will be enough to show that F (x) 1, as x and
F (x) 0, as x . Indeed, in that case, we would have all the conditions needed to use
Problem 6.40 to construct a probability space and a random variable X on it so that F is the cdf of
X; the required measure would be the distribution = X of X.
To show that F (x) 0, 1 as x , we use tightness (note that this is the only place in the
proof where it is used). For > 0, we pick M > 0 such that n ([M, M ]) 1 , for all n N. In
terms of corresponding cdfs, this implies that
Fn (M ) and Fn (M ) 1 for all n N.
94

We can assume that M and M are continuity points of F (why?), so that
F (M ) = lim Fmk (M ) and F (M ) = lim Fmk (M ) 1 ,
k
so that limx F (x) 1 and limx F (x) . The claim follows from the arbitrariness of
> 0.
(Relatively weakly compact tight): Suppose to the contrary, that M is relatively weakly compact, but not tight. Then, there exists > 0 such that for each n N there exists n M such that
n ([n, n]) < 1 , and, consequently,
n ([M, M ]) < 1 for n M.
(7.2)
The sequence {n }nN admits a weakly-convergent subsequence {nk }kN . By (7.2), we have
lim sup nk ([M, M ]) 1 , for each M > 0,
k
so that ([M, M ]) 1 for all M > 0. Continuity of probability implies that (R) 1 - a
contradiction with the fact that is a probability measure on B(R).
The following problem cases tightness in more operational terms:
Problem 7.15 Let M be a non-empty set of probability measures on R. Show that M is tight if
and only if there exists a non-decreasing function : [0, ) [0, ) such that
1. (x) as x , and
R
2. supM (|x|) (dx) < .
Prohorovs theorem goes well with the following problem (it will be used soon):
Problem 7.16 Let be a probability measure on B(R) and let {n }nN be a sequence of probability
measures on B(R) with the property that every subsequence {nk }kN of {n }nN has a (further)
subsequence {nkl }lN which converges towards . Show that {n }nN is convergent. (Hint: If
R
w
n
6
, then there exists
f Cb and a subsequence {nk }kN of {n }nN such that f dnk
R
converges, but not to f d.)
We conclude with a comparison between convergence in distribution and convergence in probability.
Proposition 7.17 (Relation between and ) Let {Xn }nN be a sequence of random variables.
D
Then Xn X implies Xn X, for any random variable X. Conversely, Xn X implies Xn X

if there exists c R such that P[X = c] = 1.
P
P ROOF Assume that Xn X. To show that Xn X, the Portmanteau theorem guarantees that
it will be enough to prove that lim supn P[Xn F ] P[X F ], for all closed sets F . For F R,
we define F = {x R : d(x, F ) }. Therefore, for a closed set F , we have
P[Xn F ] = P[Xn F, |X Xn | > ] + P[Xn F, |X Xn | ]
P[|X Xn | > ] + P[X F ].
95

because X F if Xn F and |X Xn | . Taking a lim sup of both sides yields
lim sup P[Xn F ] P[X F ] + lim sup P[|X Xn | > ] = P[X F ].
n
Since >0 F = F , the statement follows.

For the second part, without loss of generality, we assume c = 0. Given m N, let fm Cb (R)
be a continuous function with values in [0, 1] such that fm (0) = 1 and fm (x) = 0 for |x| > 1/m.
Since fm (x) 1[1/m,1/m] (x), we have
P[|Xn | 1/m] E[fm (Xn )] fm (0) = 1,
for each m N.
D
Remark 7.18 It is not true that Xn X implies Xn X in general. Here is a simple example:
take = {1, 2} with uniform probability, and define Xn (1) = 1 and Xn (2) = 2, for n odd and
Xn (1) = 2 and Xn (2) = 1, for n even. Then all Xn have the same distribution, so we have
D
Xn X1 . On the other hand P[|Xn X1 | 21 ] = 1, for n even. In fact, it is not hard to see that
P
Xn
6 X for any random variable X.
7.2
Characteristic functions
A characteristic function is simply the Fourier transform, in probabilistic language. Since we will
be integrating complex-valued functions, we define (both integrals on the right need to exist)
Z
Z
Z
f d = f d + i f d,
where f and f denote the real and the imaginary part of a function f : R C. The reader will
easily figure out which properties of the integral transfer from the real case.
Definition 7.19 (Characteristic functions) The characteristic function of a probability measure

on B(R) is the function : R C given by
Z
(t) = eitx (dx)
When we speak of the characteristic function X of a random variable X, we have the characteristic function X of its distribution X in mind. Note, however, that
X (t) = E[eitX ].
While difficult to visualize, characteristic functions can be used to learn a lot about the random
variables they correspond to. We start with some properties which follow directly from the definition:
96
Proposition 7.20 (First properties of characteristic functions) Let X, Y and {Xn }nN be a random variables.
1. X (0) = 1 and |X (t)| 1, for all t.
2. X (t) = X (t), where bar denotes complex conjugation.
3. X is uniformly continuous.
4. If X and Y are independent, then X+Y = X Y .
5. For all t1 < t2 < < tn , the matrix A = (aij )1i,jn given by
ajk = X (tj tk ),
is Hermitian and positive semi-definite, i.e., A = A and T A 0, for any Cn ,
D
6. If Xn X, then Xn (t) X (t), for each t R.

P ROOF
1. Immediate.
2. eitx = eitx .

R
R
3. We have
|X (t) X (s)| = (eitx eisx ) (dx) h(ts), where h(u) = eiux 1 (dx).
iux
Since e 1 2, dominated convergence theorem implies that limu0 h(u) = 0, and, so,
X is uniformly continuous.
4. Independence of X and Y implies the independence of exp(itX) and exp(itY ). Therefore,

X+Y (t) = E[eit(X+Y ) ] = E[eitX eitY ] = E[eitX ]E[eitY ] = X (t)Y (t).
5. The matrix A is Hermitian by (2). To see that it is positive semidefinite, note that ajk =
E[eitj X eitk X ], and so
!
n
n
n
n X
n
X
X
X
X
itj X
it
X
k
j k ajk = E
j e
k e
= E[|
j eitj X |2 ] 0.
j=1 k=1
j=1
j=1
k=1
6. For f Cb (R), we have f (Xn ) f (X), a.s., and so, by the dominated convergence theorem
applied to the cases f (x) = cos(tx) and f (x) = sin(tx), we have
X (t) = E[exp(itX)] = E[lim exp(itXn )] = lim E[exp(itXn )] = lim Xn (t).
n
Remark 7.21 We do not prove (or use) it in these notes, but it can be shown that a function :
R C, continuous at the origin with (0) = 1 is a characteristic function of some probability
measure on B(R) if and only if it satisfies (5). This is known as Bochners theorem.
97

Here is a simple problem you can use to test your understanding of the definitions:
Problem 7.22 Let and be two probability measures on B(R), and let and be their characteristic functions. Show that Parsevals identity holds:
Z
Z
its
e
(t) (dt) =
(t s) (dt), for all s R.
R
Our next result shows can be recovered from its characteristic function :
Theorem 7.23 (Inversion theorem) Let be a probability measure on B(R), and let = be its
characteristic function. Then, for a < b R, we have
(7.3)
Z T
1
lim
2 T
T
((a, b)) + 12 ({a, b}) =
eita eitb
(t) dt.
it
P ROOF We start by picking a < b and noting that

eita eitb
=
it
eity dy,
so that, by Fubinis theorem, the integral in (7.3) is well-defined:

Z
F (a, b, T ) =
exp(ity)(t) dy dt,
[T,T ][a,b]
where
F (a, b, T ) =
Another use of Fubinis theorem yields:
Z
F (a, b, T ) =
T
T
[T,T ][a,b]R
=
=
Set
f (a, b, T ) =
T
T
exp(ity) exp(itx) dy dt (dx)

!
[T,T ][a,b]
[T,T ]
1 it(ax)
it (e
1
it
eita eitb
(t) dt.
it
exp(it(y x)) dy dt
it(ax)
it(bx)
dt
(dx)
eit(bx) ) dt and K(T, c) =
(dx).
T
0
sin(ct)
t
dt,
and note that, since cos is an even and sin an odd function, we have
Z T

sin((bx)t)
sin((ax)t)
dt = 2K(T ; a x) 2K(T ; b x).

f (a, b, T ; x) = 2
t
t
0
98

RT
(Note: The integral T it1 exp(it(a x)) dt is not defined; we really need to work with the full
f (a, b, T ; x) to get the cancellation above.)
Since
R T sin(ct)
R cT sin(s)
0 ct d(ct) = 0
s ds = K(cT ; 1), c > 0
K(T ; c) = 0,
(7.4)
c=0
K(|c| T ; 1),
c < 0,
Problem 5.24 implies that
2, c > 0
0,
0, c = 0
lim K(T ; c) =
and so lim f (a, b, T ; x) = ,

T
T
2, c < 0
2,
x [a, b]c
x = a or x = b
a<x<b
Observe first that the function T 7 K(T ; 1) is continuous on [0, ) and has a finite limit as T
so that supT 0 |K(T ; 1)| < . Furthermore, (7.4) implies that |K(T ; c)| supT 0 K(T ; 1) for any
c R and T 0 so that
sup{|f (a, b, T ; x)| : x R, T 0} < .
Therefore, we can use the dominated convergence theorem to conclude that

Z
Z
1
1
1
lim 2 F (a, b, T ; x) = lim 2 f (a, b, T ; x) (dx) = 2
lim f (a, b, T ; x) (x)
T
= 12 ({a}) + ((a, b)) + 21 ({b}).
Corollary 7.24 (Characteristic-ness of characteristic functions) For probability measures 1

and 2 on B(R), the equality 1 = 2 implies that 1 = 2 .
P ROOF By Theorem 7.23, we have 1 ((a, b)) = 2 ((a, b)) for all a, b C where C is the set of all
x R such that 1 ({x}) = 2 ({x}) = 0. Since C c is at most countable, it is straightforward to see
that the family {(a, b) : a, b C} of intervals is a -system which generates B(R).
Corollary 7.25 (InversionRfor integrably characteristic functions) Suppose that is a probability measure on B(R) with R | (t)| dt < . Then and d
d is a bounded and continuous
function given by
Z
1
d
= f, where f (x) =
eitx (t) dt.
d
2 R

P ROOF Since is integrable and eitx = 1, f is well defined. For a < b we have

Z b
Z b
Z bZ
Z
1
1
itx
itx
f (x) dx =
e
(t) dt dx =
e
dx dt
(t)
2 a R
2 R
a
a
Z ita
Z T ita
(7.5)
1
e
eitb
e
eitb
1
(t) dt = lim
(t) dt
=
T 2 T
2 R
it
it
= ((a, b)) + 21 ({a, b}),
99

by Theorem 7.23, where the use of Fubinis theorem above is justified by the fact that the function
(t, x) 7 eitx (t) is integrable on [a, b] R, for all a < b. For a, b such that ({a}) = ({b}) = 0,
Rb
the equation (7.5) implies that ((a, b)) = a f (x) dx. The claim now follows by the -theorem.
Example 7.26 (Common distributions and their characteristic functions) Here is a list of some
common distributions and the corresponding characteristic functions:
1. Continuous distributions.
Name
Parameters
Uniform
a 0
Exponential
>0
Double Exponential
>0
Cauchy
R, > 0
Density fX (x)
1
ba
1
2 2
Ch. function X (t)

eita eitb
it(ba)
1[a,b] (x)
(x)2
2 2
exp(
exp(it 12 2 t2 )
1
1it
exp(x)1[0,) (x)
1
2
1
1+t2
exp( |x|)
( 2 +(x)2 )
exp(it |t|)
2. Discrete distributions.
Name
Parameters
pn = P[X = n], n Z
Ch. function X (t)
Dirac
m N0
1{m=n}
exp(itm)
Coin-toss
p (0, 1)
p1 = p, p1 = (1 p)
cos(t)
Geometric
p (0, 1)
pn (1 p), n N0
1p
1eit p
Poisson
>0
n
n!
exp((eit 1))
, n N0
3. A singular distribution.
10
Name
Ch. function X (t)
Cantor
e 2 it
k=1
cos(
t
3k
7.3
Tail behavior
We continue by describing several methods one can use to extract useful information about the
tails of the underlying probability distribution from a characteristic function.
100
Proposition 7.27 (Existence of moments implies regularity of X at 0) Let X be a random

dn
variable. If E[|X|n ] < , then (dt)
n X (t) exists for all t and
dn
(dt)n X (t)
= E[eitX (iX)n ].
In particular
d
E[X n ] = (i)n (dt)
n X (0).
P ROOF We give the proof in the case n = 1 and leave the general case to the reader:
Z
Z
Z
(h)(0)
eihx 1
eihx 1
lim
= lim
(dx) =
lim h (dx) =
ix (dx),
h
h
h0
h0 R
R h0
where the passage of the limit under the integral sign is justified by the dominated convergence
theorem which, in turn, can be used since
Z
ihx
e 1
|x|
,
and
|x| (dx) = E[|X|] < .
h
R
Remark 7.28
1. It can be shown that for n even, the existence of

the finiteness of the n-th moment E[|X|n ].
2. When n is odd, it can happen that
dn
(dt)n X (0)
dn
(dt)n X (0)
(in the appropriate sense) implies
exists, but E[|X|n ] = - see Problem 7.40.
Finer estimates of the tails of a probability distribution can be obtained by finer analysis of the
behavior of around 0:
Proposition 7.29 (A tail estimate) Let be a probability measure on B(R) and let = be its
characteristic function. Then, for > 0 we have
Z
2 2 c
1
([ , ] )
(1 (t)) dt.
P ROOF Let X be a random variable with distribution . We start by using Fubinis theorem to get
Z
Z
Z
itX
1
1
1
(1 (t)) dt = 2 E[
(1 e ) dt] = E[ (1 cos(tX)) dt] = E[1 sin(X)
2
X ].
1
0 and 1 sin(x)
1 |x|
for all x. Therefore, if we use the
It remains to observe that 1 sin(x)
x
x
c
first inequality on [2, 2] and the second one on [2, 2] , we get
Z
sin(x)
1
1
1 x 2 1{|x|>2} so that 2
(1 (t)) dt 12 P[|X| > 2] = 21 ([ 2 , 2 ]c ).
101

Problem 7.30 Use the inequality of Proposition 7.29 to show that if (t) = 1 + O(|t| ) for some
R
R
> 0, then R |x| (dx) < , for all < . Give an example where R |x| (dx) = .
(Note: f (t) = g(t) + O(h(t)) means that sup|t| |f (t)g(t)|
< , for some > 0.)
h(t)
Problem 7.31 (Riemann-Lebesgue theorem) Suppose that . Show that
lim (t) = lim (t) = 0.
(Hint:P
Use (and prove) the fact that f L1+ (R) can be approximated in L1 (R) by a function of the
form nk=1 k 1[ak ,bk ] .)
7.4
The continuity theorem
Theorem 7.32 (Continuity theorem) Let {n }nN be a sequence of probability distributions on

B(R), and let {n }nN be the sequence of their characteristic functions. Suppose that there exists a
function : R C such that
1. n (t) (t), for all t R, and
2. is continuous at t = 0.
w
Then, is the characteristic function of a probability measure on B(R) and n .

P ROOF We start by showing that the continuity of the limit implies tightness of {n }nN . Given
> 0 there exists > 0 such that 1(t) /2 for |t| . By the dominated convergence theorem
we have
Z
Z
1
1
2 2 c
(1 n (t)) dt =
(1 (t)) dt .
lim sup n ([ , ] ) lim sup
n
By taking an even smaller > 0, we can guarantee that

sup n ([ 2 , 2 ]c ) ,
nN
which, together with the arbitrariness of > 0 implies that {n }nN is tight.
Let {nk }kN be a convergent subsequence of {n }nN , and let be its limit. Since nk ,
we conclude that is the characteristic function of . It remains to show that the whole sequence
converges to weakly. This follows, however, directly from Problem 7.16, since any convergent
subsequence {nk }kN has the same limit .
Problem 7.33 Let be a characteristic function of some probability measure on B(R). Show that
(t)
= e(t)1 is also a characteristic function of some probability measure
on B(R).
102
7.5
Additional Problems
Problem 7.34 (Scheffs Theorem) Let {Xn }nN be absolutely-continuous random variables with
densities fXn , such that fXn (x) f (x), -a.e., where f is the density of the absolutely-continuous
R
D
random variable X. Show that Xn X. (Hint: Show that R |fXn f | d 0 by writing the
integrand in terms of (f fXn )+ f . )
Problem 7.35 (Convergence of moments) Let {Xn }nN and X be random variables with a common uniform bound, i.e., such that
M > 0, n N, |Xn | M, |X| M, a.s.
Show that the following two statements are equivalent:
D
1. Xn X (where denotes convergence in distribution), and

2. E[Xnk ] E[X k ], as n , for all k N.
(Hint: Use the Weierstrass approximation theorem: Given a 0 there exists a polynomial P such that supx[a,b] |f (x) P (x)| .)
Problem 7.36 (Total-variation convergence) A sequence {n }nN of probability measures on B(R)
is said to converge to the probability measure in (total) variation if
sup |n (A) (A)| 0 as n .
AB(R)
Compare convergence in variation to weak convergence: if one implies the other, prove it. Give
counterexamples, if they are not equivalent.
Problem 7.37 (Convergence of Maxima) Let {Xn }nN be an iid sequence of standard normal (N (0, 1))
random variables. Define the sequence of up-to-date-maxima {Mn }nN by
Mn = max(X1 , . . . , Xn ).
Show that
1. Show that limx
P[X1 >x]
1
exp( 2 x2 )
x1
P[X1 > x]
1
1
1
3 , x > 0,
x
(x)
x x
(7.6)
where, (x) =
by parts).
= (2) 2 by establishing the following inequality
1
2
exp( 12 x2 ) is the density of the standard normal. (Hint: Use integration
2. Prove that for any R, limx
P[X1 >x+ x ]
P[X1 >x]
= exp().
3. Let {bn }nN be a sequence of real numbers with the property that P[X1 > bn ] = 1/n. Show
that P[bn (Mn bn ) x] exp(ex ).
4. Show that limn
bn
2 log n
= 1.
103

5. Show that
Mn
2 log n
1 in probability. (Hint: Use Problem 7.17.)
Problem 7.38 Check that the expressions for characteristic functions in Example 7.26 are correct.
(Hint: Not much computing is needed. Use the inversion theorem. For 2., start with the case
= 0, = 1 and derive a first-order differential equation for .)
Problem 7.39 (Atoms from the characteristic function) Let be a probability measure on B(R),
and let = be its characteristic function.
R T ita
1
(t) dt.
1. Show that ({a}) = limT 2T
T e
2. Show that if limt |(t)| = limt |(t)| = 0, then has no atoms.
3. Show that converse of (2) is false. (Hint: Prove that |(tn )| = 1 along a suitably chosen
sequence tn , where is the characteristic function of the Cantor distribution. )
Problem 7.40 (Existence of X (0) does not imply that X L1 ) Let X be a random variable which
takes values in Z \ {2, 1, 0, 1, 2} with
P[X = k] = P[X = k] =
C
,
k2 log(k)
for k = 3, 4, . . . ,
P
1
where C = 12 ( k3 k2 log(k)
)1 (0, ). Show that X (0) = 0, but X 6 L1 .
(Hint: Argue that, in order to establish that X (0) = 0, it is enough to show that
X cos(hk)1
= 0.
lim h1
k2 log(k)
h0
k3
Then split the sum at k close to 2/h and use (and prove) the inequality |cos(x) 1| min(x2 /2, x).
Bounding sums by integrals may help, too. )
Problem 7.41 (Multivariate characteristic functions) Let X = (X1 , . . . , Xn ) be a random vector.
The characteristic function = X : Rn C is given by
(t1 , t2 , . . . , tn ) = E[exp(i
n
X
tk Xk )].
k=1
P
We will also use the shortcut t for (t1 , . . . , tn ) and t X for the random variable nk=1 tk Xk . Take
for granted the following statement (the proof of which is similar to the proof of the 1-dimensional
case):
Fact. Suppose that X 1 and X 2 are random vectors with X 1 (t) = X 2 (t) for all t Rn . Then X 1
and X 2 have the same distribution, i.e. X 1 = X 2 .
Prove the following statements
1. Random variables X and Y are independent if and only if (X,Y ) (t1 , t2 ) = X (t1 )Y (t2 ) for
all t1 , t2 R.
2. Random vectors X 1 and X 2 have the same distribution if and only if random variables tX 1
and t X 2 have the same distribution for all t Rn . (This fact is known as Walds device.)
104

An n-dimensional random vector X is said to be Gaussian (or, to have the multivariate normal
distribution) if there exists a vector Rn and a symmetric positive semi-definite matrix Rnn
such that
X (t) = exp(i t 21 t t),
where t is interpreted as a column vector, and () is transposition. This is denoted as X
N (, ). X is said to be non-degenerate if is positive definite.
3. Show that a random vector X is Gaussian, if and only if the random vector t X is normally
distributed (with some mean and variance) for each t Rn . (Note: Be careful, nothing in the
second statement tells you what the mean and variance of t X are.)
4. Let X = (X1 , X2 , . . . , Xn ) be a Gaussian random vector. Show that Xk and Xl , k 6= l, are
independent if and only if they are uncorrelated.
5. Construct a random vector (X, Y ) such that both X and Y are normally distributed, but that
X = (X, Y ) is not Gaussian.
6. Let X = (X1 , X2 , . . . , Xn ) be a random vector consisting of n independent random variables
with Xi N (0, 1). Let Rnn be a given positive semi-definite symmetric matrix, and
Rn a given vector. Show that there exists an affine transformation T : Rn Rn such
that the random vector T (X) is Gaussian with T (X) N (, ).
7. Find a necessary and sufficient condition on and such that the converse of the previous
problem holds true: For a Gaussian random vector X N (, ), there exists an affine
transformation T : Rn Rn such that T (X) has independent components with the N (0, 1)distribution (i.e. T (X) N (0, yI), where yI is the identity matrix).
Problem 7.42 (Slutskys Theorem) Let X, Y , {Xn }nN and {Yn }nN be random variables defined
D
on the same probability space, such that Xn X and Yn Y . Show that

D
1. It is not necessarily true that Xn + Yn X + Y . For that matter, we do not necessarily have
D
(Xn , Yn ) (X, Y ) (where the pairs are considered as random elements in the metric space
R2 ).
2. If, in addition to (11.13), there exists a constant c R such that P[Y = c] = 1, show that
D
g(Xn , Yn ) g(X, c), for any continuous function g : R2 R. (Hint: It is enough to show
D
that (Xn , Yn ) (Xn , c). Use Problem 7.41).)
Problem 7.43 (Convergence of a normal sequence)

1. Let {Xn }nN be a sequence of normally-distributed random variables converging weakly
towards a random variable X. Show that X must be a normal random variable itself. (Hint:
Use the following fact: for a sequence {n }nN of real numbers, the following two statements are
equivalent
n R, and t R, exp(itn ) exp(it).
You dont need to prove it, but feel free to try.)
a.s.
Lp
2. Let Xn be a sequence of normal random variables such that Xn X. Show that Xn X

for all p 1.
105
Chapter
Classical Limit Theorems

8.1
The weak law of large numbers
We start with a definitive form of the weak law of large numbers. We need two lemmas, first:
Lemma 8.1 (A simple inequality) Let u1 , u2 , . . . , un and w1 , w2 , . . . , wn be complex numbers, all

of modulus at most M > 0. Then

n
n
n
Y

X
Y

n1
uk
|uk wk | .
wk M
(8.1)

k=1
k=1
k=1
P ROOF We proceed by induction. For n = 1, the claim is trivial. Suppose that (8.1) holds. Then

n
n
n+1
n
n+1

Y
Y

Y
Y
Y

uk
uk
uk |un+1 wn+1 | + |wn+1 |
wk
wk

k=1
k=1
k=1
M n |un+1 wn+1 | + M M n1
= M (n+1)1
n+1
X
k=1
k=1
n
X
k=1
k=1
|uk wk |
|uk wk | .
Lemma 8.2 (Convergence to the exponential) Let {zn }nN be a sequence of complex numbers
with zn z C. Then (1 + znn )n ez .
(8.2)
zn
n
and wk = ezn /n for k = 1, . . . , n, we get

(1 + zn )n ezn nMnn1 1 + zn ezn /n ,
n
n
P ROOF Using Lemma 8.1 with uk = 1 +
106
CHAPTER 8. CLASSICAL LIMIT THEOREMS

1 + zn , ezn /n ). Let K = supnN |zn | < , so that ezn /n n e|zn | eK .
where Mn = max(
n
n
n
K
Similarly, 1 + znn (1 + K
n ) e . Therefore
L = sup Mnn1 < .
(8.3)
nN
P
To estimate the last term in (8.2), we start with the Taylor expansion eb = 1 + b + k2
1
which converges absolutely for all b C. Then, we use the fact that k!
2k+1 , to obtain
X

2 X k+1

b
k 1
b
|b|
2
= |b|2 , for |b| 1.

k!
(8.4)
k2
bk
k! ,
k2
Since |zn | /n 1 for large-enough n, it follows from (8.2), (8.4) and (8.3), that

lim sup (1 +
zn n
n)
2

ezn lim sup nL znn = 0.
n
It remains to remember that ezn ez to finish the proof.
Theorem 8.3 (Weak law of large numbers) Let {Xn }nN be an iid sequence of random variables
with the (common) distribution and the characteristic function = such that (0) exists. Then,
c = i (0) R and
n
1X
Xk c in probability.
n
k=1
P ROOF Since (s) = (s), we have

(0) = lim
s0
(s)1
s
= lim
s0
(s)1
s
= lim
s0
(s)1
s
= (0).
Therefore, c = i (0) R.
P
D
Let Sn = nk=1 Xk . According to Proposition 7.17, it will be enough to show that n1 Sn c =
i (0) R. Moreover, by Theorem 7.32, all we need to do is show that 1 (t) eitc = et (0) ,
n Sn
for all t R.
The iid property of {Xn }nN and the fact that X (t) = X (t) imply that
1
n Sn
(t) = (( nt ))n = (1 +
zn n
n) ,
where zn = n(( nt ) 1). By the assumption, we have lims0

Therefore, by Lemma 8.2 above, we have 1 (t) eitc .
(s)1
s
= ic, and so zn t (0).
n Sn
Remark 8.4
P
1. It can be shown that the converse of Theorem 8.3 is true in the following sense: if n1 Sn c
R, then (0) exists and (0) = ic. Thats why we call the result of Theorem 8.3 definitive.
107

2. X1 L1 implies (0) = E[X1 ], so that Theorem 8.3 covers the classical case. As we have
seen in Problem 7.40, there are cases when (0) exists but E[|X1 |] = .
Problem 8.5 Let {Xn }nN be iid with the Cauchy distribution (remember, the density of the Cauchy
1
distribution is given by fX (x) = (1+x
2 ) , x R). Show that X is not differentiable at 0 and show
that there is no constant c such that
P
1
n Sn c,
P
where Sn = nk=1 Xk . (Hint: What is the distribution of n1 Sn ?)
8.2
An iid-central limit theorem
We continue with a central limit theorem for iid sequences. Unlike in the case of the (weak) law of
large numbers, existence of the first moment will not be enough - we will need to assume that the
second moment is finite, too. We will see how this assumption can be relaxed when we state and
prove the Lindeberg-Feller theorem. We start with an estimate of the error term in the Taylor
expansion of the exponential function of imaginary argument:
Lemma 8.6 (A tight error estimate for the exponential) For R we have

n

X
k
i
||n+1
||n
(i)
e
min( (n+1)! , 2 n! )
k!

k=0
P ROOF If we write the remainder in the Taylor formula in the integral form (derived easily using
integration by parts), we get
ei
n
X
(i)k
k!
= Rn (), where Rn () = in+1
k=0
eiu (u)
du.
n!
The usual estimate of Rn gives:

|Rn ()|
1
n!
||
0
(|| u)n du =
||n+1
(n+1)! .
We could also transform the expression for Rn by integrating it by parts:

Rn () =
=
since n = n
R
0
in+1
n!
in
(n1)!
1 n
i
Z
n
i
iu
e ( u)
( u)n1 du
n1
du

eiu ( u)n1 du ,
( u)n1 du. Therefore
|Rn ()|
1
(n1)!
||
0

(|| u)n1 eiu 1 du
108
2
n!
||
0
n(|| u)n1 du =
2||n
n! .

While the following result can be obtained as a direct consequence of twice-differentiability of the
function at 0, we use the (otherwise useful) estimate based on Lemma 8.6 above:
Corollary 8.7 (A two-finite-moment regularity estimate) Let X be a random variable with

E[X] = and E[X 2 ] = < , and let the function r : [0, ) [0, ) be defined by
r(t) = E[X 2 min(t |X| , 1)], t 0.
(8.5)
Then
1. limt0 r(t) = 0 and

2. X (t) 1 it + 21 t2 t2 r(|t|).
P ROOF The inequality in (2) is a direct consequence of Lemma 8.6 (with the extra factor
glected). As for (1), it follows from the dominated convergence theorem because
1
6
ne-
X 2 min(1, t |X|) X 2 L1 and lim X 2 min(1, t |X|) = 0.

t0
Theorem 8.8 (Central Limit Theorem - iid version) Let {Xn }nN be an iid sequence of random
variables with 0 < Var[X1 ] < . Then
Pn
(X )
k=1
k
2 n
where N (0, 1), = E[X1 ] and 2 = Var[X1 ].
P ROOF By considering the sequence {(Xn )/ 2 }nN , instead of {Xn }nN , we may assume
that = 0 and
P = 1. Let be the characteristic function of the common distribution of {Xn }nN
and set Sn = nk=1 Xk , so that
1 (t) = (( tn ))n ,
n
Sn
By Theorem 7.32, the problem reduces to whether the following statement holds:
(8.6)
lim (( tn ))n = e 2 t , for each t R.
Corollary 8.7 guarantees that
where r is given by (8.5), i.e.,

(t) 1 + 1 t2 t2 r(t) for all t R,
2

t
( n ) 1 +
t2
2n
109
t2
n r(t/
n).

2
t
) yields:
Lemma 8.1 with u1 = = un = ( tn ) and w1 = = wn = (1 2n

t2 n
) t2 r(t/ n),
(( tn ))n (1 2n
for n
2
t2
1). Since limn r(t/ n) = 0, we have

t2 n
lim (( tn ))n (1 2n
) = 0,
(so that max(|( tn )|, |1
t2
2n |)
and (8.6) follows from the fact that (1
t2 n
2n )
e 2 t , for all t.
8.3
The Lindeberg-Feller Theorem
Unlike Theorem 8.8, the Lindeberg-Feller Theorem does not require summands to be equall distributed - it only asks for no single term to dominated the sum. As usual, we start with a technical
lemma:
Lemma 8.9 ( (*) Convergence to the exponential for triangular arrays) Let (cn,m ), n N,
m = 1, . . . , n be a (triangular) array of real numbers with
Pn
Pn
1.
m=1 cn,m c R, and
m=1 |cn,m | is a bounded sequence,
2. mn 0, as n , where mn = max1mn |cn,m | 0
Then
n
Y
m=1
(1 + cn,m ) ec as n .
1
P ROOF Without
2 for all n, and note that the statement is
Pnloss of generality we assume that mn 1. It is straightforward to check that

1
+ 2x 1 satisfies f (x) > 0 for x > 0 and f (x) < 0
f (0) = 0 and that the derivative f (x) = 1+x
for x (1/2, ). It follows that f (x) 0 for x [1/2, ) so that (the absolute value can be
inserted since x log(1 + x))
|log(1 + x) x| x2 for x 21 .
Since mn < 21 , we have |log(1 + cn,m ) cn,m | c2n,m , and so

n
n
n
X
X
X

c2n,m
|log(1 + cn,m ) cn,m |
(log(1 + cn,m ) cn,m )

m=1
mn
because
Pn
m=1 |cn,m |
m=1
m=1
is bounded and mn 0.
n
X
m=1
110
|cn,m | 0,
Theorem 8.10 (Lindeberg-Feller) Let Xn,m , n N, m = 1, . . . , n be a (triangular) array of random variables such that
1. E[Xn,m ] = 0, for all n N, m = 1, . . . , n,
2. Xn,1 , . . . , Xn,n are independent,
Pn
2
2
3.
m=1 E[Xn,m ] > 0, as n ,
4. for each > 0, sn () 0, as n , where sn () =

Then
Pn
2
m=1 E[Xn,m 1{|Xn,m |} ].
Xn,1 + + Xn,n , as n ,
where N (0, 1).
2
2 ]. Just like in the proofs of Theorems 8.3 and 8.8, it
P ROOF (*) Set n,m = Xn,m , n,m
= E[Xn,m
will be enough to show that
n
Y
m=1
n,m (t) e 2
2 t2
, for all t R.
2 t2 to conclude that
We fix t 6= 0 and use Lemma 8.1 with un,m = n,m (t) and wn,m = 1 21 n,m
Dn (t)
Mnn1
n
X

2
2
n,m (t) 1 + 1 n,m
,
t
2
m=1
where

n
n

Y
Y

2
t2 )
n,m (t)
Dn (t) =
(1 12 n,m

m=1
m=1

2 ). Assumption (4) in the statement implies that
and Mn = 1 max1mn (1 21 t2 n,m
2
2
2
2
n,m
= E[Xn,m
1{|Xm,n |} ] + E[Xn,m
1{|Xm,n |<} ] 2 + E[Xn,m
1{|Xm,n |<} ]
2 + sn (),
2
2
2 and
and so sup1mn n,m
0, as n . Therefore, for n large enough, we have 21 t2 n,m
Mn = 1.
According to Corollary 8.7 we now have (for large-enough n)
Dn (t) t2
t2
n
X
2
min(t |Xn,m | , 1)]
E[Xn,m
m=1
n
X
m=1
2
1{|Xn,m |} ] + E[t |Xn,m |3 1{|Xn,m |<} ]
E[Xn,m
t2 sn () + t3
n
X
m=1
2
E[Xn,m
1{|Xn,m |<} ] t2 sn () + 2t3 2 .
111

Therefore, lim supn Dn (t) 2t3 2 , and so, since > 0 is arbitrary, we
have limn Dn (t) = 0.
Pm
2
2
Our last task is to remember that max1mn n,m 0, note that m=1 n,m
2 (why?), and
use Lemma 8.9 to conclude that
n
Y
m=1
2
(1 12 n,m
t2 ) e 2
2 t2
Problem 8.11 Show how the iid central limit theorem follows from the Lindeberg-Feller theorem.
Example 8.12 (Cycles in a random permutation) Let : Sn be a random element taking
values in the set Sn of all permutations of the set {1, . . . , n}, i.e., the set of all bijections :
{1, . . . , n} {1, . . . , n}. One usually considers the probability measure on such that is uni1
, for each Sn . A random element in Sn whose
formly distributed over Sn , i.e. P[ = ] = n!
distribution is uniform over Sn is called a random permutation.
Remember that each permutation Sn be decomposed into cycles; a cycle is a collection
(i1 i2 . . . ik ) in {1, . . . , n} such that (il ) = il+1 for l = 1, . . . , k 1 and (ik ) = i1 . For example,
the permutation : {1, 2, 3, 4} {1, 2, 3, 4}, given by (1) = 3, (2) = 1, (3) = 2, (4) = 4 has
two cycles: (132) and (4). More precisely, start from i1 = 1 and follow the sequence ik+1 = (ik ),
until the first time you return to ik = 1. Write these number in order (i1 i2 . . . ik ) and pick j1
{1, 2, . . . , n} \ {i1 , . . . , ik }. If no such j1 exists, consist of a single cycle. If it does, we repeat the
same procedure starting from j1 to obtain another cycle (j1 j2 . . . jl ), etc. In the end, we arrive at
the decomposition
(i1 i2 . . . ik )(j1 j2 . . . jl ) . . .
of into cycles.
Let us first answer the following, warm-up, question: what is the probability p(n, m) that 1 is a
member of a cycle of length m? Equivalently, we can ask for the number c(n, m) of permutations
in which 1 is a member of a cycle of length m. The easiest way to solve this is to note that each such
permutation corresponds to a choice of (m 1) district numbers of {2, 3, . . .} - these will serve as
n1
the remaining elements of the cycle containing 1. This can be done in m1
ways. Furthermore,
the m 1 elements to be in the same cycle with 1 can be ordered in (m 1)! ways. Also, the
remaining n m elements give rise to (n m)! distinct permutations. Therefore,

n1
c(n, m) =
(m 1)!(n m)! = (n 1)!, and so p(n, m) = n1 .
m1
This is a remarkable result - all cycle lengths are equally likely. Note, also, that 1 is not special in
any way.
Our next goal is to say something about the number of cycles - a more difficult task. We start
by describing a procedure for producing a random permutation by building it from cycles. The
reader will easily convince his-/herself that the outcome is uniformly distributed over all permutations. We start with n 1 independent random variables 2 , . . . , n such that i is uniformly
distributed over the set {0, 1, 2, . . . , n i + 1}. Let the first cycle start from X1 = 1. If 2 = 0, then
we declare (1) to be a full cycle and start building the next cycle from 2. If 2 6= 0, we pick the
2 -th smallest element - let us call it X2 - from the set of remaining n 1 numbers to be the second
element in the first cycle. After that, we close the cycle if 3 = 0, or append the 3 -th smallest
element - lets call it X3 - in {1, 2, . . . , n} \ {X1 , X2 } to the cycle. Once the cycle (X1 X2 . . . Xk )
is closed, we pick the smallest element in {1, 2, . . . , n} \ {X1 , X2 , . . . , Xk } - lets call it Xk+1 - and
repeat the procedure starting from Xk+1 and using k+1 , . . . , n as sources of randomness.
112

Let us now define the random variables (we stress the dependence on n here) Yn,1 , . . . , Yn,n
by Yn,k = 1{k =0} . In words, Yn,k is in indicator of the event when a cycle ends right after the
position k. It is clear that Yn,1 , . . . , Yn,k are independent (they are functions of independent vari1
ables 1 , . . . , n ). Also, p(n, k) = P[Yn,k = 1] = nk+1
. The number of cycles Cn is the same as the
Pn
number of closing parenthesis, so Cn = k=1 Yk,n . (Btw, can you derive the identity p(n, m) = n1
by using random variables Yn,1 , . . . , Yn,n ?)
It is easy to compute
E[Cn ] =
n
X
E[Yn,k = 1] =
k=1
n
X
1
nk+1
=1+
k=1
1
2
+ +
1
n
= log(n) + + o(1),
where 0.58 is the Euler-Mascheroni constant, and an = bn + o(n) means that |bn an | 0, as
n .
If we want to know more about the variability of Cn , we can also compute its variance:
Var[Cn ] =
n
X
Var[Yn,k ] =
k=1
n
X
k=1
( nk+1
1
)
(nk+1)2
= log(n) +
2
6
+ o(1).
The Lindeberg-Feller theorem will give us the precise asymptotic behavior of Cn . For m =
1, . . . , n, we define
Yn,m E[Yn,m ]
p
,
Xn,m =
log(n)
so that Xn,m , m = 1, . . . , n are independent and of mean 0. Furthermore, we have

lim
n
n
X
2
E[Xn,m
]
m=1
= lim
n
2
log(n)+ 6 +o(1)
log(n)
= 1.
Finally, for > 0 and log(n) > 2/, we have P[|Xn,m | > ] = 0, so
n
X
m=1
2
1{|Xn,m |} ] = 0.
E[Xn,m
Having checked that all the assumption of the Lindeberg-Feller theorem are satisfied, we conclude
that
Cn log(n) D
, where N (0, 1).

log(n)
It follows that (if we believe that the approximation is good) the number of cycles in a random
permutation with n = 8100 is at most 18 with probability 99%.
How about variability? Here is histogram of the number of cycles from 1000 simulations for
n = 8100, together with the appropriately-scaled
density of the normal distribution with mean
p
log(8100) and standard deviation log(8100). The quality of approximation leaves something to
113

be desired, but it seems to already work well in the tails: only 3 of 1000 had more than 17 cycles:
140
120
100
80
60
40
20
10
15
20
8.4
Additional Problems
P
Problem 8.13 p
(Lyapunovs theorem) Let {Xn }nN be an independent sequence, let Sn = nm=1 Xm ,
and let n = Var[Sn ]. Suppose that n > 0 for all n N and that there exists a constant > 0
such that
n
X
lim n(2+)
E[|Xm E[Xm ]|2+ ] = 0.
n
Show that
m=1
Sn E[Sn ] D
, where N (0, 1).
n
Problem 8.14 (Self-normalized sums) Let {Xn }nN be iid random variables with E[X1 ] = 0, =
p
E[X12 ] > 0 and P[X1 = 0] = 0. Show that the sequence {Yn }nN given by
Pn
Xk
Yn = qPk=1
,
n
2
X
k=1 k
converges in distribution, and identify its limit.

(Hint: Use Slutskys theorem of Problem 7.42; you dont have to prove it.)
114
Chapter
Conditional Expectation
9.1
The definition and existence of conditional expectation
For events A, B with P[B] > 0, we recall the familiar object

P[A|B] =
P[AB]
P[B] .
We say that P[A|B] the conditional probability of A, given B.

It is important to note that
the condition P[B] > 0 is crucial. When X and Y are random variables defined on the same
probability space, we often want to give a meaning to the expression P[X A|Y = y], even though
it is usually the case that P[Y = y] = 0. When the random vector (X, Y ) admits a joint density
fX,Y (x, y), and fY (y) > 0, the concept of conditional density fX|YR =y (x) = fX,Y (x, y)/fY (y) is
introduced and the quantity P[X A|Y = y] is given meaning via A fX|Y =y (x, y) dx. While this
procedure works well in the restrictive case of absolutely continuous random vectors, we will
see how it is encompassed by a general concept of a conditional expectation. Since probability is
simply an expectation of an indicator, and expectations are linear, it will be easier to work with
expectations and no generality will be lost.
Two main conceptual leaps here are: 1) we condition with respect to a -algebra, and 2) we
view the conditional expectation itself as a random variable. Before we illustrate the concept in
discrete time, here is the definition.
Definition 9.1 (Conditional Expectation) Let G be a sub--algebra of F, and let X L1 be a

random variable. We say that the random variable is (a version of) the conditional expectation of
X with respect to G - and denote it by E[X|G] - if
1. L1 .
2. is G-measurable,
3. E[1A ] = E[X1A ], for all A G.
115
CHAPTER 9. CONDITIONAL EXPECTATION

Example 9.2 Suppose that (, F, P) is a probability space where = {a, b, c, d, e, f }, F = 2 and
P is uniform. Let X, Y and Z be random variables given by (in the obvious notation)

a b c d e f
a b c d e f
a b c d e f
and Z
, Y
X
3 3 3 3 2 2
2 2 1 1 7 7
1 3 3 5 5 7
We would like to think about E[X|G] as the average of X() over all which are consistent with
the our current information (which is G). For example, if G = (Y ), then the information contained
in G is exactly the information about the exact value of Y . Knowledge of the fact that Y = y does
not necessarily reveal the true , but certainly rules out all those for which Y () 6= y.
1
111 22 34
111 2 34
In our specific case, if we know that Y = 2, then = a or = b, and the expected value of X,
given that Y = 2, is 12 X(a) + 21 X(b) = 2. Similarly, this average equals 4 for Y = 1, and 6 for
Y = 7. Let us show that the random variable defined by this average, i.e.,

a b c d e f
,
2 2 4 4 6 6
satisfies the definition of E[X|(Y )], as given above. The integrability is not an issue (we are on
a finite probability space), and it is clear that is measurable with respect to (Y ). Indeed, the
atoms of (Y ) are {a, b}, {c, d} and {e, f }, and is constant over each one of those. Finally, we
need to check that
E[1A ] = E[X1A ], for all A (Y ),
which for an atom A translates into
() =
1
P[A] E[X1A ]
X( )P[{ }|A], for all A.
The moral of the story is that when A is an atom, part (3) of Definition 9.1 translates into a requirement that be constant on A with value equal to the expectation of X over A with respect to the
conditional probability P[|A]. In the general case, when there are no atoms, (3) still makes sense
and conveys the same message.
Btw, since the atoms of (Z) are {a, b, c, d} and {e, f }, it is clear that
(
3, {a, b, c, d},
E[X|(Z)]() =
6, {e, f }.
116

Look at the illustrations above and convince yourself that
E[E[X|(Y )]|(Z)] = E[X|(Z)].
A general result along the same lines - called the tower property of conditional expectation - will be
stated and proved below.
Our first task is to prove that conditional expectations always exist. When is finite (as explained
above) or countable, we can always construct them by averaging over atoms. In the general case,
a different argument is needed. In fact, here are two:
Proposition 9.3 (Conditional expectation - existence and a.s.-uniqueness) Let G be a sub-algebra G of F. Then
1. there exists a conditional expectation E[X|G] for any X L1 , and
2. any two conditional expectations of X L1 are equal P-a.s.
P ROOF (Uniqueness): Suppose that and both satisfy (1),(2) and (3) of Definition 9.1. Then
E[1A ] = E[ 1A ], for all A G.
For An = { n1 }, we have An G and so
E[1An ] = E[ 1An ] E[( + n1 )1An ] = E[1An ] + n1 P[An ].
Consequently, P[An ] = 0, for all n N, so that P[ > ] = 0. By a symmetric argument, we also
have P[ < ] = 0.
(Existence): By linearity, it will be enough to prove that the conditional expectation exists for
X L1+ .
1. A Radon-Nikodym argument. Suppose, first, that X 0 and E[X] = 1, as the general case
follows by additivity and scaling. Then the prescription
Q[A] = E[X1A ],
defines a probability measure on (, F), which is absolutely continuous with respect to P. Let QG
be the restriction of Q to G; it is trivially absolutely continuous with respect to the restriction PG of
P to G. The Radon-Nikodym theorem - applied to the measure space (, G, PG ) and the measure
QG PG - guarantees the existence of the Radon-Nikodym derivative
=
dQG
L1+ (, G, PG ).
dPG
For A G, we thus have

G
E[X1A ] = Q[A] = QG [A] = EP [1A ] = E[1A ].

where the last equality follows from the fact that 1A is G-measurable. Therefore, is (a version
of) the conditional expectation E[X|G].
117

1. An L2 -argument. Suppose, first, that X L2 . Let H be the family of all G-measurable
denote the closure of H in the topology induced by L2 -convergence. Being
elements in L2 . Let H
satisfies all the conditions of Problem 4.26 so that
a closed and convex (why?) subset of L2 , H
at the minimal L2 -distance from X (when X H,
we take = X). The same
there exists H
problem states that has the following property:
E[( )(X )] 0 for all H,

is a linear space, we have
and, since H
E[( )(X )] = 0, for all H.

A G, to conclude that
It remains to pick of the form = + 1A H,
E[X1A ] = E[1A ], for all A G.
Our next step is to show that is G-measurable (after a modification on a null set, perhaps).
there exists a sequence {n }nN such that n in L2 . By Corollary 4.20, n a.s.
Since H,
,
k
0
for some subsequence {nk }kN of {n }nN . Set = lim inf kN nk L ([, ], G) and =
1{| |<} , so that = , a.s., and is G-measurable.
We still need to remove the restriction X L2+ . We start with a general X L1+ and define
2
Xn = min(X, n) L
+ L+ . Let n = E[Xn |G], and note that E[n+1 1A ] = E[Xn+1 1A ]
E[Xn 1A ] = E[n 1A ]. It follows (just like in the proof of uniqueness above) that n n+1 , a.s.
We define = supn n , so that n , a.s. Then, for A G, the monotone-convergence theorem
implies that
E[X1A ] = lim E[Xn 1A ] = lim E[n 1A ] = E[1A ],
n
and it is easy to check that 1{<}
L1 (G)
is a version of E[X|G].
Remark 9.4 There is no canonical way to choose the version of the conditional expectation. We
follow the convention started with Radon-Nikodym derivatives, and interpret a statement such
at E[X|G], a.s., to mean that , a.s., for any version of the conditional expectation of X
with respect to G.
If we use the symbol L1 to denote the set of all a.s.-equivalence classes of random variables in
1
L , we can write:
E[|G] : L1 (F) L1 (G),
but L1 (G) cannot be replaced by L1 (G) in a natural way. Since X = X , a.s., implies that E[X|G] =
E[X |G], a.s. (why?), we consider conditional expectation as a map from L1 (F) to L1 (G)
E[|G] : L1 (F) L1 (G).
9.2
Properties
Conditional expectation inherits many of the properties from the ordinary expectation. Here
are some familiar and some new ones:
Proposition 9.5 (Properties of the conditional expectation) Let X, Y, {Xn }nN be random variables in L1 , and let G and H be sub--algebras of F. Then
118
1. (linearity) E[X + Y |G] = E[X|G] + E[Y |G], a.s.

2. (monotonicity) X Y , a.s., implies E[X|G] E[Y |G], a.s.
3. (identity on L1 (G)) If X is G-measurable, then X = E[X|G], a.s. In particular, c = E[c|G], for
any constant c R.
4. (conditional Jensens inequality) If : R R is convex and E[|(X)|] < then
E[(X)|G] (E[X|G]), a.s.
5. (Lp -nonexpansivity) If X Lp , for p [1, ], then E[X|G] Lp and
||E[X|G]||Lp ||X||Lp .
In particular,
E[|X| |G] |E[X|G]| a.s.
6. (pulling out whats known) If Y is G-measurable and XY L1 , then
E[XY |G] = Y E[X|G], a.s.
7. (L2 -projection) If X L2 , then = E[X|G] minimizes E[(X )2 ] over all G-measurable
random variables L2 .
8. (tower property) If H G, then
E[E[X|G]|H] = E[X|H], a.s..
9. (irrelevance of independent information) If H is independent of (G, (X)) then
E[X|(G, H)] = E[X|G], a.s.
In particular, if X is independent of H, then E[X|H] = E[X], a.s.
10. (conditional monotone-convergence theorem) If 0 Xn Xn+1 , a.s., for all n N and
Xn X L1 , a.s., then
E[Xn |G] E[X|G], a.s.
11. (conditional Fatous lemma) If Xn 0, a.s., for all n N, and lim inf n Xn L1 , then
E[lim inf X|G] lim inf E[Xn |G], a.s.
n
12. (conditional dominated-convergence theorem) If |Xn | Z, for all n N and some Z L1 ,

and if Xn X, a.s., then
E[Xn |G] E[X|G], a.s. and in L1 .
119

P ROOF Only some of the properties are proved in detail. The others are only commented upon,
since they are either similar to the other ones or otherwise not hard.
1. (linearity) E[(X + Y )1A ] = E[(E[X|G] + E[Y |G])1A ], for A G.
2. (monotonicity) Use A = {E[X|G] > E[Y |G]} G to obtain a contradiction if P[A] > 0.
3. (identity on L1 (G)) Check the definition.
4. (conditional Jensens inequality) Use the result of Lemma 4.22 which states that (x) =
supnN (an + bn x), where {an }nN and {bn }nN are sequences of real numbers.
5. (Lp -nonexpansivity) For p [1, ), apply conditional Jensens inequality with (x) = |x|p .
The case p = follows directly.
6. (pulling out whats known) For Y G-measurable and XY L1 , we need to show that
(9.1)
E[XY 1A ] = E[Y E[X|G]1A ], for all A G.
Let us prove a seemingly less general statement:

(9.2)
E[ZX] = E[ZE[X|G]], for all G-measurable Z with ZX L1 .
P
The statement (9.1) will follow from it by taking Z = Y 1A . For Z = nk=1 k 1Ak , (9.2) is
a consequence of the definition of conditional expectation and linearity. Let us assume that
both Z and X are nonnegative and ZX L1 . In that case we can find a non-decreasing
sequence {Zn }nN of non-negative simple random variables with Zn Z. Then Zn X L1
for all n N and the monotone convergence theorem implies that
E[ZX] = lim E[Zn X] = lim E[Zn E[X|G]] = E[ZE[X|G]].
n
Our next task is to relax the assumption X L1+ to the original one X L1 . In that case, the
Lp -nonexpansivity for p = 1 implies that
|E[X|G]| E[|X| |G] a.s., and so |Zn E[X|G]| Zn E[|X| |G] ZE[|X| |G].
We know from the previous case that
E[ZE[|X| |G]] = E[Z |X|], so that ZE[|X| |G] L1 .
We can, therefore, use the dominated convergence theorem to conclude that
E[ZE[X|G]] = lim E[Zn E[X|G]] = lim E[Zn X] = E[ZX].
n
Finally, the case of a general Z follows by linearity.

7. (L2 -projection) It is enough to show that XE[X|G] is orthogonal to all G-measurable L2 .
for that we simply note that for L2 , x G, we have
E[(X E[X|G])] = E[X] E[E[X|G]] = E[X] E[E[X|G]] = 0.
8. (tower property) Use the definition.
120

9. (irrelevance of independent information) We assume X 0 and show that
E[X1A ] = E[E[X|G]1A ], a.s. for all A (G, H).
(9.3)
Let L be the collection of all A (G, H) such that (9.3) holds. It is straightforward that L is
a -system, so it will be enough to establish (9.3) for some -system that generates (G, H).
One possibility is P = {G H : G G, H H}, and for G H P we use independence
of 1H and E[X|G]1G , as well as the independence of 1H and X1G to get
E[E[X|G]1GH ] = E[E[X|G]1G 1H ] = E[E[X|G]1G ]E[1H ] = E[X1G ]E[1H ]
= E[X1GH ]
10. (conditional monotone-convergence theorem) By monotonicity, E[Xn |G] L0+ (G), a.s.
The monotone convergence theorem implies that
E[1A ] = lim E[1A E[Xn |G]] = lim E[1A Xn ] = E[1A X], for all A G.
n
11. (conditional Fatous lemma) Set Yn = inf kn Xk , so that Yn Y = lim inf k Xk . By monotonicity,
E[Yn |G] inf E[Xk |G], a.s.,
kn
and the conditional monotone-convergence theorem implies that

E[Y |G] = lim E[Yn |G] lim inf E[Xn |G], a.s.
n
nN
12. (conditional dominated-convergence theorem) By the conditional Fatous lemma, we have

E[Z + X|G] lim inf E[Z + Xn |G], and E[Z X|G] lim inf E[Z Xn |G], a.s.,
n
and the a.s.-statement follows.

Problem 9.6
1. Show that the condition H G is necessary for the tower property to hold in general. (Hint:
Take = {a, b, c}.)
2. For X, Y L2 and a sub--algebra G of F, show that the following self-adjointness property
holds
E[X E[Y |G]] = E[E[X|G] Y ] = E[E[X|G] E[Y |G]].
3. Let H and G be two sub--algebras of F. Is it true that
H = G if and only if E[X|G] = E[X|H], a.s., for all X L1 ?
4. Construct two random variables X and Y in L1 such that E[X|(Y )] = E[X], a.s., but X and
Y are not independent.
9.3
Regular conditional distributions
Once we have a the notion of conditional expectation defined and analyzed, we can use it to define
other, related, conditional quantities. The most important of those is the conditional probability:
121
Definition 9.7 (Conditional probability) Let G be a sub--algebra of F. The conditional probability of A F, given G - denoted by P[A|G] - is defined by
P[A|G] = E[1A |G].
It is clear (from the conditional version of the monotone-convergence theorem) that
X
P[nN An |G] =
P[An |G], a.s.
(9.4)
nN
We can, therefore, think of the conditional probability as a countably-additive map from events
to (equivalence classes of) random variables A 7 P[A|G]. In fact, this map has the structure of a
vector measure:
Definition 9.8 (Vector Measures) Let (B, || ||) be a Banach space, and let (S, S) be a measurable
space. A map : S B is called a vector measure if
1. () = 0, and
2. for each pairwise disjoint sequence {An }nN in S, (n An ) =
in B converges absolutely).
nN (An )
Proposition 9.9 (Conditional probability as a vector measure) The

A 7 P[A|G] L1 is a vector measure with values in B = L1 .
(where the series
conditional
probability
P ROOF Clearly P[0|G] = 0, a.s. Let {An }nN be a pairwise-disjoint sequence in F. Then

P[An |G] 1 = E[|E[1An |G]|] = E[1An ] = P[An ],
L
and so

X

P[A
|G]

n
nN
which implies that
nN P[An |G]
N

X

P[An |G]
P[A|G]
n=1
L1
L1
nN
P[An ] = P[n An ] 1 < ,
converges absolutely in L1 . Finally, for A = nN An , we have

X

= E[
1An |G]
n=N +1
L1
= P[
n=N +1 An ] 0 as N .
It is tempting to try to interpret the map A 7 P[A|G]() as a probability measure for a fixed .
It will not work in general; the reason is that P[A|G] is defined only a.s., and the exceptional
sets pile up when uncountable families of events A are considered. Even if we fixed versions
122

P[A|G] L0+ , for each A F, the countable additivity relation P
(9.4) holds only almost surely so
there is no guarantee that, for a fixed , P[nN An |G]() = nN P[An |G](), for all pairwise
disjoint sequences {An }nN in F.
There is a way out of this predicament in certain situations, and we start with a description of
an abstract object that corresponds to a well-behaved conditional probability:
Definition 9.10 (Measurable kernels) Let (R, R) and (S, S) be measurable spaces. A map :
R S R is called a (measurable) kernel if
1. x 7 (x, B) is R-measurable for each B S, and
2. B 7 (x, B) is a measure on S for each x R.
Definition 9.11 (Regular conditional distributions) Let G be a sub--algebra of F, let (S, S) be

a measurable space, and let e : S be a random element in S. A kernel e|G : S [0, 1] is
called the regular conditional distribution of e, given G, if
e|G (, B) = P[e B|G](), a.s., for all B S.
Remark 9.12
1. When (S, S) = (, F), and e() = , the regular conditional distribution of e (if it exists) is
called the regular conditional probability. Indeed, in this case, e|G (, B) = P[e B|G] =
P[B|G], a.s.
2. It can be shown that regular conditional distributions not need to exist in general if S is too
large.
When (S, S) is small enough, however, regular conditional distributions can be constructed.
Here is what we mean by small enough:
Definition 9.13 (Borel spaces) A measurable space (S, S) is said to be a Borel space (or a nice
space) if it is isomorphic to a Borel subset of R, i.e., if there one-to-one map : S R such that both
and 1 are measurable.
Problem 9.14 Show that Rn , n N (together with their Borel -algebras) are Borel spaces. (Hint:
Show, first, that there is a measurable bijection : [0, 1] [0, 1] [0, 1] such that 1 is also
measurable. Use binary (or decimal, or . . . ) expansions.)
Remark 9.15 It can be show that any Borel subset of any complete and separable metric space is
a Borel space. In particular, the coin-toss space is a Borel space.
123
Proposition 9.16 (A criterion for existence of regular conditional distributions) Let G be a

sub--algebra of F, and let (S, S) be a Borel space. Any random element e : S admits a regular
conditional distribution.
P ROOF (*) Let us, first, deal with the case S = R, so that e = X is a random variable. Let Q be
a countable dense set in R. For q Q, consider the random variable P q , defined as an arbitrary
version of
P q = P[X q|G].
By redefining each P q on a null set (and aggregating the countably many null sets - one for
each q Q), we may suppose that P q () P r (), for q r, q, r Q, for all and that
limq P q () = 1 and limq P q () = 0, for all . For x R, we set
F (, x) =
inf
qQ,q>x
P q (),
so that, for each , F (, ) is a right-continuous non-decreasing function from R to [0, 1],

with limx F (, x) = 1 and limx F (, x) = 0, for all . Moreover, as an infimum of
countably many random variables, the map 7 F (, x) is a random variable for each x R.
By (the proof of) Proposition 6.40, for each , there exists a unique probability measure
e|G (, ) on R such that e|G (, (, x]) = F (, x), for all x R. Let L denote the set of all B B
such that
1. 7 e|G (, B) is a random variable, and
2. e|G (, B) is a version of P[X B|G].
It is not hard to check that L is a -system, so we need to prove that (1) and (2) hold for all B in
some -system which generates B(R). A convenient -system to use is P = {(, x] : x R}.
For B = (, x] P, we have e|G (, B) = F (, x), so that (1) holds. To check (2), we need to
show that F (x, ) = P[X x|G], a.s. This follows from the fact that
F (, x) = inf P q = lim P q = lim P[X q|G] = P[X x|G], a.s.,
q>x
qx
qx
by the conditional dominated convergence theorem.

Turning to the case of a general random element e which takes values in a Borel space (S, S),
we pick a one-to-one measurable map f : S R whose inverse 1 is also measurable. Then
X = (e) is a random variable, and so, by the above, there exists a kernel X|G : B(R) [0, 1]
such that
X|G (, A) = P[(e) A|G], a.s.
We define the kernel e|G : S [0, 1] by
e|G (, B) = X|G (, (B)).

Then, e|G (, B) is a random variable for each B S and for a pairwise disjoint sequence {Bn }nN
in S, we have
e|G (, n Bn ) = X|G (, (n Bn )) = X|G (, n (Bn ))
X
X
=
X|G (, (Bn )) =
e|G (, Bn ),
nN
nN
124

which shows that e|G is a kernel; we used the measurability of 1 to conclude that (Bn ) B(R)
and the injectivity of to ensure that {(Bn )}nN is pairwise disjoint. Finally, we need to show
that e|G (, B) is a version of the conditional probability P[e B|G]. By injectivity of , we have
P[e B|G] = P[(e) (B)|G] = X|G (, (B)) = e|G (, B), a.s.
Remark 9.17 Note that the regular conditional distribution is not unique, in general. Indeed, we
can redefine it arbitrarily (as long as it remains a kernel) on a set of the form N S S, where
P[N ] = 0, without changing any of its defining properties. This will, in these notes, never be an
issue.
One of the many reasons why regular conditional distributions are useful is that they sometimes
allow non-conditional thinking to be transferred to the conditional case:
Proposition 9.18 (Conditional expectation as a parametrized integral) Let X be an Rn -valued

n
random vector, let G
R be a sub--algebra of F, and let g : R R be a Borel function with the property
1
g(X) L . Then Rn g(x)X|G (, dx) is a G-measurable random variable and
Z
E[g(X)|G] =
g(x)X|G (, dx), a.s.
Rn
P ROOF When g = 1B , for B Rn , the statement follows by the very definition of the regular
condition distribution. For the general case, we simply use the standard machine.
Just like we sometimes express the distribution of a random variable or a vector in terms of its
density, cdf or characteristic function, we can talk about the conditional density, conditional cdf or
the conditional characteristic function. All of those will correspond to the case covered in Proposition 9.16 and all conditional distributions will be assumed to be regular. For x = (x1 , . . . , xn )
and y = (y1 , . . . , yn ), y n x means y1 x1 , . . . , yn xn .
Definition 9.19 (Other regular conditional quantities) Let X : Rn be a random vector, let
G be a sub--algebra of F, and let X|G : B(Rn ) [0, 1] be the regular conditional distribution
of X given G.
1. The (regular) conditional cdf of X, given G is the map F : Rn [0, 1], given by
F (, x) = X|G (, {y Rn : y n x}), for x Rn ,
2. A map fX|G : Rn [0, ) is called the conditional density of X with respect to G if
a) fX|G (, ) is Borel measurable for all ,
b) fX|G (, x) is G-measurable for each x Rn , and

R
c) B fX|G (, x) dx = X|G (, B), for all and all B B(Rn ),
125
3. The conditional characteristic function of X, given G is the map X|G : Rn C,

given by
Z
X|G (, t) =
eitx X|G (, dx), for t Rn and .
Rn
To illustrate the utility of the above concepts, here is a versatile result (see Example 9.23 below):
Proposition 9.20 (Regular conditional characteristic funtions and independence) Let X be a

random vector in Rn , and let G be a sub--algebra of F. The following two statements are equivalent:
1. There exists a (deterministic) function : Rn C such that for P-almost all ,
X|G (, t) = (t), for all t Rn .
2. (X) is independent of G.
Moreover, whenever the two equivalent statements hold, is the characteristic function of X.
P ROOF (1) (2). By Proposition 9.18, we have X|G (, t) = E[eitX |G], a.s. If we replace X|G by
, multiplying both sides by a bounded G-measurable random variable Y and take expectations,
we get
(t)E[Y ] = E[Y eitX ].
In particular, for Y = 1 we get (t) = E[eitX ], so that
(9.5)
E[Y eitX ] = E[Y ]E[eitX ],
for all G-measurable and bounded Y , and all t Rn . For Y of the form Y = eisZ , where Z is
a G-measurable random variable, relation (9.5) and (a minimal extension of) part (1) of Problem
7.41, we conclude that X and Z are independent. Since Z is arbitrary and G-measurable, X and
G are independent.
(2) (1). If (X) is independent of G, so is eitX , and so, the irrelevance of independent
information property of conditional expectation implies that
(t) = E[eitX ] = E[eitX |G] = X|G (, t), a.s.
One of the most important cases used in practice is when a random vector (X1 , . . . , Xn ) admits a
density and we condition on the -algebra generated by several of its components. To make the
notation more intuitive, we denote the first d components (X1 , . . . , Xd ) by X o (for observed) and
the remaining n d components (Xd+1 , . . . , Xn ) by X u (for unobserved).
126
Proposition 9.21 (Conditional densities) Suppose that the random vector X = (X o , X u ) =

(X1 , . . . , Xd , Xd+1 , . . . , Xn ) admits a density fX : Rn [0, ) and that the -algebra G = (X o )
is generated by the random vector X o = (X1 , . . . , Xd ), for some d {1, . . . , n 1}. Then, for
X u = (Xd+1 , . . . , Xn ), there exists a conditional density fX u |G : Rnd [0, ), of X u given
G, and (a version of it) is given by
fX
|G (, x
)=
fX (X o (),xu )
,
f (X o (),y) dy
Rnd X
f0 (xu ),
R
Rnd
f (X o , y) dy > 0,
otherwise,
for x Rnd and , where f0 : Rnd R is an arbitrary density function.

P ROOF First, we note that fX u |G is constructed from the jointly Borel-measurable function fX and
the random vector X o in an elementary way, and is, thus, jointly measurable in G B(Rnd ). It
remains to show that
Z
fX u |G (, xu ) dxu is a version of P[X u A|G], for all A B(Rnd ).
A
Equivalently, we need to show that

Z
o
E[1{X Ao }
fX u |G (, xu ) dxu ] = E[1{X o Ao } 1{X u Au } ],
Au
for all Ao B(Rd ) and Au B(Rnd ).

R
Fubinis theorem, and the fact that fX o (xo ) = Rnd f (xo , y) dy is the density of X o yield
Z
Z
u
u
u
o
fX |G (, x ) dx ] =
E[1{X o Ao } fX u |G (, xu )] dxo
E[1{X Ao }
u
u
A
ZA Z
fX u |G (xo , xu )fX o (xo ) dxo dxu
=
u Ao
A
Z Z
fX (xo , xu ) dxo dxu
=
Ao
Au
= P[X Ao , X u Au ].
The above result expresses a conditional density, given G = (X o ), as a (deterministic) function of

X o . Such a representation is possible even when there is no joint density. The core of the argument
is contained in the following problem:
Problem 9.22 Let X be a random vector in Rd , and let G = (X) be the -algebra generated by X.
Then, a random variable Z is G-measurable if and only if there exists a Borel function f : Rd R
with the property that Z = f (X).
Let X o be a random vector in Rd . For X L1 the conditional expectation E[X|(X o )] is (X o )measurable, so there exists a Borel function f : Rd R such that E[X|(X o )] = f (X o ), a.s. Note
that f is uniquely defined only up to X o -null sets. The value f (xo ) at xo Rd is usually denoted
by E[X|X o = xo ].
127

Example 9.23 (Conditioning normals on their components) Let X = (X o , X u ) Rd Rnd be
a multivariate normal random vector with mean = (o , u ) and the variance-covariance matrix
X
T ], where X
= X . A block form of the matrix is given by
= E[X

oo ou
,
=
uo uu
Where
o (X
o )T ] Rdd
oo = E[X
o (X
u )T ] Rd(nd)
ou = E[X
u (X
o )T ] R(nd)d
uo = E[X
u (X
u )T ] R(nd)(nd) .
uu = E[X
We assume that oo is invertible. Otherwise, we can find a subset of components of X o whose
variance-covariance matrix in invertible and which generate the same -algebra (why?). The mau
o o T
that the random vectors
trix A = uo 1
oo ohas the property that E[(X AX )(X ) ] = 0, i.e.,
o
o
AX
and X
are uncorrelated. We know, however, that X
= (X
o, X
u ) is a Gaussian random
X
o AX
o is independent of X
o . It follows from Proposition
vector, so, by Problem 7.41, part (3), X
o
o
AX
, given G = (X
o ) is deterministic
9.20 that the conditional characteristic function of X
and given by
u
o
nd
E[eit(X AX ) |G] = X
.
u AX
o (t), for t R
o is G-measurable, we have
Since AX
u
E[eitX |G] = eit eitAX e 2 t
T t
, for t Rnd .
= E[(X
u AX
o )(X
u AX
o )T ]. A simple calculation yields that, conditionally on G,
where
X u is multivariate normal with mean X u |G and variance-covariance matrix X u |G given by
X u |G = o + A(X o o ), X u |G = uu uo 1
oo ou .
Note how the mean gets corrected by a multiple of the difference between the observed value
X o and its (unconditional) expected value. Similarly, the variance-covariance matrix also gets
o
corrected by uo 1
oo ou , but this quantity does not depend on the observation X .
Problem 9.24 Let (X1 , X2 ) be a bivariate normal vector with Var[X1 ] > 0. Work out the exact
form of the conditional distribution of X2 , given X1 in terms of i = E[Xi ], i2 = Var[Xi ], i = 1, 2
and the correlation coefficient = corr(X1 , X2 ).
9.4
Additional Problems
Problem 9.25 (Conditional expectation for non-negative random variables) A parallel definition
of conditional expectation can be given for random variables in L0+ . For X L0+ , we say that Y is
a conditional expectation of X with respect to G - and denote it by E[X|G] - if
(a) Y is G-measurable and [0, ]-valued, and
(b) E[Y 1A ] = E[X1A ] [0, ], for A G.
128

Show that
1. E[X|G] exists for each X L0+ .
2. E[X|G] is unique a.s. (Hint: The argument in the proof of Proposition 9.3 needs to be modified before it can be used.)
3. E[X|G] no longer necessarily exists for all X L0+ if we insist that E[X|G] < , a.s., instead
of E[X|G] [0, ], a.s.
Problem 9.26 (How to deal with the independent component) Let f : R2 R be a bounded
Borel-measurable function, and let X and Y be independent random variables. Define the function g : R R by
g(y) = E[f (X, y)].
Show that the function g is Borel-measurable, and that
E[f (X, Y )|Y = y] = g(y), Y a.s.
Problem 9.27 (Some exercises in conditional probability)
1. Let X, Y1 , Y2 be random variables. Show that the random vectors (X, Y1 ) and (X, Y2 ) have
the same distribution if and only if P[Y1 B|(X)] = P[Y2 B|(X)], for all B B(R).
2. Let {Xn }nN be a sequence of non-negative integrable random variables, and let {Fn }nN
P
be sub--algebras of F. Show that Xn 0 if E[Xn |Fn ] 0. Does the converse hold? (Hint:
P
Prove that for Xn L0+ , we have Xn 0 if and only if E[min(Xn , 1)] 0.)
3. Let G be a complete sub--algebra of F. Suppose that for X L1 , E[X|G] and X have

the same distribution. Show that X is G-measurable. (Hint: Use the conditional Jensens
inequality.)
Problem 9.28 (A characterization of G-measurability) Let (, F, P) be a complete probability space
and let G be a sub--algebra of F. Show that for a random variable X L1 the following two
statements are equivalent:
1. X is G-measurable.
2. For all L , E[X] = E[XE[|G]].
Problem 9.29 (Conditioning a part with respect to the sum) Let X1 , X2 , . . . be a sequence of iid
r.v.s with finite first moment, and let Sn = X1 + X2 + + Xn . Define G = (Sn ).
1. Compute E[X1 |G].
2. Supposing, additionally, that X1 is normally distributed, compute E[f (X1 )|G], where f :
R R is a Borel function such that f (X1 ) L1 .
129
Chapter
10
Discrete Martingales
10.1
Discrete-time filtrations and stochastic processes
One of the uses of -algebras is to single out the subsets of to which probability can be assigned.
This is the role of F. Another use, as we have seen when discussing conditional expectation, is
to encode information. The arrow of time, as we perceive it, points from less information to more
information. A useful mathematical formalism is the one of a filtration.
Definition 10.1 (Filtered probability spaces) A filtration is a sequence {Fn }nN0 , where N0 =
N {0}, of sub--algebras of F such that Fn Fn+1 , for all n N0 . A probability space with a
filtration - (, F, {Fn }nN0 , P) - is called a filtered probability space.
We think of n N0 as the time-index and of Fn as the information available at time n.
Definition 10.2 (Discrete-time stochastic process) A (discrete-time) stochastic process is a
sequence {Xn }nN0 of random variables.
A stochastic process is a generalization of a random vector; in fact, we can think of a stochastic processes as an infinite-dimensional random vector. More precisely, a stochastic process is a
random element in the space RN0 of real sequences. In the context of stochastic processes, the
sequence (X0 (), X1 (), . . . ) is called a trajectory of the stochastic process {Xn }nN0 . This dual
view of stochastic processes - as random trajectories (sequences) or as sequences of random variables - can be supplemented by another interpretation: a stochastic process is also a map from the
product space N0 into R.
Definition 10.3 (Adapted processes) A stochastic process {Xn }nN0 is said to be adapted with
respect to the filtration {Fn }nN0 if Xn is Fn -measurable for each n N0 .
130
CHAPTER 10. DISCRETE MARTINGALES

Intuitively, the process {Xn }nN0 is adapted with respect to the filtration {Fn }nN0 if its value
Xn is fully known at time n (assuming that the time-n information is given by Fn ).
The most common way of producing filtrations is by generating them from stochastic processes. More precisely, for a stochastic process {Xn }nN0 , the filtration {FnX }nN0 , given by
FnX = (X0 , X1 , . . . , Xn ), n N0 ,
is called the filtration generated by {Xn }nN0 . Clearly, X is always adapted to the filtration generated by X.
10.2
Martingales
Definition 10.4 ((Sub-, super-) martingales) Let {Fn }nN0 be a filtration. A stochastic process
{Xn }nN0 is called an {Fn }nN0 -supermartingale if
1. {Xn }nN0 is {Fn }nN0 -adapted,
2. Xn L1 , for all n N0 , and
3. E[Xn+1 |Fn ] Xn , a.s., for all n N0 .
A process {Xn }nN0 is called a submartingale if {Xn }nN0 is a supermartingale. A martingale
is a process which is both a supermartingale and a submartingale at the same time, i.e., for which the
equality holds in (3).
Remark 10.5 Very often, the filtration {Fn }nN0 is not explicitly mentioned. Then, it is often clear
from the context, i.e., the existence of an underlying filtration {Fn }nN is assumed throughout.
Alternatively, if no filtration is pre-specified, the filtration {FnX }nN0 , generated by {Xn }nN0 is
used. It is important to remember, however, that the notion of a (super-, sub-) martingale only
makes sense in relation to a filtration.
The fundamental examples of martingales are (additive or multiplicative) random walks:
Example 10.6
1. An additive random walk. Let {n }nN be a sequence of iid random variables with n L1
and E[n ] = 0, for all n N. We define
X0 = 0, Xn =
n
X
k=1
k , for n N.
The process {Xn }nN0 is a martingale with respect to the filtration {FnX }nN0 generated by
it (which is the same as (1 , . . . , n )). Indeed, Xn L1 (FnX ) for all n N and
E[Xn+1 |Fn ] = E[n+1 + Xn |Fn ] = Xn + E[n+1 |Fn ] = Xn + E[n+1 ] = Xn , a.s.,
where we used the irrelevance of independent information-property of conditional expectation (in this case n+1 is independent of FnX = (X0 , . . . , Xn ) = (1 , . . . , n )).
It is easy to see that if {n }nN are still iid, but E[n ] > 0, then {Xn }nN0 is a submartingale.
When E[n ] < 0, we get a supermartingale.
131

2. A multiplicative random walk. Let {n }nN be an iid sequence in L1 such that E[n ] = 1. We
define
n
Y
X0 = 1, Xn =
k , for n N.
k=1
The process {Xn }nN0 is a martingale with respect to the filtration {FnX }nN0 generated it.
Indeed, Xn L1 (FnX ) for all n N and
E[Xn+1 |Fn ] = E[n+1 Xn |Fn ] = Xn E[n+1 |Fn ] = Xn E[n+1 ] = Xn , a.s.,
where, in addition to the irrelevance of independent information we also used pulling

our whats known.
Is it true that, if E[n ] > 1, we get a submartingale and that if E[n ] < 1, we get a supermartingale?
3. Walds martingales Let {n }nN be an independent sequence, and let n (t) be the characteristic function of n . Assuming that n (t) 6= 0, for n N, the previous example implies that
the process {Xn }nN0 , defined by
X0t = 1, Xnt =
n
Y
eitk
k (t) ,
k=1
n N,
is a martingale. Actually, it is complex-valued, so it would be better to say that its real and
imaginary parts are both martingales. This martingale will be important in the study of
hitting times of random walks.
4. Lvy martingales. For X L1 , we define
Xn = E[X|Fn ], for n N0 .
The tower property of conditional expectation implies that {Xn }nN0 is a martingale.
5. An urn scheme. An urn contains b black and w white balls on day n = 0. On each subsequent
day, a ball is chosen (each ball in the urn has the same probability of being picked) and then
put back together with another ball of the same color. Therefore, at the end of day n, here
are n + b + w balls in the urn. Let Bn denote the number of black balls in the urn at day n,
and let define the process {Xn }nN0 by
Xn =
Bn
b+w+n ,
n N0 ,
to be the proportion of black balls in the urn at time n. Let {Fn }nN0 denote the filtration
generated by {Xn }nN0 . The conditional probability - given Fn - of picking a black ball at
time n is Xn , i.e.,
P[Bn+1 = Bn + 1|Fn ] = Xn and P[Bn+1 = Bn |Fn ] = 1 Xn .
Therefore,
E[Xn+1 |Fn ] = E[Xn+1 1{Bn+1 =Bn } |Fn ] + E[Xn+1 1{Bn+1 =Bn +1} |Fn ]
Bn +1
Bn
1{Bn+1 =Bn } |Fn ] + E[ b+w+n+1
1{Bn+1 =Bn +1} |Fn ]
= E[ b+w+n+1
Bn
b+w+n+1 (1
Xn ) +
Bn +1
b+w+n+1 Xn
132
Bn (1Xn )+(Bn +1)Xn

b+w+n+1
= Xn .

How does this square with your intuition? Should not a high number of black balls translate
into a high probability of picking a black ball? This will, in turn, only increase the number
of black balls with high probability. In other words, why is {Xn }nN0 not a submartingale
b
(which is not a martingale), at least for large b+w
?
To get some feeling for the definition, here are several simple exercises:
Problem 10.7
1. Let {Xn }nN0 be a martingale. Show that E[Xn ] = E[X0 ], for all n N0 . Give an example of
a process {Yn }nN0 with Yn L1 , for all n N0 which is not a martingale, but E[Yn ] = E[Y0 ],
for all n N0 .
2. Let {Xn }nN0 be a martingale. Show that
E[Xm |Fn ] = Xn , for all m > n.
3. Let {Xn }nN0 be a submartingale, and let : R R be a convex function such that (Xn )
L1 , for all n N0 . Show that {(Xn )n }nN0 n N0 is a submartingale, provided that either
a) is nondecreasing, or
b) {Xn }nN0 is a martingale.
In particular, if {Xn }nN is a submartingale, so are {Xn+ }nN0 and {eXn }nN0 . (Hint: Use
conditional Jensens inequality.)
10.3
Predictability and martingale transforms
Definition 10.8 (Predictable processes) A process {Hn }nN is said to be predictable with respect
to the filtration {Fn }nN0 if Hn is Fn1 -measurable for n N.
A process is predictable if you can predict its tomorrows value today. We often think of predictable processes as strategies: let {n }nN be a sequence of random variables which we interpret
as gambles. At time n we can place a bet of Hn dollars, thus realizing a gain/loss of Hn n . Note
that a negative Hn is allowed - the player wins money if n < 0 and loses if n > 0 in that case.
If {Fn }nN0 is a filtration generated by the gambles, i.e., F0 = {, } and Fn = {1 , . . . , n }, for
n N, then Hn Fn1 , so that it does not use any information about n : we are allowed to adjust
our bet according to the outcomes of previous gambles, but we dont know the outcome of n until
after the bet is placed. Therefore, the sequence {Hn }nN is a predictable sequence with respect to
{Fn }nN0 .
Problem 10.9 Characterize predictable submartingales and predictable martingales. (Note: To
comply with the setting in which the definition of predictability is given (processes defined on N
and not on N0 ), simply discard the value at 0.)
133
Definition 10.10 (Martingale transforms) Let {Fn }nN0 be a filtration and let {Xn }nN0 be a process adapted to {Fn }nN0 . The stochastic process {(H X)n }nN0 , defined by
(H X)0 = 0, (H X)n =
n
X
k=1
Hk (Xk Xk1 ), for n N,
is called the martingale transform of X by H.
Remark 10.11
1. The process {(H X)n }nN0 is called the martingale transform of X, even if neither H nor X
is a martingale. It is most often applied to a martingale X, though - hence the name.
2. In terms of the gambling interpretation given above, X plays the role of the cumulative gain
(loss) when a $1-bet is placed each time:
X0 = 0, Xn =
n
X
k=1
k , where k = Xn Xn1 , for n N.
If we insist that {n }nN is a sequence of fair bets, i.e., that there are no expected gains/losses
in the n-th bet, even after we had the opportunity to learn from the previous n 1 bets, we
arrive to the condition
E[n |Fn1 ] = 0, i.e., that {Xn }nN0 is a martingale.
The following proposition states that no matter how well you choose your bets, you cannot make
(or loose) money by betting on a sequence of fair games. A part of result is stated for submartingales; this is for convenience only. The reader should observe that almost any statement about
submartingales can be turner into a statement about supermartingales by a simple change of sign.
Proposition 10.12 (Stability of martingales under martingale transforms) Let {Xn }nN0 be
adapted, and let {Hn }nN be predictable. Then, the martingale transform H X of X by H is
1. a martingale, provided that {Xn }nN0 is a martingale and Hn (Xn Xn1 ) L1 , for all n N,
2. a submartingale, provided that {Xn }nN0 is a submartingale, Hn 0, a.s., and Hn (Xn
Xn1 ) L1 , for all n N.
P ROOF Just check the definition and use properties of conditional expectation.
Remark 10.13 The martingale transform is the discrete-time analogue of the stochastic integral.
Note that it is crucial that H be predictable if we want a martingale transform of a martingale to
be a martingale. Otherwise, we just take Hn = sgn(Xn Xn1 ) Fn and obtain a process which
is not a martingale unless X is constant. This corresponds to a player who knows the outcome of
the game before the bet is placed and places the bet of $ 1 which is guaranteed to win.
134
10.4
Stopping times
Definition 10.14 (Random and stopping times) A random variable T with values in N0 {}
is called a random time. A random time is said to be a stopping time with respect to the filtration
{Fn }nN0 if
{T n} Fn , for all n N.
Remark 10.15
1. Stopping times are simply random instances with the property that at every instant you
can answer the question Has T already happened? using only the currently-available
information.
2. The additional element + is used as a placeholder for the case when T does not happen.
Example 10.16
1. Constant (deterministic) times T = m, m N0 {} are obviously stopping time. The set
of all stopping times can be thought of as an enlargement of the set of time-instances. The
meaning of when Red Sox win the World Series again is clear, but it does not correspond
to a deterministic time.
2. Let {Xn }nN0 be a stochastic process adapted to the filtration {Fn }nN0 . For a subset B
B(R), we define the random time TB by
TB = min{n N0 : Xn B}.
Tb is called the hitting time of the set B and is a stopping time. Indeed,
{TB n} = {X0 B} {X1 B} {Xn B} Fn .
3. Let {nP
}nN be an iid sequence of coin tosses, i.e. P[i = 1] = P[i = 1] = 21 , and let
Xn = nk=1 k be the corresponding random walk. For N N, let S be the random time
defined by
S = max{n N : Xn = 0}.
S is called the last visit time to 0 before time N N. Intuitively, S is not a stopping time
since, in order to know whether S had already happened at time m < N , we need to know
that Xk 6= 0, for k = m + 1, . . . , N , and, for that, we need the information which is not
contained in Fm . We leave it to the reader to make this comment rigorous.
Stopping times have good stability properties, as the following proposition shows. All stopping times are with respect to an arbitrary, but fixed filtration {Fn }nN0 .
135
Proposition 10.17 (Properties of stopping times)

1. A random time T is a stopping time if and only if the process Xn = 1{nT } is {Fn }nN0 -adapted.
2. If S and T are stopping times, then so are S + T , max(S, T ), min(S, T ).
3. Let {Tn }nN be a sequence of stopping times such that T1 T2 . . . , a.s. Then T = supn Tn
is a stopping time.
4. Let {Tn }nN be a sequence of stopping times such that T1 T2 . . . , a.s. Then T = inf n Tn is
a stopping time.
P ROOF
1. Immediate.
2. Let us show that S + T is a stopping time and leave the other two to the reader:
{S + T n} = nk=0 ({S k} {T n k}) Fn .
3. For m N0 , we have {T m} = nN {Tn m} Fm .
4. For m N0 , we have {Tn m} = {Tn < m}c = {Tn m 1}c Fm1 . Therefore,
{T m} = {T < m + 1} = {T m + 1}c = nN {Tn m + 1}c Fm .
Stopping times are often used to produce new processes from old ones. The most common
construction runs the process X until time T and after that keeps it constant and equal to its value
at time T . More precisely:
Definition 10.18 (Stopped processes) Let {Xn }nN0 be a stochastic process, and let T be a stopping
time. The process {Xn }nN0 stopped at T , denoted by {XnT }nN0 is defined by
XnT () = XT ()n () = Xn ()1{nT ()} + XT () 1{n>T ()} .
The (sub)martingale property is stable under stopping:
Proposition 10.19 (Stability under stopping) Let {Xn }nN0 be a (sub)martingale, and let T be a
stopping time. Then the stopped process {XnT }nN is also a (sub)martingale.
P ROOF Let {Xn }nN0 be a (sub)martingale. We note that the process Kn = 1{nT } , is predictable,
non-negative and bounded, so its martingale transform (K X) is a (sub)martingale. Moreover,
(K X)n = XT n X0 = XnT X0 ,
and so, X T is a (sub)martingale, as well.
136
10.5
Convergence of martingales
A judicious use of a predictable processes in a martingale transform yields the following important result:
Theorem 10.20 (Martingale convergence) Let {Xn }nN0 be a martingale such that
sup E[|Xn |] < .
nN0
Then, there exists a random variable X L1 (F) such that Xn X, a.s.

P ROOF We pick two real numbers a < b and define two sequences of stopping times as follows:
T0 = 0,
S1 = inf{n 0 : Xn a},
S2 = inf{n T1 : Xn a},
T1 = inf{n S1 : Xn b}
T2 = inf{n S2 : Xn b}, etc.
In words, let S1 be the first time X falls under a. Then, T1 is the first time after S1 when X exceeds
b, etc. We leave it to the reader to check that {Tn }nN and {Sn }nN are stopping times. These
two sequences of stopping times allow us to construct a predictable process {Hn }nN which takes
values in {0, 1}. Simply, we buy low and sell high:
(
X
1, Sk < n Tk for some k N,
Hn =
1{Sk <nTk } =
0, otherwise.
kN
Let Una,b be the number of completed upcrossings by time n, i.e., the process defined by
Una,b = inf{k N : Tk n}
A bit of accounting yields:
(H X)n (b a)Una,b (Xn a) .
(10.1)
Indeed, the total gains from the strategy H can be split into two components. First, every time a
passage from below a to above b is completed, we pocket at least (b a). After that, if X never
falls below a again, H remains 0 and our total gains exceeds (b a)Una,b , which, in turn, trivially
dominates (b a)Una,b (Xn a) . The other possibility is that after the last upcrossing, the
process does reach the value below a at a certain point. The last upcrossing already happened, so
the process never hits a value above b after that; it may very well happen that we lose on this
transaction. The loss is overestimated by (Xn a) .
Then the inequality (10.1) and fact that the martingale transform (by a bounded process) of a
martingale is a martingale yield
E[Una,b ]
1
ba E[(H
X)n ] +
1
ba E[(Xn
137
a) ]
E[|Xn |]+|a|
.
ba

a,b
a,b
Let U
be the total number of upcrossings, i.e., U
= limn Una,b . Using the monotone convergence theorem, we get the so-called upcrossings inequality
a,b
E[U
]
|a|+supnN0 E[|Xn |]
ba
a,b
a,b
The assumption that supnN0 E[|Xn |] < , implies that E[U
] < and so P[U
< ] = 1. In
words, the number of upcrossings is almost surely finite (otherwise, we would be able to make
money by betting on an unfair game).
a,b
It remains to use the fact that U
< , a.s., to deduce that {Xn }nN0 converges. First of all,
by passing to rational numbers and taking countable intersections of probability-one sets, we can
assert that
a,b
P[U
< , for all a 0.
n
This can be strengthened to

P[lim inf Xn < a 0,
n
which is, however, a contradiction since, on the event {lim inf n Xn < a < b < lim supn Xn }, the
process X completes infinitely many upcrossings.
a.s.
We conclude that there exists an [, ]-valued random variable X such that Xn X .
a.s.
In particular, we have |Xn | |X |, and Fatous lemma yields
E[|X |] lim inf E[|Xn |] sup E[|Xn |] < .
n
Some, but certainly not all, results about martingales can be transferred to submartingales (supermartingales) using the following proposition:
Proposition 10.21 (Doob-Meyer decomposition) Let {Xn }nN0 be a submartingale. Then, there
exists a martingale {Mn }nN0 and a predictable process {An }nN (with A0 = 0 adjoined) such that
An L1 , An An+1 , a.s., for all n N0 and
Xn = Mn + An , for all n N0 .
P ROOF Define
An =
n
X
k=1
E[(Xk Xk1 )|Fk1 ],
n N.
Then {An }nN is clearly predictable and An+1 An , a.s., thanks to the submartingale property of
X. Finally, set Mn = Xn An , so that
E[Mn Mn1 |Fn1 ] = E[Xn Xn1 (An An1 )|Fn1 ]
= E[Xn Xn1 |Fn1 ] (An An1 ) = 0.

138
Corollary 10.22 (Submartingale convergence) Let {Xn }nN0 be a submartingale such that
sup E[Xn+ ] < .
nN0
Then, there exists a random variable X L1 (F) such that Xn X, a.s.

P ROOF Let {Mn }nN0 and {An }nN be as in Proposition 10.21. Since Mn = Xn An Xn ,
a.s., we have E[Mn+ ] E[Xn+ ] supn E[Xn+ ] < . Finally, since E[Mn ] = E[Mn+ ] E[Mn ]
supn E[Xn+ ] E[M0 ] < , we have
sup E[|Mn |] < .
nN
a.s.
Therefore, Mn M , for some M L1 . Since {An }nN is non-negative and non-decreasing,

there exists A L0+ such that An A 0, a.s., and so
a.s.
X n X = M + A .
It remains to show that X L1 , and for that, it suffices to show that E[A ] < . Since E[An ] =
E[Xn ] E[Mn ] C = supn E[Xn+ ] + E[|M0 |] < , monotone convergence theorem yields E[A ]
C < .
Remark 10.23 Corollary 10.22 - or the simple observation that E[|Xn |] = 2E[Xn+ ] E[X0 ] when
E[Xn ] = E[X0 ] - implies that it is enough to assume supn E[Xn+ ] < in the original martingaleconvergence theorem (Theorem 10.20).
Corollary 10.24 (Convergence of non-negative supermartingales) Let {Xn }nN0 be a nonnegative supermartingale (or a non-positive submartingale). Then there exists a random variable
X L1 (F) such that Xn X, a.s.
To convince yourself that things can go wrong if the boundedness assumptions are not met, here
is a problem:
Problem 10.25 Give an example of a submartingale {Xn }nN with the property that Xn ,
a.s. and E[Xn ] . (Hint: Use the Borel-Cantelli lemma.)
10.6
Additional problems
Problem 10.26 (Combining (super)martingales) In 1., 2. and 3. below, {Xn }nN0 and {Yn }nN0
are martingales. In 4., they are only supermartingales.
1. Show that the process {Zn }nN0 given by Zn = Xn Yn = max{Xn , Yn } is a submartingale.
2. Give an example of {Xn }nN0 and {Yn }nN0 such that {Zn }nN0 (defined above) is not a
martingale.
139

3. Does the product {Xn Yn }nN0 have to be a martingale? A submartingale? A supermartingale? (Provide proofs or counterexamples).
4. Let T be an {Fn }nN0 -stopping time (and remember that {Xn }nN0 and {Yn }nN0 are supermartingales). Show that the process {Zn }nN0 , given by
(
Xn (), n < T ()
Zn () =
Yn (), n T (),
is a supermartingale, provided that XT YT , a.s. (Note: This result is sometimes called the
switching principle. It says that if you switch from one supermartingale to a smaller one at a
stopping time, the resulting process is still a supermartingale.)
Problem 10.27 (An urn model) An urn contains B0 N black and W0 N white balls at time 0.
At each time we draw a ball (each ball in the urn has the same probability of being picked), throw
it away, and replace it with C N balls of the same color. Let Bn denote the number of black balls
at time n, and let Xn denote the proportion of black balls in the urn.
1. Show that there exists a random variable X such that
Lp
Xn X, for all p [1, ).

2. Find an expression for P[Bn = k], k = 1, . . . , n + 1, n N0 , when B0 = W0 = 1, C = 2, and
use it to determine the distribution of X in that case. (Hint: Guess the form of the solution
for n = 1, 2, 3, and prove that you are correct for all n.)
Problem 10.28 (Stabilization of integer-valued submartingales)
1. Let {Mn }nN0 be an integer-valued (i.e., P[Mn Z] = 1, for n N0 ) submartingale bounded
from above. Show that there exists an N0 -valued random variable T with the property that
n N0 , Mn = MT on {T n}, a.s.
2. Can such T always be found in the class of stopping times? (Note: This is quite a bit harder
than the other two parts.)
3. Let {Xn }nN0 be a simple biased random walk, i.e.,
X0 = 0, Xn =
n
X
k=1
k , n N,
where {n }nN are iid with P[1 = 1] = p and P[1 = 1] = 1 p, for some p (0, 1). Under
the assumption that p 12 , show that X hits any nonnegative level with probability 1, i.e.,
that for a N, we have P[a < ] = 1, where a = inf{n N : Xn = a}.
Problem 10.29 (An application to gambling) Let {n }nN0 be an iid sequence with P[n = 1] =
1 P[n = 1] = p ( 21 , 1). We interpret {n }nN0 as outcomes of a series of gambles. A gambler
starts with Z0 > 0 dollars, and in each play wagers a certain portion of her wealth. More precisely,
the wealth of the gambler at time n N is given by
Zn = Z0 +
n
X
k=1
140
C k k ,

where {Cn }nN0 is a predictable process such that Ck [0, Zk1 ), for k N. The goal of the gambler is to maximize the return on her wealth, i.e., to choose a strategy {Cn }nN0 such that the
expectation T1 E[log(ZT /Z0 )], where T N is some fixed time horizon, is the maximal possible.
(Note: It makes sense to call the random variable R such that ZT = Z0 eRT the return, and, consequently E[R] = T1 E[log(ZT /Z0 )], the expected return. Indeed, if you put Z0 dollars in a bank and
accrue (a compound) interest with rate R (0, ), you will get Z0 eRT dollars after T years. In
our case, R is not deterministic anymore, but the interpretation still holds.)
1. Define = H( 21 )H(p), where H(p) = p log p(1p) log(1p) and show that the process
{Wn }nN0 given by
Wn = log(Zn ) n, for n N0
is a supermartingale. Conclude that
E[log(ZT )] log(Z0 ) + T,
for any choice of {Cn }nN0 .
2. Show that the upper bound above is attained for some strategy {Cn }nN0 .
(Note: The quantity H(p) is called the entropy of the distribution of 1 . This problem shows how
it appears naturally in a gambling-theoretic context: the optimal rate of return equals to excess
entropy H( 21 ) H(p). )
Problem 10.30 (An application to analysis) Let = [0, 1), F = B[0, 1), and P = , where denotes the Lebesgue measure on [0, 1). For n N and k {0, 1, . . . , 2n 1}, we define
Ik,n = [k2n , (k + 1)2n ), Fn = (I0,n , I1,n , . . . , I2n1 ,n ).
In words, Fn is generated by the n-th dyadic partition of [0, 1). For x [0, 1), let kn (x) be the
unique number in {0, 1, . . . , 2n 1} such that x Ikn (x),n . For a function f : [0, 1) R we define
the process {Xnf }nN0 by

Xnf (x) = 2n f (kn (x) + 1 2n ) f kn (x)2n , x [0, 1).
1. Show that {Xnf }nN0 is a martingale.
2. Assume that the function f is Lipschitz, i.e., that there exists K > 0 such that |f (y) f (x)|
K |y x|, for all x, y [0, 1). Show that the limit X f = limn Xnf exists a.s.
3. Show that, for f Lipschitz, X f has the property that
Z y
X f () d, for all 0 x < y < 1.
f (y) f (x) =
x
(Note: This problem gives an alternative proof of the fact that Lipschitz functions are absolutely
continuous.)
141
Chapter
11
Uniform Integrability
11.1
Uniform integrability
Uniform integrabilty is a compactness-type concept for families of random variables, not unlike
that of tightness.
Definition 11.1 (Uniform integrability) A non-empty family X L0 of random variables is said

to be uniformly integrable (UI) if

sup E[|X| 1{|X|K} ] = 0.
lim
K
XX
Remark 11.2 It follows from the dominated convergence theorem (prove it!) that
lim E[|X| 1{|X|K} ] = 0 if and only if X L1 ,
i.e., that for integrable random variables, far tails contribute little to the expectation. Uniformly
integrable families are simply those for which the size of this contribution can be controlled uniformly over all elements.
We start with a characterization and a few basic properties of uniform-integrable families:
Proposition 11.3 (A criterion for uniform integrability) A family X L0 of random variables

is uniformly integrable if and only if
1. there exists C > 0 such that E[|X|] C, for all X X , and
2. for each > 0 there exists > 0 such that for any A F, we have
P[A] sup E[|X| 1A ] .
XX
142
CHAPTER 11. UNIFORM INTEGRABILITY

P ROOF UI (1), (2). Assume X is UI and choose K > 0 such that supXX E[|X| 1{|X|>K} ] 1.
Since
E[|X|] = E[|X| 1{|X|K} ] + E[|X| 1{|X|>K} ] K + E[|X| 1{|X|>K} ],
for any X, we have
sup E[|X|] K + 1,
XX
and (1) follows with C = K + 1.

For (2), we take > 0 and use the uniform integrability of X to find a constant K > 0 such that
supXX E[|X| 1{|X|>K} ] < /2. For = 2K

and A F, the condition P[A] implies that
E[|X| 1A ] = E[|X| 1A 1{|X|K} ] + E[|X| 1A 1{|X|>K} ] KP[A] + E[|X| 1{|X|>K} ] .
(1), (2) UI. Let C > 0 be the bound from (1), pick > 0 and let > 0 be such that (2) holds.
For K = C , Markovs inequality gives
P[|X| K]
1
K E[|X|]
for all X X . Therefore, by (2), E[|X| 1{|X|K} ] for all X X .

Remark 11.4 Boundedness in L1 is not enough for uniform integrability. To construct the counterexample, take (, F, P) = ([0, 1], B([0, 1]), ), and define
(
n, n1 ,
Xn () =
0, otherwise.
Then E[Xn ] = n n1 = 1, but, for K > 0, E[|Xn | 1{|Xn |K} ] = 1, for all n K, so {Xn }nN is not UI.
Problem 11.5 Let X and Y be two uniformly-integrable families (on the same probability space).
Show that the following families are also uniformly integrable:
1. {Z L0 : |Z| |X| for some X X }.
2. {X + Y : X X , Y Y}.
Another useful characterization of uniform integrability uses a class of functions which converge
to infinity faster than any linear function:
Definition 11.6 (Test functions of uniform integrability) A Borel function : [0, ) [0, )
is called a test function of uniform integrability if
lim
(x)
x
143
= .
Proposition 11.7 (A UI criterion using test functions) A nonempty family X L0 is uniformly

integrable if and only if there exists a test function of uniform integrability such that
sup E[(|X|)] < .
(11.1)
XX
Moreover, if it exists, the function can be chosen in the class of non-decreasing convex functions.
P ROOF Suppose, first, that (11.1) holds for some test function of uniform integrability and that
the value of the supremum is 0 < M < . For n > 0, there exists Cn R such that (x) nM x,
for x Cn . Therefore,
M E[(|X|)] E[(|X|)1{|X|Cn } ] nM E[|X| 1{|X|Cn } ], for all X X .
Hence, supXX E[|X| 1{|X|Cn } ] n1 , and the uniform integrability of X follows.
Conversely, suppose that X is uniformly integrable. By definition, there exists a sequence
{Cn }nN (which can always be chosen so that 0 < Cn < Cn+1 for n N, Cn ) such that
sup E[|X| 1{|X|Cn } ]
XX
1
.
n3
Let the function : [0, ) [0, ) be continuous and piecewise affine with (x) = 0 for x
[0, C1 ], and the derivative equal to n on (Cn , Cn+1 ), so that
lim
(x)
x
= lim (x) = .
x
Then,
E[(|X|)] = E[
=
n=1
|X|
0
() d] =
C1
X
E[ ()1{|X|} ] d =
n
n=1
Cn+1
Cn
E[1{|X|} ] d
n (E[|X| Cn+1 ] E[|X| Cn ])
Clearly,
E[|X| Cn+1 ] E[|X| Cn ] = E[(|X| Cn )1{Cn |X|<Cn+1 } ] + (Cn+1 Cn )E[1{|X|Cn+1 } ]
E[|X| 1{|X|Cn } ] + E[Cn+1 1{|X|Cn+1 } ]
E[|X| 1{|X|Cn } ] + E[|X| 1{|X|Cn+1 } ]

and so
E[(|X|)]
nN
2
n2
2
,
n3
< .
Corollary 11.8 (Lp -boundedness, p > 1 implies UI) For p > 1, let X be a nonempty family of
random variables bounded in Lp , i.e., such that supXX ||X||Lp < . Then X is uniformly integrable.
144

Problem 11.9 Let X be a nonempty uniformly-integrable family in L0 . Show that conv X is uniformlyintegrable, where conv X is the smallest convex set in L0 which contains X , i.e., conv X is the set
of all random variables of the form X = 1 X1 + + n Xn , for n N, k 0, k = 1, . . . , n,
P
n
k=1 k = 1 and X1 , . . . , Xn X .
Problem 11.10 Let C be a non-empty family of sub--algebras of F, and let X be a random variable in L1 . The family
X = {E[X|F] : F C},
is uniformly integrable. (Hint: Argue that it follows directly from Proposition 11.7 that E[(|X|)] <
for some test function of uniform integrability. Then, show that the same can be used to prove
that X is UI.)
11.2
First properties of uniformly-integrable martingales
When it is known that the martingale {Xn }nN is uniformly integrable, a lot can be said about its
structure. We start with a definitive version of the dominated convergence theorem:
Proposition 11.11 (A master dominated-convergence theorem) Let {Xn }nN be a sequence of

random variables in Lp , p 1, which converges to X L0 in probability. Then, the following
statements are equivalent:
1. the sequence {|X|pn }nN is uniformly integrable,
Lp
2. Xn X, and
3. ||Xn ||Lp ||X||Lp < .
a.s.
P ROOF (1) (2): Since there exists a subsequence {Xnk }kN such that Xnk X, Fatous lemma
implies that
E[|X|p ] = E[lim inf |Xnk |p ] lim inf E[|Xnk |p ] sup E[|X|p ] < ,
k
XX
where the last inequality follows from the fact that uniformly-integrable families are bounded in
L1 .
Now that we know that X Lp , uniform integrability of {|Xn |p }nN implies that the family
P
{|Xn X|p }nN is UI (use Problem 11.5, (2)). Since Xn X if and only if Xn X 0, we
can assume without loss of generality that X = 0 a.s., and, consequently, we need to show that
E[|Xn |p ] 0. We fix an > 0, and start by the following estimate
(11.2)
E[|Xn |p ] = E[|Xn |p 1{|Xn |p /2} ] + E[|Xn |p 1{|Xn |p >/2} ] /2 + E[|Xn |p 1{|Xn |p >/2} ].
By uniform integrability there exists > 0 such that supnN E[|Xn |p 1A ] < /2, whenever P[A] .
Convergence in probability now implies that there exists n0 N such that for n n0 , we have
P[|Xn |p > /2] . It follows directly from (11.2) that for n n0 , we have E[|Xn |p ] .

(2) (3): ||Xn ||Lp ||X||Lp ||Xn X||Lp 0
145

(3) (1): For M 0, define the function M : [0, ) [0, ) by
x [0, M 1]
x,
M (x) = 0,
x [M, )
interpolated linearly, x (M 1, M ).
For a given > 0, dominated convergence theorem guarantees the existence of a constant M > 0
(which we fix throughout) such that
E[|X|p ] E[M (|X|p )] < .

2
Convergence in probability, together with continuity of M , implies that M (Xn ) M (X) in
probability, for all M , and it follows from boundedness of M and the bounded convergence
theorem that
E[M (|Xn |p )] E[M (|X|p )].
(11.3)
By the assumption (3) and (11.3), there exists n0 N such that

E[|Xn |p ] E[|X|p ] < /4 and E[M (|X|p )] E[M (|Xn |p )] < /4, for n n0 .
Therefore, for n n0 .
E[|Xn |p 1{|Xn |p >M } ] E[|Xn |p ] E[M (|Xn |p )] /2 + E[|X|p ] E[M (|X|p )] .
Finally, to get uniform integrability of the entire sequence, we choose an even larger value of M
to get E[|Xn |p 1{|Xn |p >M } ] for the remaining n < n0 .
Problem 11.12 For Y L1+ , show that the family {X L0 : |X| Y, a.s.} is uniformly integrable. Deduce the dominated convergence theorem from Proposition 11.11
Since convergence in Lp implies convergence in probability, we have:
Corollary 11.13 (Lp -convergence, p 1, implies UI) Let {Xn }nN0 be an Lp -convergent sequence, for some p 1. Then family {Xn : n N0 } is UI.
Since UI (sub)martingales are bounded in L1 , they converge by Theorem 10.20. Proposition
11.11 guarantees that, additionally, convergence holds in L1 :
Corollary 11.14 (Convergence of UI (sub)martingales) Uniformly-integrable (sub)martingales
converge a.s., and in L1 .
For martingales, uniform integrability implies much more:
146
Proposition 11.15 (Structure of UI martingales) If {Xn }nN0 be a martingale. Then, the following are equivalent:
1. {Xn }nN0 is a Lvy martingale, i.e., it admits a representation of the form Xn = E[X|Fn ], a.s.,
for some X L1 (F),
2. {Xn }nN0 is uniformly integrable.
3. {Xn }nN0 converges in L1 ,
In that case, convergence also holds a.s., and the limit is given by E[X|F ], where F = (nN0 Fn ).
P ROOF (1) (2). The representation Xn = E[X|Fn ], a.s., and Problem 11.10 imply that {Xn }nN0
is uniformly integrable.
(2) (3). Corollary 11.14.
(3) (2). Corollary 11.13.
(2) (1). Corollary 11.14 implies that there exists a random variable Y L1 (F) such that
Xn Y a.s., and in L1 . For m N and A Fm , we have |E[Xn 1A Y 1A ]| E[|Xn Y |] 0, so
E[Xn 1A ] E[Y 1A ]. Since E[Xn 1A ] = E[E[X|Fn ]1A ] = E[X1A ], for n m, we have
E[Y 1A ] = E[X1A ], for all A n Fn .
The family n Fn is a -system which generated the sigma algebra F = (n Fn ), and the family
of all A F such that E[Y 1A ] = E[X1A ] is a -system. Therefore, by the Theorem, we have
E[Y 1A ] = E[X1A ], for all A F .
Therefore, since Y F , we conclude that Y = E[X|F ].
Example 11.16 There exists a non-negative (and therefore a.s.-convergent) martingale which is
not uniformly integrable (and therefore, not L1 -convergent).
Let {Xn }nN0 be a simple random
P
walk starting from 1, i.e. X0 = 1 and Xn = 1 + nk=1 k , where {n }nN is an iid sequence with
P[n = 1] = P[n = 1] = 12 , n N. Clearly, {Xn }nN0 is a martingale, and so is {Yn }nN0 ,
where Yn = XnT and T = inf{n N : Xn = 0}. By convention, inf = +. It is well known
that a simple symmetric random walk hits any level eventually, with probability 1 (we will prove
this rigorously later), so P[T < ] = 1, and, since Yn = 0, for n T , we have Yn 0, a.s., as
n . On the other hand, {Yn }nN0 is a martingale, so E[Yn ] = E[Y0 ] = 1, for n N. Therefore,
E[Yn ] 6 E[X], which can happen only if {Yn }nN0 is not uniformly integrable.
11.3
Backward martingales
If, instead of N0 , we use N0 = {. . . , 2, 1, 0} as the time set, the notion of a filtration is readily
extended: it is still a family of sub--algebras of F, parametrized by N0 , such that Fn1 Fn ,
for n N0 .
147
Definition 11.17 (Backward submartingales) A stochastic process {Xn }nN0 , is said to be a

backward submartingale with respect to the filtration {Fn }nN0 , if
1. {Xn }nN0 is {Fn }nN0 -adapted,
2. Xn L1 , for all n N0 , and
3. E[Xn |Fn1 ] Xn1 , for all n N0 .
If, in addition to (1) and (2), the inequality in (3) is, in fact, an equality, we say that {Xn }nN0 is a
backward martingale.
One of the most important facts about backward submartingales is that they (almost) always
converge a.s., and in L1 .
Proposition 11.18 (Backward submartingale convergence) Suppose that {Xn }nN0 is a backward submartingale such that
lim E[Xn ] > .
n
Then {Xn }nN0 is uniformly integrable and there exists a random variable X L1 (n Fn ) such
that
(11.4)
Xn X a.s. and in L1 ,
and
(11.5)
X E[Xm | n Fn ], a.s., for all m N0 .
P ROOF We start by decomposing {Xn }nN0 in the manner

Pn of Doob and Meyer. For n N0 ,
set An = E[Xn Xn1 |Fn1 ] 0, a.s., and An =
k=0 Ak , for n N0 . The backward
submartingale property of {Xn }nN0 implies that E[Xn ] L = limn E[Xn ] > , so
E[An ] = E[X0 Xn ] E[X0 ] L, for all n N0 .
The monotone convergence theorem implies that E[A ] < , where A =
process {Mn }nN0 defined by Mn = Xn An is a backward martingale. Indeed,
n=0 An .
The
E[Mn Mn1 |Fn1 ] = E[Xn Xn1 An |Fn1 ] = 0.

Since all backward martingales are uniformly integrable (why?) and the sequence {An }nN0
is uniformly dominated by A L1 - and therefore uniformly integrable - we conclude that
{Xn }nN0 is also uniformly integrable.
To prove convergence, we start by observing that the uniform integrability of {Xn }nN0 implies that supnN0 E[Xn+ ] < . A slight modification of the proof of the martingale convergence
148

theorem (left to a very diligent reader) implies that Xn X , a.s. for some random variable X n Fn . Uniform integrability also ensures that the convergence holds in L1 and that
X L1 .
In order to show (11.5), it is enough to show that
(11.6)
E[X 1A ] E[Xm 1A ],
for any A n Fn , and any m N0 . We first note that since Xn E[Xm |Fn ], for n m 0, we
have
E[Xn 1A ] E[E[Xm |Fn ]1A ] = E[Xm 1A ],
for any A n Fn . It remains to use the fact the L1 -convergence of {Xn }nN0 implies that
E[Xn 1A ] E[X 1A ], for all A F.
Remark 11.19 Even if lim E[Xn ] = , the convergence Xn X still holds, but not in L1 and
X may take the value with positive probability.
Corollary 11.20 (Backward martingale convergence) If {Xn }nN0 is a backward martingale,
then Xn X = E[X0 | n Fn ], a.s., and in L1 .
11.4
Applications of backward martingales
We can use the results about the convergence of backward martingales to give a non-classical
proof of the strong law of large numbers. Before that, we need a useful classical result.
Proposition 11.21 (Kolmogorovs 0-1 law) Let {n }nN be a sequence of independent random variables, and let the tail -algebra F be defined by
\
Fn , where Fn = (n , n+1 , . . . ).
F =
nN
Then F is P-trivial, i.e., P[A] {0, 1}, for all A F .

P ROOF Define Fn = (1 , . . . , n ), and note that Fn1 and Fn are independent -algebras. Therefore, F Fn is also independent of Fn , for each n N. This, in turn, implies that F is
independent of the -algebra F = (n Fn ). On the other hand, F F , so F is independent of itself. This implies that P[A] = P[A A] = P[A]P[A], for each A F , i.e., that
P[A] {0, 1}.
Theorem 11.22 (Strong law of large numbers) Let {n }nN be an iid sequence of random variables in L1 . Then
1
1
n (1 + + n ) E[1 ], a.s. and in L .
149

P ROOF For notational reasons, backward martingales are indexed by N instead of N0 . For
n N, let Sn = 1 + + n , and let Fn be the -algebra generated by Sn , Sn+1 , . . . . The process
{Xn }nN is given by
Xn = E[1 |Fn ], for n N0 .
Since (Sn , Sn+1 , . . . ) = ((Sn ), (n+1 , n+2 , . . . )), and (n+1 , n+2 , . . . ) is independent of 1 ,
for n N, we have
Xn = E[1 |Fn ] = E[1 |(Sn )] = n1 Sn ,
where the last equality follows from Problem 9.29. Backward martingales converge a.s., and in
L1 , so for the random variable X = limn n1 Sn we have
E[X ] = lim E[ n1 Sn ] = E[1 ].
n
On the other hand, since limn n1 Sk = 0, for all k N, we have X = limn n1 (k+1 + + n ),
for any k N, and so X (k+1 k, k+2 , . . . ). By Proposition 11.21, X is measurable in
a P-trivial -algebra, and is, thus, constant a.s. (why?). Since E[X ] = E[1 ], we must have
X = E[1 ], a.s.
11.5
Exchangeability and de Finettis theorem (*)
We continue with a useful generalization of the Kolmogorovs 0-1 law, where we extend the ideas
about the use of symmetry in the proof of Theorem 11.22 above.
Definition 11.23 (Symmetric functions) A function f : Rn R is said to be symmetric if

f (x1 , . . . , xn ) = f (x(1) , . . . , x(n) ),
for each permutation Sn .
For k, n N, k n, let Snk denote the set of all injections : {1, 2, . . . , k} {1, 2, . . . , n}. Note
n!
that Snk has n(n 1) . . . (n k + 1) = (nk)!
= k! nk elements and that for k = n, Sn = Snn is the set
of all permutations of the set {1, 2, . . . , n}.
Definition 11.24 (Symmetrization) For a function f : Rk R, and n k the function fnsim :
Rn R, given by
X
1
fnsim (x1 , . . . , xn ) = n(n1)...(nk+1)
f (x(1) , . . . , x(k) ),
k
Sn
is called the n-symmetrization of f .
Problem 11.25
1. Show that a function f : Rn R is symmetric if and only if f = fnsim .
150

2. Show that for each 1 k n, and each function f : Rk R, the n-symmetrization fnsim of
f is a symmetric function.
Definition 11.26 (Exchangeable -algebra) For n N, let En be the -algebra generated by Xn+1 ,
Xn+2 , . . . , in addition to all random variables of the form f (X1 , . . . , Xn ), where f is a symmetric Borel
function f : Rn R.
The exchangeable -algebra E is defined by
\
E=
En .
nN
Remark 11.27 The exchangeable -algebra clearly contains the tail -algebra, and we can interpret it as the collection of all events whose occurrence is not affected by a permutation of the order
of X1 , X2 , . . . .
Example 11.28 Consider the event
A = { : lim sup
kN
k
X
j=1
Xj () 0}.
This event is not generally in the tail -algebra (why?), but it is always in the exchangeable algebra. Indeed, A can be written as
{ : lim sup
k
k
X
j=n+1
Xj () (X1 () + + Xn ())},
and as such belongs to En , since it can be represented as a combination of a random variable

P
lim supk kj=n+1 Xj measurable in (Xn+1 , . . . ) and a symmetric function f (x1 , . . . , xn ) = x1 +
+ xn of (X1 , . . . , Xn ).
For iid sequences, there is no real difference between the exchangeable and the tail -algebra: they
are both trivial. Before we prove this fact, we need a lemma and a problem.
Lemma 11.29 (Symmetrization as conditional expectation) Let {Xn }nN be an iid sequence and
let f : Rk R, k N be a bounded Borel function. Then
(11.7)
fnsim (X1 , . . . , Xn ) = E[f (X1 , . . . , Xk )|En ], a.s., for n k.
Moreover,
(11.8)
fnsim (X1 , . . . , Xn ) E[f (X1 , . . . , Xk )|E], a.s., and in L1 , as n .
151

P ROOF By Problem 11.25, for n k, fnsim (X1 , . . . , Xn ) En , and so
fnsim (X1 , . . . , Xn ) = E[fnsim (X1 , . . . , Xn )|En ]
X
1
= n(n1)...(nk+1)
E[f (X(1) , . . . , X(k) )|En ].
k
Sn
By symmetry and definition of En , we expect E[f (X(1) , . . . , X(k) )|En ] not to depend on . To
prove this in a rigorous way, we must show that,
E[f (X(1) , . . . , X(k) ) f (X1 , . . . , Xk )|En ] = 0, a.s.,
for each Snk . For that, in turn, it will be enough to pick Snk and show that
E[g(X1 , . . . , Xn )(f (X(1) , . . . , X(k) ) f (X1 , . . . , Xk ))] = 0,
for any bounded symmetric function g : Rn R. Notice that the iid property implies that for any
permutation Sn , and any bounded Borel function h : Rn R, we have
E[h(X1 , . . . , Xn )] = E[h(X(1) , . . . , X(n) )].
In particular, for the function h : Rn R given by
h(x1 , . . . , xn ) = g(x1 , . . . , xn )f (x(1) , . . . , x(k) ),
and the permutation Sn with ((i)) = i, for i = 1, . . . , k (it exists since is an injection), we
have
E[g(X1 , . . . , Xn )f (X(1) , . . . , f(k) )] = E[h(X1 , . . . , Xn )]
= E[h(X(1) , . . . , X(n) ]
= E[g(X(1) , . . . , X(n) )f (X1 , . . . , Xn )]
= E[g(X1 , . . . , Xn )f (X1 , . . . , Xk )],
where the last equality follows from the fact that g is symmetric.
Finally, to prove (11.8), we simply combine (11.7), the definition E = n En of the exchangeable
-algebra and the backward martingale convergence theorem (Proposition 11.18).
Problem 11.30 Let G be a sub--algebra of F, and let X be a random variable with E[X 2 ] <
such that E[X|G] is independent of X. Show that X = E[X], a.s.
(Hint: Use the fact that E[X|G] is the L2 -projection of X onto L2 (G), and, consequently, that
E[(X E[X|G])E[X|G]] = 0. )
Proposition 11.31 (Hewitt-Savage 0-1 Law) The exchangeable -algebra of an iid sequence is trivial, i.e., P[A] {0, 1}, for A E.
P ROOF We pick a Borel function f : Rk R such that |f (x)| C, for x Rk . The idea of
the proof is to improve the conclusion (11.8) of Lemma 11.29 to fnsim E[f (X1 , . . . , Xk )]. We
(n1)!
n!
n!
n!
start by observing that for n > k, out of (nk)!
terms in fnsim , exactly (nk)!
(n1k)!
= nk (nk)!
152

involve X1 , so that the sum of all those terms does not exceed nk C. It follows that all occurrences
of X1 in fnsim (X1 , . . . , Xn ) disappear in the limit, i.e., more precisely, that the limit of the sequence
consisting of only those terms in fnsim (X1 , . . . , Xn ) which do not contain X1 coincides with the
limit of (all terms in) fnsim (X1 , . . . , Xn ). Therefore, the limit E[f (X1 , . . . , Xk )|E] can be obtained as a
limit of a sequence of combinations of the random variables X2 , X3 , . . . and is, hence, measurable
with respect to (X2 , X3 , . . . ).
We can repeat the above argument with X1 replaced by X2 , X3 ,. . . , Xk , to conclude that
E[f (X1 , . . . , Xk )|E] is measurable with respect to the -algebra (Xk+1 , Xk+1 , . . . ), which is independent of f (X1 , . . . , Xk ). Therefore, by Problem 11.30, we have
E[f (X1 , . . . , Xk )|E] = E[f (X1 , . . . , Xk )], a.s.
In particular, for A E we have E[1A f (X1 , . . . , Xk )] = P[A]E[f (X1 , . . . , Xk )], for any bounded
Borel function f : Rk R. It follows that the -algebras (X1 , . . . , Xk ) and E are independent,
for all k N. The - theorem implies that E is then also independent of F = (X1 , X2 , . . . ) =
(k (X1 , . . . , Xk )). On the other hand E F , and so E is independent of itself. It follows that
P[A] = P[A A] = P[A]2 , for all A E, and, so, P[A] {0, 1}.
The following generalization of the strong law of large numbers follows directly from (11.8) and
Proposition 11.31.
Corollary 11.32 (A strong law for symmetric functions) Let {Xn }nN be an iid sequence, and let
f : Rk R, k N, be a Borel function with f (X1 , . . . , Xk ) L1 . Then
fnsim (X1 , . . . , Xn ) E[f (X1 , . . . , Xk )].
Remark 11.33 The random variables of the form fnsim (X1 , . . . , Xn ), for some Borel function f :
Rk R are sometimes called U-statistics, and are used as estimators in statistics. Corollary 11.32
can be interpreted as consistency statement for U-statistics.
Example 11.34 For f (x1 , x2 ) = (x1 x2 )2 , we have
fnsim (x1 , . . . , xn ) =
n
2
( )
1i<jn
(xi xj )2 ,
and so, by Corollary 11.32, we have that for an iid sequence {Xn }nN with 2 = Var[X1 ] < , we
have
X
1
(Xi Xj )2 E[(X1 X2 )2 ] = 2 2 , a.s. and in L1 .
n
(2)
1i<jn
As a last application of the backward martingale convergence theorem, we prove de Finettis

theorem.
153
Definition 11.35 (Exchangeable sequences) A sequence {Xn }nN of random variables is said to
be exchangeable If
(d)
(X1 , . . . , Xn ) = (X(1) , . . . , X(n) ),

for all n N and all permutations Sn
Example 11.36 Iid sequences are clearly exchangeable, but the inclusion is strict. Here is a typical
example of an exchangeable sequence which is not iid. Let , {Yn }nN be independent uniformly
distributed random variable in (0, 1). We define
Xn = 1{Yn } .
Think of a situation where the parameter (0, 1) is chosen randomly, and then an unfair coin
with the probability of obtaining heads is minted and tossed repeatedly. The distribution of the
coin-toss is the same for every toss, but the values are not independent. Intuitively, if we are told
the value of X1 , we have a better idea about the value of , which, in turn, affects our distribution
of Y2 . To show that that {Xn }nN is an exchangeable sequence which is not iid we compute
E[X1 X2 ] = E[P[Y1 , Y2 ]|()] =
1
0
P[Y1 , Y2 x] dx =
as, well as
E[X1 ]E[X2 ] = P[Y1 ]P[Y2 ] = (
1
0
1
0
x2 dx = 13 ,
x dx)2 = 41 .
To show that {Xn }nN is exchangeable, we need to compare the distribution of (X1 , . . . , Xn ) and
(X(1) , . . . , X(n) ) for Sn , n N. For a choice (b1 , . . . , bn ) {0, 1}n , we have
P[(X(1) , . . . , X(n) ) = (b1 , . . . , bn )]
= E[E[1{(X(1) ,...,X(n) )=(b1 ,...,bn )} |()]]
Z 1
=
P[1{Y(1) x} = b1 , . . . , 1{Y(n) x} = bn ] dx
0
=
=
n
1Y
0 i=1
Z 1Y
n
0 i=1
P[1{Y(i) x} = bi ] dx
P[1{Y1 x} = bi ] dx
where the last equality follows from the fact that {Yn }nN are iid. Since the final expression above
does not depend on , the sequence {Xn }nN is indeed exchangeable.
Problem 11.37 (A consequence of exchangeability) Let {Xn }nN be an exchangeable sequence
with E[X12 ] < . Show that
E[X1 X2 ] 0.
(Hint: Expand the inequality E[(X1 E[X1 ] + + Xn E[Xn ])2 ] 0 and use exchangeability.)
154
Definition 11.38 (Conditional iid) A sequence {Xn }nN is said to be conditionally iid with respect to a sub--algebra G of F, if
P[X1 A1 , X2 A2 , . . . , Xn An |G] = P[X1 A1 |G] P[X1 An |G], a.s.,
for all n N, and all A1 , . . . , An B(R).
Problem 11.39 Let the sequence {Xn }nN be as in Example 11.36. Show that
1. {Xn }nN is conditionally iid with respect to (), and
2. () is the exchangeable -algebra for {Xn }nN . (Hint: Consider the limit limn
1
n
Pn
k=1 Xk .)
The result of Problem 11.39 is not a coincidence. In a sense, all exchangeable sequences have the
structure similar to that of {Xn }nN above:
Theorem 11.40 (de Finetti) Let {Xn }nN be an exchangeable sequence, and let E be the corresponding exchangeable -algebra. Then, conditionally on E, {Xn }nN are iid.
P ROOF Let f be of the form f = gh, where g : Rk1 R and h : R R are Borel functions with
|g(x)| Cg , for all x Rk1 and |h(x)| Ch , for all x R. The product Pn = n(n 1) . . . (n k +
2)gnsim (X1 , . . . , Xn ) nhsim
n (X1 , . . . , Xn ) can be expanded into
X
X
g(X(1) , . . . , X(k1) )
h(Xi )
Pn =
k1
Sn
i{1,...,n}
g(X(1) , . . . , X(k1) )h(X(k) ) +
k
Sn
g(X(1) , . . . , X(k1) )
k1
X
h(X(j) ).
j=1
k1
Sn
A bit of simple algebra, where f j (x1 , . . . , xk1 ) = g(x1 , . . . , xk1 )h(xj ), for j = 1, . . . , k 1 yields
fnsim =
The sum
Pk1
j=1
sim sim
n
nk+1 gn hn
1
nk+1
k1
X
fnj,sim .
j=1
fnj,sim is bounded by (k 1)Cg Ch , and so, upon letting n , we get

= 0.
lim fnsim gnsim hsim
n
n
Therefore, the relation (11.8) of Lemma 11.29 applied to f sim , g sim and hsim implies that
E[f (X1 , . . . , Xk )|E] = E[g(X1 , . . . , Xk1 )|E] E[h(Xk )|E].
155

We can repeat this procedure with g = g h , for some bounded Borel functions g : Rk1
R and h : R R to split the conditional expectation E[g(X1 , . . . , Xk1 )|E] into the product
E[g (X1 , . . . , Xk2 )|E]E[h (Xk1 )|E]. After k 1 such steps, we get
E[h1 (X1 )h2 (X2 ) . . . , hk (Xk )|E] = E[h1 (X1 )|E] E[hk (Xk )|E],
for any selection of bounded Borel functions hi : R R, i = 1, . . . , k. We pick hi = 1Ai , for
Ai B(R), to conclude that
P[X1 A1 , . . . , Xk Ak |E] = P[X1 A1 |E] P[Xk Ak |E],
for all Ai B(R), i = 1, . . . , k. The full conditional iid property follows from exchangeability.
11.6
Additional Problems
Problem 11.41 (A UI martingale not in H 1 ) Set = N, F = 2N , and P is the probability measure

on F characterized by P[{k}] = 2k , for each k N. Define the filtration {Fn }nN by

Fn = {1}, {2}, . . . , {n 1}, {n, n + 1, . . . } , for n N.
Let Y : [1, ) be a random variable such that E[Y ] < and E[Y K] = , where K(k) = k,
for k N.
1. (3pts) Find an explicit example of a random variable Y with the above properties.
2. (5pts) Find an expression for Xn = E[Y |Fn ] in terms of the values Y (k), k N.
(k) := sup
3. (12pts) Using the fact that X
nN |Xn (k)| Xk (k) for k N, show that {Xn }nN
is a uniformly integrable martingale which is not in H 1 . (Note: A martingale {Xn }nN is said
L1 .)
to be in H 1 if X
Problem 11.42 (Scheffs lemma) Let {Xn }nN0 be a sequence of random variables in L1+ such
that Xn X, a.s., for some X L1+ . Show that E[Xn ] E[X] if and only if the sequence
{Xn }nN0 is UI.
Problem 11.43 (Hunts lemma) Let {Fn }nN0 be a filtration, and let {Xn }nN0 be a sequence in L0
such that Xn X, for some X L0 , both in L1 and a.s.
1. (Hunts lemma). Assume that |Xn | Y , a.s., for all n N and some Y L1+ Prove that
(11.9)
E[Xn |Fn ] E[X|(n Fn )], a.s.
(Hint: Define Zn = supmn |Xm X|, and show that Zn 0, a.s., and in L1 .)
2. Find an example of a sequence {Xn }nN in L1 such that Xn 0, a.s., and in L1 , but E[Xn |G]
1An
does not converge to 0, a.s., for some G F. (Hint: Look for Xn of the form Xn = n P[A
n]
and G = (n ; n N).)
(Note: The existence of such a sequence proves that (11.9) is not true without an additional
assumption, such as the one of uniform domination in (1). It provides an example of a
property which does not generalize from the unconditional to the conditional case.)
156

Problem 11.44 (Krickebergs decomposition) Let {Xn }nN0 be a martingale. Show that the following two statements are equivalent:
1. There exists martingales {Xn+ }nN0 and {Xn }nN0 such that Xn+ 0, Xn 0, a.s., for all
n N0 and Xn = Xn+ Xn , n N0 .
2. supnN0 E[|Xn |] < .
+
(Hint: Consider limn E[Xm+n
|Fm ], for m N0 . )
Problem 11.45 (Branching processes) Let be a probability measure on B(R) with (N0 ) = 1,
which we call the offspring distribution. A population starting from one individual (Z0 = 1)
evolves as follows. The initial member leaves a random number Z1 of children and dies. After
that, each of the Z1 children of the initial member, produces a random number of children and
dies. The total number of all children of the Z1 members of the generation 1 is denoted by Z2 . Each
of the Z2 members of the generation 2 produces a random number of children, etc. Whenever
an individual procreates, the number of children has the distribution , and is independent of
the sizes of all the previous generations including the present one, as well as of the numbers of
children of other members of the present generation.
1. Suppose that a probability space and iid sequence {n }nN of random variables with the
distribution is given. Show how you would construct a sequence {Zn }nN0 with the above
properties. (Hint: Zn+1 is a sum of iid random variables with the number of summands
equal to Zn .)
2. For a distribution on N0 , we define the the generating function P : [0, 1] [0, 1] of by
X
({k})xk .
P (x) =
kN0
Show that each P is continuous, non-decreasing and convex on [0, 1] and continuously differentiable on (0, 1).
3. Let P = P be the generating function of the offspring distribution , and for

P n N0 , we
define Pn (x) as the generating function of the distribution of Zn , i.e., Pn (x) = kN0 P[Zn =
k]xk . Show that Pn (x) = P (P (. . . P (x) . . . )) (there are n P s). (Hint: Note that P (x) = E[xZn ]
for x > 0 and use the result of Problem 9.26)
4. Define the extinction probability pe by pe = P[Zn = 0, for some n N]. Prove that pe is a
fixed point of the map P , i.e., that P (pe ) = pe . (Hint: Show that pe = limn P (n) (0), where
P (n) is the n-fold composition of P with itself.)
5. Let = E[Z1 ], be the expected number of offspring. Show that when 1 and ({1}) <
1, we have pe = 1, i.e., the population dies out with certainty if the expected number of
offspring does not exceed 1. (Hint: Draw a picture of the functions x and P (x) and use (and
prove) the fact that, as a consequence of the assumption 1, we have P (x) < 1 for all
x < 1.)
6. Assuming that 0 < < , show that the process {Xn }nN0 , given by Xn = Zn /n , is a
martingale (with respect to the filtration {Fn }nN0 , where Fn = (Z0 , Z1 , . . . , Zn )).
P
7. Identify all probability measures with (N0 ) = 1, and kN0 k({k}) = 1 such that the
branching process {Zn }nN0 with the offspring distribution is uniformly integrable.
157
Index
probability, 115, 122
conjugate exponents, 53
continuity of probability, 72
convergence
almost surely (a.s.), 73
almost-everywhere, 45
everywhere, 46
in L1 , 51
in Lp , 52
in distribution, 89
in measure, 58
in probability, 73
in total variation, 103
weak, 89
convex cone, 37
convolution
of L1 -functions, 84
of probability measures, 83
countable set, 5
countable-cocountable -algebra, 7
cycle, 112
-system, 6
-system, 6
-algebra, 6
-trivial, 45
countable-cocountable, 7
exchangeable, 151
generated by a family of maps, 10
generated by a map, 10
tail, 149
cylinder set, 12
algebra, 6
almost surely (a.s.), 73
almost-everywhere equality, 44
asymptotic density, 34
atom of a measure, 20
axiom of choice, 21
Bell numbers, 17
Bochners theorem, 97
Borel -algebra, 7
Borel function, 9
Darboux sums, 48
Dirac function, 20
distribution
2 , 86
coin-toss, 78
cummulative (cdf), 75
exponential, 86
joint, 75
marginal, 75
of a random element, 75
of a random variable, 74
of a random vector, 75
regular conditional, 123
Cantor set, 34
characteristic function, 96
choice functions, 10
coin-toss space, 12
completion of a measure, 34
composition of functions, 5
conditional
probability, regular, 123
characteristic function, 126
density, 125
expectation, 115
iid property, 155
158

of -algebras, 78
of events, 78
of random variables, 78
pairwise, 78
indicator function, 14
inequality
arithmetic-geometric, 57
Cauchy-Schwarz, 54
Hlders, 53
Jensens, 56
Markov, 57
Minkowski, 54
upcrossings, 138
Youngs, 53
integral
Lebesgue-Stieltjes, 82
Legesgue, 39
Riemann, 48
isometry, 65
singular, 77
standard normal, 86
uniform on (a, b), 84
uniform on S 1 , 34
elementary outcome, 72
entropy of a distribution, 141
equally distributed random variables, 74
essential supremum, 52
essentially bounded from above, 52
events, 72
certain, 72
mutually-exclusive, 72
eventually-periodic sequence, 35
exchangeable sequence, 154
expectation, 73
exponential moment, 86
extended set of real numbers, 14
extinction probability, 157
family of sets, 5
decreasing, 5
increasing, 5
pairwise disjoint, 5
filtration, 130
generated by a process, 131
function
n-symmetrization of, 150
a version of, 68
Caratheodory function, 71
characteristic, 96
cummulative distribution (cdf), 75
essentially-bounded, 52
generating function, 157
integrable, 38
measure-preserving, 27
null, 44
probability-density (pdf), 76
Riemann-integrable, 48
simple, 36
symmetric, 150
last visit time, 135

Lebesgue measure, 29
lemma
Borel-Cantelli, 23
Borel-Cantelli II, 87
Fatous lemma, 43
limit inferior, 15, 23
limit superior, 15, 23
martingale, 131
backward, 148
martingale transform, 134
measurable kernel, 123
measurable set, 8
measurable space, 8
measure, 19
-additivity, 19
-finite, 19
absolutely continuous, 66
atom-free, 20
Cantor measure, 70
countably additive, 19
counting, 20
diffuse, 20
equivalent, 66
finite, 19
finitely-additive, 19
Lebesgue, 29
generating function, 157

hitting time, 135
iid (independent and identically distributed),
86
independence, 77
159

discrete, 76
iid, 86
uncorrelated, 82
random vector, 73
absolutely-continuous, 76
relative weak compactness, 93
positive, 19
probability measure, 19
product, 63
real, 31
regular, 35
signed, 31
singular, 66
support of, 70
translation-invariant, 29
uniform, 20
vector measure, 122
measure space, 19
complete, 34
product, 63
measure-preserving map, 27
sample space, 72
section of a set, 61
set
-continuity, 90
convex, 58
countable, 5
exceptional, 45
null, 20, 44
set function, 19
simple-function representation, 36
space
Banach, 55
Borel, 123
complete metric, 55
nice, 123
normed, 50
pseudo-Banach, 55
pseudo-normed, 50
topological, 7
Standard Machine, 36
stochastic process
adapted, 130
discrete-time, 73, 130
predictable, 133
stopped at T , 136
stopping time, 135
submartingale, 131
backward, 148
summability, 31
supermartingale, 131
natural projections, 11
norm, 50
offspring distribution, 157
parallelogram identity, 58
Parsevals identity, 98
partition, 17
finite measurable, 31
of a set, 5
of an interval, 48
point mass, 20
probability space, 72
filtered, 130
product
of measurable spaces, 11
of measure spaces, 63
of sets, 10
product cylinder set, 12
pseudo metric, 33
pseudo norm, 50
pseudo-random numbers, 85
pull-back, 8
push-forward, 8, 28
theorem
-, 25
Caratheodorys extension theorem, 24
continuity, 102
de Finettis, 155
dominated-convergence theorem, 43
Fubini-Tonelli, 64
Hahn-Jordan, 32
inversion, 98
monotone-convergence theorem, 39
Radon-Nikodym derivative, 68
random element, 73
random permutation, 112
random time, 135
random variable, 72
absolutely-continuous, 76
extended-valued, 73
random variables
160

Prohorovs, 94
Radon-Nikodym, 68
Scheff, 103
Slutskys, 105
weak law of large numbers, 107
theorem:Lindeberg-Feller, 111
tightness, 93
topology, 7
total variation
of a measure, 31
total variation norm
of a measure, 32
trajectory of a stochastic process, 130
trivial -algebra, 7
U-statistics, 153
ultra-metric, 162
ultrafilter, 21
uncountable, 5
uniform integrability, 142
test-functions, 143
vector measure, 122
161

Theory of Probability Zitcovic PDF

Uploaded by

Theory of Probability Zitcovic PDF

Uploaded by

Theory of Probability

Measure theory, classical probability and stochastic analysis

Department of Mathematics, The University of Texas at Austin

Lebesgue Spaces and Inequalities

Theorems of Fubini-Tonelli and Radon-Nikodym

Basic Notions of Probability

Weak Convergence and Characteristic Functions

Classical Limit Theorems

CHAPTER 1. MEASURABLE SPACES

Problem 1.3 Show that:

CHAPTER 1. MEASURABLE SPACES

Definition 1.7 (Topology) A topology on a set S is a family of subsets of S which contains

CHAPTER 1. MEASURABLE SPACES

CHAPTER 1. MEASURABLE SPACES

CHAPTER 1. MEASURABLE SPACES

Definition 1.23 (Generation by several functions) Let (f ) be a family of maps

2. for a familty f : S T , , we have

where f1 (T ) = {f1 (B) : B T }.

Products of measurable spaces

Definition 1.25 (Products, choice functions) Let

CHAPTER 1. MEASURABLE SPACES

Definition 1.27 (Natural projections) For 0 , the function 0 :

is called the (natural) projection onto the coordinate 0 .

Definition 1.28 (Products of measurable spaces) Let {(S

CHAPTER 1. MEASURABLE SPACES

CHAPTER 1. MEASURABLE SPACES

, where i(s1 , s2 ) = inf{i N : s1i 6= s2i },

for sj = (sj1 , sj2 , . . . ), j = 1, 2.

Real-valued measurable functions

CHAPTER 1. MEASURABLE SPACES

Proposition 1.37 (Products and maxima preserve measurability) Let f, g be in L0 . Then

CHAPTER 1. MEASURABLE SPACES

and an infimum in R in an obvious way.

metric d : R R [0, 1) given by

subsequence of {xn }nN }.

we immediately have the -algebra B(R)

if and only if the sets f 1 ({}),

CHAPTER 1. MEASURABLE SPACES

1. supn fn , inf n fn L0 (R),

2. lim supn fn , lim inf n fn L0 (R),

CHAPTER 1. MEASURABLE SPACES

Problem 1.44 Which of the following are -algebras on R?

CHAPTER 1. MEASURABLE SPACES

A triple (S, S, ) consisting of a non-empty set, a -algebra S on it and a measure on S is called a

Definition 2.3 (Terminology) A measure on the measurable space (S, S) is called

4. measure or atom-free, if ({x}) = 0, whenever x S and {x} S.

2. Dirac measure. For x S, we define the set function x on S by

Proposition 2.6 (First properties of measures) Let (S, S, ) be a measure space.

(Continuity with respect to increasing sequences)

(Continuity with respect to decreasing sequences)

(Subadditivity w.r.t. general sequences)

(Bk ) = lim (nk=1 Bk ) = lim (An ).

4. Consider the increasing sequence {Bn }nN in S given by Bn = A1 \ An . By De Morgan laws,

= (A1 ) lim (An ).

Subtracting both sides from (A1 ) < produces the statement.

On the other hand,

(A1 A2 ) = (A1 \ A2 ) + (A2 \ A1 ) + (A1 A2 ).

Induction can be used to show that

Since all (An ) are nonnegative, we now have

Problem 2.8 Let (S, S, ) be a finite measure space. Show that

Proposition 2.9 (Borel-Cantelli Lemma P

Extensions of measures and the coin-toss space

C (Cn1 ,...,nk ;b1 ,...,bk ) =

which is clearly not a cylinder.

Then, there exists a measure

Remark 2.11 In words, a -additive measure on an algebra A can be extended to a -additive

lim (An ) = 0, whenever An .

Proposition 2.15 (Existence of the coin-toss measure) There

P ROOF Thanks to Lemma 2.14, Theorem 2.10 can now be used.

forms a partition of {1, 1}N .

n (A) = C (Tn (A)).

The Lebesgue measure

Let f : S [0, 1] be the mapping given by

Hence, the map f is Lipschitz and, therefore, continuous.

([a, b)) = b a, ({a}) = 0 for 0 a < b 1.

The Lebesgue measure has another important property: