Mathematical Foundations of Machine Learning: (NMAG 469, FALL TERM 2018-2019)

MATHEMATICAL FOUNDATIONS OF MACHINE
LEARNING
(NMAG 469, FALL TERM 2018-2019)
∗
HÔNG VÂN LÊ
Contents
1. Learning, machine learning and artificial intelligence 3
1.1. Learning, inductive learning and machine learning 3
1.2. A brief history of machine learning 5
1.3. Current tasks and types of machine learning 6
1.4. Basic questions in mathematical foundations of machine
learning 9
1.5. Conclusion 10
2. Statistical models and frameworks for supervised learning 11
2.1. Discriminative model of supervised learning 11
2.2. Generative model of supervised learning 15
2.3. Empirical Risk Minimization and overfitting 17
2.4. Conclusion 19
3. Statistical models and frameworks for unsupervised learning and
reinforcement learning 19
3.1. Statistical models and frameworks for density estimation 19
3.2. Statistical models and frameworks for clustering 22
3.3. Statistical models and frameworks for dimension reduction and
manifold learning 23
3.4. Statistical model and framework for reinforcement learning 25
3.5. Conclusion 25
4. Fisher metric and maximum likelihood estimator 25
4.1. The space of all probability measures and total variation norm 25
4.2. Fisher metric on a statistical model 28
4.3. MSE and Cramér-Rao inequality 30
4.4. Efficient estimators and MLE 32
4.5. Consistency of MLE 32
4.6. Conclusion 33
5. Consistency of a learning algorithm 33
5.1. Consistent learning algorithm and its sample complexity 33
5.2. Uniformly consistent learning and VC-dimension 37
Date: December 7, 2018.

∗
Institute of Mathematics of ASCR, Zitna 25, 11567 Praha 1, email: hvle@math.cas.cz.
1
2 HÔNG VÂN LÊ ∗
5.3. Fundamental theorem of binary classification 39

5.4. Conclusions 40
6. Generalization ability of a learning machine and model selection 41
6.1. Covering number and sample complexity 41
6.2. Rademacher complexities and sample complexity 44
6.3. Model selection 46
6.4. Conclusion 47
7. Support vector machines 48
7.1. Linear classifier and hard SVM 48
7.2. Soft SVM 51
7.3. Sample complexities of SVM 53
7.4. Conclusion 54
8. Kernel based SVMs 55
8.1. Kernel trick 55
8.2. PSD kernels and reproducing kernel Hilbert spaces 57
8.3. Kernel based SVMs and their generalization ability 60
8.4. Conclusion 61
9. Neural networks 61
9.1. Neural networks as computing devices 62
9.2. The expressive power of neural networks 64
9.3. Sample complexities of neural networks 65
9.4. Conclusion 67
10. Appendix 1: Some basic notions in probability theory 67
10.1. Dominating measures and the Radon-Nikodym theorem 67
10.2. Conditional expectation, conditional probability (measure)
and joint distribution 68
11. Appendix 2: Concentration-of-measure inequalities 72
11.1. Markov’s inequality 72
11.2. Hoeffding’s inequality 72
11.3. Bernstein’s inequality 72
11.4. McDiarmid’s inequality 73
References 73
It is not knowledge, but the act of learning ... which grants the greatest
enjoyment.
Carl Friedrich Gauss
Machine learning is an interdisciplinary field in the intersection of math-

ematical statistics and computer sciences. Machine learning studies sta-
tistical models and algorithms for deriving predictors or meaningful pat-
terns from empirical data. Machine learning techniques are applied in
search engine, speech recognition and natural language processing, image
detection, robotics etc.. In our course we address the following questions:
MACHINE LEARNING 3
What is the mathematical model of learning? How to quantify the diffi-

culty/hardness/complexity of a learning problem? How to choose a learning
algorithm? How to measure success of machine learning?
The syllabus of our course:
1. Supervised learning and unsupervised learning.
2. Generalization ability of machine learning.
3. Fisher metric and stochastic gradient descend.
4. Support vector machine, Kernel machine and Neural network.
Recommended Literature.
1. F. Cucker and S. Smale, On mathematical foundations of learning,
Bulletin of AMS, 39(2001), 1-49.
2. K. P. Murphy, Machine learning: a probabilistic perspective (MIT
press, 2012).
3. M. Sugiyama, Introduction to Statistical Machine Learning, Elsevier,
2016.
4. S. Shalev-Shwart, and S. Ben-David, Understanding Machine Learning:
From Theory to Algorithms, Cambridge University Press, 2014.
1. Learning, machine learning and artificial intelligence

Machine learning is the foundation of countless important applications
including speech recognition, image detection, self-driving car and many
thing more which I shall discuss today in my lecture. Machine learning
techniques are developed using many mathematical theories. In my lecture
course I shall explain the mathematical model of machine learning and how
do we design a machine which shall learn successfully.
In my today lecture I shall discuss the following topics.
1. What are learning, inductive learning and machine learning.
2. History of machine learning and artificial intelligence.
3. Current tasks and main types of machine learning.
4. Basic questions in mathematical foundation of machine learning.
1.1. Learning, inductive learning and machine learning. To start our

discussion on machine learning let us begin first with the notion of learning.
Every one from us know what is learning from our experiences at the very
early age.
(a) Small children learn to speak by observing, repeating and mimicking
adults’ phrases. At the beginning their language is very simple and often er-
roneous. Gradually they speak freely with less and less mistakes. Their way
of learning is inductive learning: from examples of words and phrases they
learn the rules of combinations of these words and phrases into meaningful
sentences.
(b) In school we learn mathematics, physics, biology, chemistry by fol-
lowing the instructions of our teachers and those in textbooks. We learn
general rules and apply them to particular cases. This type of learning is
deductive learning. Of course we also learn inductively in school by search-

ing similar patterns in new problems and then apply the most appropriate
methods possibly with modifications for solving the problem.
(c) Experimental physicists design experiments and observe the outcomes
of the experiments to validate/support or dispute/refute a statement/conjecture
on the nature of the observables. In other words experimental physicists
learn about the dependence of certain features of the observables from em-
pirical data which are outcomes of the experiments. This type of learning
is inductive learning.
In mathematical theory of machine learning, or more general, in mathe-

matical theory of learning we consider only inductive learning. Deductive
learning is not very interesting; essentially it is equivalent to performing a set
of computations using a finite set of rules and a knowledge database. Clas-
sical computer programs learn or gain some new information by deductive
learning.
Let me suggest a definition of learning, that will be updated later to be
more and more precise.
Definition 1.1. A learning is a process of gaining new knowledge, more
precisely, new correlations of features of observable by examination of em-
pirical data of the observable. Furthermore a learning is successful if the
correlations can be tested in examination of new data and will be more
precise with the increase of data.
The above definition is an expansion of Vapnik’s mathematical postula-
tion: “Learning is a problem of function estimation on the basis of empirical
data”.
Example 1.2. A classical example of learning is that of learning a physical
law by curve fitting to data. In mathematical terms, a physical law is
expressed by a function f , and data are the value yi of f at observable
points xi . Usually we also know that (or assume that) the desired function
belongs to a finite dimensional space. The goal of learning in this case is
to estimate the unknown f from a set of pairs (x1 , y1 ), · · · , (xm , ym ). For
instance, if f is assumed to be a polynomial of degree d, then f belongs to
a N -dimensional linear space, where N = d + 1, and to estimate f is the
same as to estimate the unknown coefficients w0 , · · · , wd of f , observing the
data (xi , yi ).
The most popular method of curve fitting is the least square method
which quantifies the error of the estimation in terms of the value
m
X d
X
(1.1) (fw (xi ) − yi )2 with fw (x) = wj x j
i=1 j=0
which the desired function f should minimize. If the measurements gener-

ating the data (xi , yi ) were exact, then f (xi ) would be equal to yi and the
MACHINE LEARNING 5
learning problem is an interpolation problem. But in general one expects

the values yi to be affected by noise.
The least square technique, going back to Gauss and Legendre 1, which
is computational efficient and relies on numerical linear algebra, solves this
minimization problem.
In the case of measurement noise, which is the reality according to quan-
tum physics, we need to use the language of probability theory to model
the noise and therefore to use tools of mathematical statistics in learning
theory. That is why statistical learning theory is important part of machine
learning theory.
1.2. A brief history of machine learning. Machine learning was born

as a domain of artificial intelligence and it was reorganized as a separated
field only in the 1990s. Below I recall several important events when the
concept of machine learning has been discussed by famous mathematicians
and computer scientists.
• In 1948 John von Neumann suggested that machine can do any thing
that peoples are able to do.
• In 1950 Alan Turing asked “Can machines think?” in “Computing Ma-
chine and Intelligence” and proposed the famous Turing test. The Turing
test is carried out as imitation game. On one side of a computer screen sits
a human judge, whose job is to chat to an unknown gamer on the other
side. Most of those gamers will be humans; one will be a chatbot with the
purpose of tricking the judge into thinking that it is the real human.
• In 1956 John McCarthy coined the term “artificial intelligence”.
• In 1959, Arthur Samuel, the American pioneer in the field of com-
puter gaming and artificial intelligence, defined machine learning as a field
of study that gives computers the ability to learn without being explicitly
programmed. The Samuel Checkers-playing Program appears to be the
world’s first self-learning program, and as such a very early demonstration
of the fundamental concept of artificial intelligence (AI).
In the early days of AI, statistical and probabilistic methods were em-
ployed. Perceptrons which are simple models used in statistics were used
for classification problems in machine learning. Perceptrons were later de-
veloped into more complicated neural networks. Because of many theoret-
ical problems and because of small capacity of hardware memory and slow
speed of computers statistical methods were out of favour. By 1980, ex-
pert systems, which were based on knowledge database, and inductive logic
programming had come to dominate AI. Neural networks returned back to
machine learning with success in the mid-1980s with the reinvention of a
new algorithm and thanks to increasing speed of computers and increasing
hardware memory.
1
The least-squares method is usually credited to Carl Friedrich Gauss (1809), but it
was first published by Adrien-Marie Legendre (1805)
Machine learning, reorganized as a separate field, started to flourish in

the 1990s. The current trend is benefited from Internet.
In the book by Russel and Norvig “Artificial Intelligence a modern Ap-
proach” (2010) AI encompass the following domains:
- natural language processing,
- knowledge representation,
- automated reasoning to use the stored information to answer questions
and to draw new conclusions;
- machine learning to adapt to new circumstances and to detect and extrap-
olate patterns,
- computer vision to perceive objects,
- robotics.
All the listed above domains of artificial intelligence except knowledge
representation and robotics are now considered domains of machine learning.
Pattern detection and recognition were and are still considered to be domain
of data mining but they become more and more part of machine learning.
Thus AI = knowledge representation + ML + robotics.
• representation learning, a new word for knowledge representation but
with a different flavor, is a part of machine learning.
• Robotics = ML + hardware.
Why did such a move from artificial intelligence to machine learning hap-
pen?
The answer is that we are able to formalize most concepts and model
problems of artificial intelligence using mathematical language and represent
as well as unify them in such a way that we can apply mathematical methods
to solve many problems in terms of algorithms that machine are able to
perform.
As a final remark on the history of machine learning I would like to
note that data science, much hyped in 2018, has the same goal as machine
learning: Data science seeks actionable and consistent pattern for predictive
uses. 2.
1.3. Current tasks and types of machine learning. Now I shall de-
scribe what current machine learning can perform and how they do it.
1.3.1. Main tasks of current machine learning. Let us give a short descrip-
tion of current applications of machine learning.
Classification task assigns a category to each item. In mathematical lan-
guage, a category in an element in a countable set. For example, document
classification may assign items with categories such as politics, email spam,
sports, or weather while image classification may assign items with cate-
gories such as landscape, portrait, or animal. The number of categories in
2according to Dhar, V. (2013). “Data science and prediction”. Communications of the

ACM. 56 (12): 64. doi:10.1145/2500499, see also wiki site on data science
MACHINE LEARNING 7
such tasks can be unbounded as in OCR, text classification, or speech recog-

nition. In short, a classification task is a construction of a function on the
set of items that takes value in a countable set of categories.
As we have remarked in the classical example of learning (Example 1.2),
usually we have ambiguous/incorrect measurement and we have to add a
“noise” to our measurement. If every thing would be exact, the classification
task is the classical interpolation function problem in mathematics.
Regression task predicts a real value, i.e., a value in R, for each item.
Examples of regression tasks include learning physical law by curve fitting
to data (Example 1.2) with application to predictions of stock values or
variations of economic variables. In this problem, the error of the prediction,
which is also called estimation in Example 1.2, depends on the magnitude
of the distance between the true and predicted values, in contrast with the
classification problem, where there is typically no notion of closeness between
various categories. In short, a regression task is a (construction of a) function
on the set of items that takes value in R. As in the classification task,
in regression problems we also need to take into account a “noise” from
incorrect measurement for the regression problem. 3
Density estimation task finds the distribution of inputs in some space.
Over one hundred year ago Karl Pearson (1980-1962), the founder of the
modern statistics, 4 proposed that all observations come from some proba-
bility distribution and the purpose of sciences is to estimate the parameter
of these distributions. A particular case of parameter estimation is den-
sity estimation problem. Density estimation problem has been proposed
by Ronald Fisher (1980-1962), the father of modern statistics and experi-
ment designs, 5 as a key element of his simplification of statistical theory,
namely he assumed the existence of a density function p(ξ) that governs the
randomness (the noise) of a problem of interest.
Digression. The measure ν is called dominated by µ (or absolutely contin-
uous with respect to µ), if ν(A) = 0 for every set A with µ(A) = 0. Notation:
3The term “regression” was coined by Francis Galton in the nineteenth century to
describe a biological phenomenon. The phenomenon was that the heights of descendants
of tall ancestors tend to regress down towards a normal average (a phenomenon also
known as regression toward the mean of population). For Galton, regression had only this
biological meaning, but his work was later extended by Udny Yule and Karl Pearson to a
more general statistical context: movement toward the mean of a statistical population.
Galton’s method of investigation is non-standard at that time: first he collected the data,
then he guessed the relationship model of the events.
4 He founded the world’s first university statistics department at University College
London in 1911, the Biometrical Society and Biometrika, the first journal of mathematical
statistics and biometry.
5Fisher introduced the main models of statistical inference in the unified framework of
parametric statistics. He described different problems of estimating functions from given
data (the problems of discriminant analysis, regression analysis, and density estimation)
as the problems of parameter estimation of specific (parametric) models and suggested the
maximum likelihood method for estimating the unknown parameters in all these models.
ν << µ. By Radon-Nykodym theorem, see Appendix, Subsection 10.1, we

can write
ν =f ·µ
and f is the density function of ν w.r.t. µ.
For example, the Gaussian distribution on the real line is dominated by
the canonical measure dx and we express the standard normal distribution
in terms of its density
1 1
f (x) = √ exp(− x2 ).
2π 2
The classical problem of density estimation is formulated as follows. Let
a statistical model A be a class of densities subjected to a given dominant
measure. Let the unknown density p(x, ξ), where ξ is a parameter that
belongs to the statistical model A. The problem is to estimate the parameter
ξ using i.i.d. data X1 , · · · , Xl distributed according to this unknown density
p(x, ξ).
Ranking task orders items according to some criterion. Web search, e.g.,
returning web pages relevant to a search query, is the canonical ranking
example. If the number of ranking is finite, then this task is close to the
classification problem, but not the same, since in the ranking task we need
to specify each rank during the task and not before the task as in the clas-
sification problem.
Clustering task partitions items into (homogeneous) regions. Clustering
is often performed to analyze very large data sets. Clustering is one of the
most widely used techniques for exploratory data analysis. In all disciplines,
from social sciences to biology to computer science, people try to get a
first intuition about their data by identifying meaningful groups among the
data points. For example, computational biologists cluster genes on the
basis of similarities in their expression in different experiments; retailers
cluster customers, on the basis of their customer profiles, for the purpose
of targeted marketing; and astronomers cluster stars on the basis of their
spacial proximity.
Dimensionality reduction or manifold learning transforms an initial repre-
sentation of items in high dimensional space into a space of lower dimension
while preserving some properties of the initial representation. A common
example involves pre-processing digital images in computer vision tasks.
Many of dimensional reduction techniques are linear. When the technique
is non-linear we speak about manifold learning technique. We can regard
clustering as dimension reduction too.
1.3.2. Main types of machine learning. The type of a machine learning task
is defined by the type of interaction between the learner and the environment.
More precisely we consider types of training data, i.e., the data available to
the learner before making decision and prediction, the outcomes and the test
data that are used to evaluate and apply the learning algorithm.
MACHINE LEARNING 9
Main types of machine learning are supervised, unsupervised and rein-

forcement.
• In supervised learning a learning machine is a device that receives labeled
training data, i.e., the pair of a known instance and its feature, also called
label. Examples of labeled data are emails that are labeled “spam” or “no
spam” and medical histories that are labeled with the occurrence or absence
of a certain disease. In these cases the learners output would be a spam
filter and a diagnostic program, respectively. Most of classification and
regression problems of machine learning belong to supervised learning. We
also interpret a learning machine in supervised learning as a student who
gives his supervisor a known instance and the supervisor answers with the
known feature.
• In unsupervised learning there is no additional label attached to the
data and the task is to describe structure of data. Since the examples (the
available data) given to the learning algorithm are unlabeled, there is no
straightforward way to evaluate the accuracy of the structure that is pro-
duced by the algorithm. Density estimation, clustering and dimensionality
reduction are examples of unsupervised learning problems. Most important
applications of unsupervised learning are finding association rules that are
important in market analysis, banking security and consists of important
part of pattern recognition, which is important for understand advanced AI.
Regarding a learning machine in unsupervised learning as a student, then
the student has to learn by himself without teacher. This learning is harder
but happens more often in life. At the current time, except few tasks, which
I shall consider in the next lecture, unsupervised learning is primarily de-
scriptive and experimental whereas supervised learning is more predictive
(and has deeper theoretical foundation).
• Reinforcement learning is the type of machine learning where a learner
actively interacts with the environment to achieve a certain goal. More
precisely, the learner collects information through a course of actions by
interacting with the environment. This active interaction justifies the ter-
minology of an agent used to refer to the learner. The achievement of the
agent’s goal is typically measured by the reward he receives from the en-
vironment and which he seeks to maximize. For examples, reinforcement
learning is used in self-driving car. Reinforcement learning is aimed at ac-
quiring the generalization ability in the same way as supervised learning,
but the supervisor does not directly give answers to the students questions.
Instead, the supervisor evaluates the students behavior and gives feedback
about it.
1.4. Basic questions in mathematical foundations of machine learn-

ing. Let me recall that a learning is a process of gaining knowledge on a
feature of observables by examination of partially available data. The learn-
ing is successful if we can make a prediction on unseen data, which improves
when we have more data. For example, in classification problem, the learn-
ing machine has to predict the category of a new item from a specific set,
after seeing a lot of labeled data consisting of items and their categories.
The classification task is a typical task in supervised learning where we can
explain how and why a learning machine works and how and why machine
learns successfully. Mathematical foundations of machine learning aim to
answer these questions in mathematical language.
Question 1.3. What is the mathematical model of learning?
To answer Question 1.3 we need to specify our definition of learning in a
mathematical language which can be used to build instructions for machines.
Question 1.4. How to quantify the difficulty/complexity of a learning prob-
lem?
We quantify the difficulty of a problem in terms of its time complexity,
which is the minimum time needed for performing computer program to
solve a problem, and in term of its resource complexity which measure the
capacity of data storage and energy resource needed to solve the problem.
If the complexity of a problem is very large then we cannot not learn it. So
Question 1.4 contains the sub-question “ why can we learn a problem?”
Question 1.5. How to choose a learning algorithm?
Clearly we want to have a best learning algorithm, once we know a model
of a machine learning which specifies the set of possible predictors (decisions)
and the associated error/reward function.
By Definition 1.1, a learning process is successful, if its prediction/estimation
improves with the increase of data. Thus the notion of success of learn-
ing process requires a mathematical treatment of asymptotic rate of er-
ror/reward in the presence of complexity of the problem.
Question 1.6. Is there a mathematical theory underlying intelligence?
I shall discuss this speculative question in the last lecture.
1.5. Conclusion. Machine learning is automatized learning, whose perfor-

mance is improves with increasing volume of empirical data. Machine learn-
ing uses mathematical statistics to model incomplete information and the
random nature of the observed data. Machine learning is the core part of ar-
tificial intelligence. Machine learning is very successful experimentally and
there are many open questions concerning its mathematical foundations.
Mathematical foundations of machine learning is necessary for building gen-
eral purpose artificial intelligence, also called Artificial General Intelligence
(AGI), or Universal Artificial Intelligence (UAI). The importance of math-
ematical foundations for AGI shall be clarified in the third lecture.
Finally I recommend some sources for further reading.

MACHINE LEARNING 11
• F. Cucker and S. Smale, On mathematical foundations of learning,

Bulletin of AMS, 39(2001), 1-49.
• B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman,
Building machines that learn and think like people. Behavioral and
Brain Sciences,(2016) 24:1-101, arXiv:1604.00289.
• S. J. Russell and P. Norvig, Artificial Intelligence A Modern Ap-
proach, Prentice Hall, 2010.
2. Statistical models and frameworks for supervised learning

Last week we discussed the concept of learning and examine several ex-
amples. Today I shall specify the concept of learning by presenting basic
mathematical models of supervised learning.
A model is simply a compact representation of possible data one could
observe. Modeling is central to the sciences. Models allow one to make
predictions, to understand phenomena, and to quantify, compare and falsify
hypotheses. A model for machine learning must be able to make predictions
and improves their ability to make predictions in light of new data.
The model of supervised learning I present today is based on Vapnik’s
statistical learning theory, which starts from the following concise concept
of learning.
Definition 2.1. ([Vapnik2000, p. 17]) Learning is a problem of function
estimation on the basis of empirical data.
There are two main model types for machine learning: discriminative
models and generative models. They are distinguished by the type of func-
tions we want to estimate for understanding the feature of observable.
2.1. Discriminative model of supervised learning. Let us consider a
toy example of a classification task, which like regression tasks (Example
1.2), is a typical example of supervised learning.
Example 2.2 (Toy example). A ML firm wants to estimate the potential
of applicants to new positions of developers of algorithms in ML of its firm
based on its experience that the potential of a software developer depends on
three qualities of an applicant: his/her analytical mathematical skill rated
by the mark (from 1 to 20) in his/her graduate diploma, his/her computer
sciences skill, rated by the mark (from 1 to 20) in his/her graduate diploma,
and his/her communication skill rated by the firm test (scaled from 1 to
5). The potential of an applicant for the open position is evaluated in scale
1-10. Since the position of developer of algorithm in ML will be periodically
re-opened and therefore they want to design a ML program to predict the
potential of applicants such that the program automatically will be improved
with time.
A discriminative model of supervised learning consists of the following
components.
• A domain set X (also called an input space) consists of elements, whose

features we like to learn. Elements x ∈ X are called random inputs (or ran-
dom instances) 6 which are distributed by an unknown probability measure
µX . In other words, the probability that x belongs to a subset A ⊂ X is
µX (A). The probability distribution µX models our incomplete information
about elements x ∈ X . In general we don’t know the distribution µX .
(In the toy example of a ML firm the domain set X is the set of all
applicants, more precisely, their representing features: the marks in math,
in CS, and in communication test. Hence X = [1, 20] × [1, 20] × [1, 5]. In
the regression example of learning a physical law (Example 1.2) the domain
set X is the set of all polynomials of degree at most d, hence X is identified
with Rd .)
• An output space Y, also called a label set, consists of possible features

(also called labels) y of inputs x ∈ X . We are interested in finding a predic-
tor/mapping h : X → Y such that a feature of x is h(x). If such a mapping
h exists and is measurable, the feature h(x) is distributed by the measure
h∗ (µX ). In general such a function does not exist, and we assume that there
exists only a probability measure µX ×Y on the space (X ×Y) that defines the
probability that y is a feature of x, i.e., the probability of (x, y) ∈ A ⊂ X ×Y
being a labeled pair is equal to µX ×Y (A). In general we don’t know µX ×Y .
(In the toy example the label set Y = [1, 10] is the set of all possible
potentials scaled from 1 to 10. In the example of learning a physical law
(Example 1.2) the label set is the set R of all possible value of f (x).)
• A training data is a sequence S = {(x1 , y1 ), · · · , (xn , yn )} ∈ (X × Y)n of

i. i.d. (independently identically distributed) observed labeled pairs. Thus
S is distributed by the product measure µnX ×Y on (X × Y)n . The number n
is called the size of S. S is thought as given by a “supervisor”.
• A hypothesis space H ⊂ Y X of possible predictors h : X → Y.

(In Example 2.2 we may wish to choose
H := {h : X → Y| h(x, y, z) = ax + by + cz for some a, b, c ∈ Z≥0 }
and in Example 1.2 we choose
H := {h : R → R| h is a polynomial of degree at most d } ∼ = Rd+1
to simplify our search for a best prediction.)
• The aim of a learner is to find a best prediction rule A that assigns a

training data S to a prediction hS ∈ H. In other words the learner needs to
6classically, elements of X are called random variables, where the word “variable”
means “unknown”. When X is an input space (resp. an output space) its elements are
also called independent (resp. dependent) variables. Since nowadays the word variable
has a different meaning, like [Ghahramani2013, p. 4], I would avoid “random variable” in
this situation
MACHINE LEARNING 13
find a rule, more precisely, an algorithm

[
(2.1) A: (X × Y)n → H, S → hS
n∈N
such that hS (x) predicts the label of (unseen) instance x with the less error.
• The error function, also called a risk function, measures the discrepancy
between a hypothesis h ∈ H and an ideal predictor. The error function is
a central notion in learning theory. This function should be defined as
the averaged discrepancy of h(x) and y, where (x, y) runs over X × Y.
The averaging is calculated using the probability measure µ := µX ×Y that
governs the distribution of labeled pair (x, y(x)). Thus a risk function R
must depend on µ, so we denote it by Rµ . It is accepted that the risk
function Rµ is defined as follows.
Z
L
(2.2) Rµ (f ) := L(x, y, f ) dµ
X ×Y
where L : X × Y × H → R is an instantaneous loss function that measure

the discrepancy between the value of prediction/hypothesis f at x and the
possible feature y:
(2.3) L(x, y, f ) := d(y, f (x)).
Here d : Y × Y is a non-negative function that vanishes at the diagonal

{(y, y)| y ∈ Y} of Y × Y. For example d(y, y 0 ) = |y − y 0 |2 . By taking
averaging over (X × Y) using µ, we effectively count only the points (x, y)
which are correlated as labeled pairs.
Note the expected risk function is well defined on H only if L(x, y, f ) ∈
L1 (X × Y, µ) for all f ∈ H.
• The main question of learning theory is to find necessary and sufficient

conditions for the existence of a prediction rule A in (2.1) such that the error
of hS converges to the error of an ideal predictor, or more precisely, to the
infimum of the error of h over h ∈ H, and then to construct such A.
Remark 2.3. (1) In our discriminative model of supervised learning we

model the random nature of training data S ∈ (X × Y)n via a probability
measure µn on (X × Y)n . We don’t need a probability measure on X to
model the random nature of x ∈ X . The main difficulty in search for the
best prediction rule A is that we don’t know µn , we know only training data
S distributed by µn .
(2) Note the expected risk function is well defined on H only if L(x, y, f ) ∈
L1 (X × Y, µ) for all f ∈ H. Since we don’t know µ, we should assume that
L ∈ L1 (X ×Y, ν) for any ν ∈ P0 , where P0 is a family of probability measures
on X × Y that contains the unknown µ.
(3) The quasi-distance function d : Y × Y → R induces a quasi-distance

function dn : Y n × Y n → R as follows
n
X
0 0
(2.4) n
d ([y1 , · · · , yn ], [y1 , · · · , yn ]) = d(yi , yi0 ),
i=1
L(dn )
and therefore it induces the expected loss function Rµn : H → R as follows
Z
L(dn )
Rµn (f ) = dn ([y1 , · · · , yn ], [f (x1 ), · · · , f (xn )])dµn
(X ×Y)n
Z
(2.5) =n L(x, y, f )dµ.
X ×Y
Thus it suffices to consider only Rµ (f ), if S is a sequence of i.i.d. observables.

(4) Now we show that the classical case of learning a physical law by fitting
to data, assuming exact measurement, is a “classical limit” of our discrimi-
native model of supervised learning. In the classical learning problem, since
we know the exact position S := {(x1 , y1 ), · · · , (xn , yn )} ∈ (X × Y)n , we as-
sign the Dirac (probability measure) µS := δx1 ,y1 × · · · × δxn ,yn to the space
(X × Y)n 7. Now let d(y, y 0 ) = |y − y 0 |2 , it is not hard to see that
n
n)
X
(2.6) RµL(d
S
(f ) = |f (xi ) − yi |2
i=1
coincides with the error of estimation in (1.1).

Example 2.4 (0-1 loss). Let us take H = Y X - the subset of all mapping
X → Y. The 0-1 instantaneous loss function L : X × Y × H → {0, 1} is
defined as follows: L(x, y, f ) := d(y, f (x)) = 1 − δfy (x) . The corresponding
expected 0-1 loss determines the probability of the answer f (x) that does
not correlate with x:
(2.7) Rµ(0−1)
X ×Y
(f ) = µX ×Y {(x, y) ∈ X × Y| f (x) 6= y} = 1 − µX ×Y ({x, f (x)}).
Example 2.5. Assume that x ∈ X is distributed by a probability measure
µX and its feature y is defined by y = h(x) where h : X → Y is a measurable
mapping. Denote by Γh : X → X × Y, x 7→ (x, y), the graph of h. Then
(x, y) is distributed by the push-forward measure µh := (Γh )∗ (µX ), where

(Γh )∗ µX (A) = µX Γ−1 −1

(2.8) h (A) = µ X Γh (A ∩ Γh (X ) .
Let us compute the expected 0-1 loss function for a mapping f ∈ H = Y X

w.r.t. the measure µh . By (2.7) and by (2.8) we have
(2.9) Rµ(0−1)
h
(f ) = 1 − µX (x|f (x) = h(x)).
(0−1)
Hence Rµh (f ) = 0 iff f = h µX -a. e..
7the probability that A contains S is δ (A)
S
MACHINE LEARNING 15
2.2. Generative model of supervised learning. In many cases a dis-

criminative model of supervised learning may not yield a successful learning
algorithm because the hypothesis space H is too small and cannot approx-
imate a desired prediction for a feature ∈ Y of instance x ∈ X with a
satisfying accuracy, i.e., the optimal performance error of the class H
L
(2.10) Rµ,H := inf RµL (h)
h∈H
that represents the optimal performance of a learner using H is quite large.
One of possible reasons of this failure is that, a feature y ∈ Y of x cannot
be accurately approximated (using an instantaneous loss function L) by any
function h : X → Y.
In general case we may wish to estimate the probability that y ∈ Y is
a feature of x. This is expressed in term of the conditional probability
P (y ∈ B|x) - the probability that a feature y of x ∈ X belongs to B ⊂ ΣY .
Digression. Conditional probability is one of most basic concepts in prob-

ability theory. In general we always have a prior information before taking
decision, e.g. before estimating the probability of a future event. Condi-
tional probability P (A|B) formalizes the probability of an event A given the
knowledge that event B happens. Here we assume that A, B are elements
of the sigma-algebra ΣX of a measurable space (X , ΣX ). If X is countable,
the concept of conditional probability can be defined straightforward:
P (A ∩ B)
(2.11) P (A|B) := .
P (B)
It is not hard to see that, given B, the conditional probability P (·|B) defined
in (2.11) is a probability measure on X , A 7→ P (A|B), which is called the
conditional probability measure given B, see e.g. [JP2003, Theorem 3.2, p.
16] for a proof. In its turn, by taking integration over X using the conditional
probability P (·|B), we obtain the notion of conditional expectation, given B,
which shall be denoted by EP (·|B) , see e.g. [JP2003, Definition 23.1, p. 197].
We can think that the conditional expectation given B is a function on ΣX .
In general case when X is not countable the definition of conditional
probability is more subtle, especially when we have to define P (A|B), where
B has null-measure. A typical situation is the case B = h−1 (x0 ), where
h : X → Z is a random variable (a measurable mapping). To treat this
important case we need to define first the notion of conditional expectation,
see Subsection 10.2 in Appendix. What is important for our applications in
many case is the notion of conditional distribution P (A|h(x) = x0 ) which is
regular w.r.t. to a measure µX , i.e., there exists a density function f (x|x0 )
on X such that Z
P (A|h(x) = x0 ) = f (x|x0 )µX .
A
We may also wish to estimate the joint distribution µ := µX ×Y of i.i.d.

labeled pairs (x, y). Once we know µ we know the expected risk RµL for an
instantaneous loss function L, and hence a minimizing sequence {hi ∈ H}

of RµL
lim RµL (hi ) = Rµ,H
L
n→∞
can be determined. In many cases we can find an explicit formula for the
Bayes optimal predictor that minimizes the expected risk value RµL , once µ
is known.
Exercise 2.6 (The Bayes Optimal Predictor). ([SSBD2014, p. 46]) If Y =
Z2 there is an explicit formula for a Bayes classifier, called the Bayes optimal
predictor. Given any probability distribution D over X × {0, 1}, the best
label predicting function from X to {0, 1} will be

1 if r(x) := D[y = 1|x] ≥ 1/2
fD (x) =
0 otherwise
Show that for every probability distribution D, the Bayes optimal predictor
fD is optimal. In other words for every classifier g we have RD (fD ) ≤ RD (g).
Exercise 2.7 (Regression optimal Bayesian estimator). In regression prob-
lem the output space Y is R. Let us define the following embedding
i1 : RX → RX ×Y : [i1 (f )](x, y) := f (x),
i2 : RY → RX ×Y : [i2 (f )](x, y) := f (y).

(These embeddings are adjoint to the projections: X : X × R → X and
X × Y → Y.) For a given probability measure µ on X × R we set
L2 (X , µ) := {f ∈ RX | i1 (f ) ∈ L2 (X × R, µ)},
L2 (R, µ) := {f ∈ RR | i2 (f ) ∈ L2 (X × R, µ)}.
Now we let F := L2 (X , µ). Let Y denote the function on R such that
Y (y) = y. Assume that Y ∈ L2 (R, µ) and define the quadratic loss function
L:X ×Y ×F →R
(2.12) L(x, y, h) := |y − h(x)|2 ,
RµL (h) = Eµ |Y − h(x)|2 = |i2 (Y ) − i1 (h)|2L2 (X ×R,µ) .

(2.13)
The expected risk RµL is called the L2 -risk, also known as mean squared error

(MSE). Show that the regression function r(x) := Eµ i2 (Y )|X = x belongs
to F and minimizes the L2 (µ)-risk.
Definition 2.8. A model of supervised learning with the aim to estimate
the conditional distribution P (y ∈ B|x), in particular, a conditional density
function p(y|x), or joint distribution of (x, y) is called a generative model of
supervised learning.
MACHINE LEARNING 17
Remark 2.9. Generative models give us more complete information of the

correlation between a feature y and an instance x but they are more com-
plicated, since even in the regular case, a conditional density function is a
function of two variables x and y and we cannot express this correlation
as a dependence of y from x. In fact, we could interpret a density func-
tion p(y|x) as a probabilistic mapping from X to Y: p(y|x) indicates the
probability that the value of a mapping in consideration at x is equal to y.
In many practical cases, following Fisher suggestion, [Vapnik2006, p. 481],
[Sugiyama2016, p. 236], we often assume that y can be expressed in terms
of a function of x up to a white noise, i.e.
(2.14) y = f (x) + ε
where ε is a random error (a measurable function on X ) with zero expecta-
tion i.e., Eµ (ε) = 0.
This simplified setting of a supervised learning is a discriminative model.
2.3. Empirical Risk Minimization and overfitting. In a discriminative
model of supervised learning our aim is to construct a prediction rule A that
assigns a predictor hS to each sequence
S = {(x1 , y1 ), · · · , (xn , yn )} ∈ (X × Y)n
of i.i.d. labeled data such that the expected error RµL (hS ) tends to the
optimal performance error Rµ,HL of the class H. One of most popular ways
to find a prediction rule A is to use the Empirical Risk Minimization.
For a loss function
L : X × Y × H → R,
and a training data S ∈ (X × Y)n we define the empirical risk of a predictor
h as follows
n
1X
(2.15) R̂SL (h) := L(xi , yi , h) ∈ R.
n
i=1
If L is fixed, then we also omit the superscript L.
The empirical risk is a function of two variables: the “empirical data” S
and the predictor h. Given S a learner can compute R̂S (h) for any function
h : X → Y. A minimizer of the empirical risk should have also “approxi-
mately” minimize the expected risk. This is the empirical risk minimization
principle, abbreviated as ERM.
Remark 2.10. We note that
L(d) 1 L(dn )
(2.16) R̂S (h) = R (h)
n µS
where µS is the Dirac measure on (X × Y)n associated to S, see (2.6). If h
is fixed, by the weak law of large numbers, the RHS of (2.16) converges in
probability to the expected risk RµL (h), so we could hope to find a condition
under which the RHS of (2.16) for a sequence of hS , instead of h, converges

L .
to Rµ,H
Example 2.11. In this example we shall show the failure of ERM in certain
cases. The 0-1 empirical risk corresponding to 0-1-loss function L : X × Y ×
Y X → {0, 1} is defined as follows
|i ∈ [n] : h(xi ) 6= yi |
(2.17) R̂S0−1 (h) :=
n
for a training data S = {(x1 , y1 ), · · · , (xn , yn )} and a function h : X → Y.
Now we assume that labeled data (x, y) is generated by a map f : X → Y,
i.e., y = f (x), and further more, x is distributed by a measure µX on X as in
Example 2.5. Then (x, f (x)) is distributed by the measure µf = (Γf )∗ (µX ).
Let H = Y X . Then f ∈ H and Rµ0−1 f
(f ) = 0. For any given ε > 0 and
any n we shall find a map f , a measure µX , and a predictor hSn such that
R̂S0−1
n
(hSn ) = 0 and Rµ0−1f
(hSn ) = ε, which shall imply that the ERM is
invalid in this case.
Set

f (xi ) if there exists i ∈ [n] s.t. xi = x
(2.18) hSn (x) =
0 otherwise.
Clearly R̂S0−1
n
(hSn ) = 0. We also note that hSn (x) = 0 except finite (at
most n) points x in X .
Let X be the unit cube I k in Rk and Y = Z2 . Let µ0 be the Lebesgue
measure on I k , k ≥ 1.We decompose X into a disjoint union of two measur-
able subsets A1 and A2 such that µX (A1 ) = ε. Let f : X → Z2 be equal 1A1
- the indicator function of A1 . By (2.7) we have
(2.19) Rµf (hSn ) = µX ({x ∈ X | hSn (x) 6= 1A1 (x)}).
Since hSn (x) = 0 a.e. on X it follows from (2.19) that
Rµf (hSn ) = µX (A1 ) = ε.
Such a predictor hSn is said to be overfitting, i.e. it fits well to training
data but not real life.
Exercise 2.12 (Empirical risk minimization). Let X × Y = Rd × R and
F := {h : X → Y| ∃v ∈ Rd : h(x) = hv, xi} be the class of linear functions in
n
Y X . For S = (xi , yi ) i=1 ∈ (X × Y)n and the quadratic loss L (defined in
(2.12)), find the hypothesis hS ∈ F that minimizes the empirical risk R̂SL .
The phenomenon of overfitting suggests the following questions concern-
ing ERM principle [Vapnik2000, p.21]
1) Can we learn in discriminative model of supervised learning using the
ERM principle?
2) If we can learn, we would like to know the rate of convergence of the
learning process as well as construction method of learning algorithms.
We shall address these questions later in our course and recommend the
books by Vapnik on statistical learning theory for further reading.
MACHINE LEARNING 19
2.4. Conclusion. In this lecture we learn discriminative model of super-

vised learning which consists of a hypothesis space H of functions X → Y
and an expected risk function RµL on H where L is an instantaneous loss
function and µ ∈ P(X × Y) is a unknown probability distribution of la-
beled pairs on X × Y. The aim of a learner is to find a prediction rule
A : S 7→ hS ∈ H such that (x, hS (x)) approximates the labeled training
data best, assuming that S is a sequence of i.i.d. labeled training data. The
ERM principle suggests that we could choose hS to be the minimizer of the
empirical risk R̂S and we hope that as the size of S increases the expected
error RµL (hS ) converges to the optimal performance error Rµ,H L . Without
further condition on H and L the ERM principle does not work.
3. Statistical models and frameworks for unsupervised

learning and reinforcement learning
Last week we learned discriminative and generative models of supervised
learning. The starting point of our models is Vapnik’s postulate: learning
is a problem of function estimation on the basis of empirical data. In super-
vised learning we are given i.i.d. labeled data and the problem is to predict
the label of a new/unseen instance. If we regard this prediction as a function
X → Y (up to a negligible noise) then we have to find/estimate a function
from a hypothesis space H ⊂ Y X such that its expected error is as small
as possible, using labeled data. If we wish instead to estimate the condi-
tional probability p(y|x) or the joint distribution of the labeled data then
our model is generative. The error function is a central notion of learning
theory that specifies the idea of “best approximation”, “best predictor”.
Today we shall study statistical models of machine learning for several
important tasks in unsupervised learning: density estimation, clustering,
dimension reduction, manifold learning and a mathematical model for rein-
forcement learning. The key problem is to specify the error function that
measures the accuracy of an estimator or the fitness of a decision.
3.1. Statistical models and frameworks for density estimation. Let
X be a measurable space and denote by P(X ) the space of all probability
measures on X . In a density estimation problem we are given a sequence
of observables Sn = (x1 , · · · , xn ) ∈ X n , which are i.i.d. by unknown proba-
bility measure µu . We have to estimate the measure µu ∈ P(X ). Further-
more, having a prior information, we assume that µu belongs to a subset
P ⊂ P(X ), which is also called a statistical model. Simplifying further, we
assume that P consists of probability measures that are dominated by a
measure µ0 ∈ P(X ). Thus we regard P as a family of density functions
on X . If P is finite dimensional, then estimating µu ∈ P is called a para-
metric problem of density estimation, otherwise it is called a nonparametric
problem. The density estimation problem encompasses the problem of esti-
mating the joint distribution in the generative model of supervised learning
as particular case.
• In the parametric density estimation problem we assume that P ⊂ P(X )

is parameterized by a nice parameter set Θ, e.g. Θ is an open set of Rn .
That is, there exists a surjective map p : Θ → P, θ 7→ pθ µ0 , which is usually
(in classical statistics) assumed to be a 1-1 map. 8 In this lecture we shall
assume that Θ is an open subset of Rn and p is a 1-1 map. Thus we shall
identify P with Θ and the parametric density estimation in this case is
equivalent to estimating the parameter p−1 (µu ) ∈ Θ. As in mathematical
models for supervised learning, we define an expected risk function Rµ :
Θ → R by averaging an instantaneous loss function L : X × Θ → R using
the unknown probability measure µu which we have to estimate. Usually this
setting of density estimation L given by the minus log-likelihood function
(3.1) L(x, θ) = − log pθ (x).
Hence the expected risk function Rµ : Θ → R is the expected log-likelihood
function:
Z
(3.2) Rµ (θ) = RµLu (θ) = − log pθ (x)pu (x)dµ0
X
where µu = pu µ0 . Given a data Sn = (x1 , · · · , xn ) ∈ X n , by (3.1), the

corresponding empirical risk function is
n
X
(3.3) R̂SLn (θ) =− log pθ (xi ) = − log[pnθ (Sn )],
i=1
where pnθ (Sn ) is the density of the probability measure µnθ on X n . It follows
that the minimizer θ of the empirical risk R̂SLn is the maximizer of the log-
likelihood function log[pnθ (Sn )]. According to ERM principle, the minimizer
θ of R̂SLn should provide an “approximation” of the density pu of the unknown
probability measure µu .
Remark 3.1. (1) For X = R the ERM principle for the expected log-
likelihood function holds. Namely one can show that the minimum of the
risk functional in (3.2), if exists is attained at a function p∗u which may differ
from pu only on a set of zero measure, see [Vapnik1998, p.30] for a proof.
(2) Note that minimizing the expected log-likelihood function Rµ (θ) is
the same as minimizing the following modified risk function [Vapnik2000,
p.32]
Z Z
∗ pθ (x)
(3.4) Rµ (θ) := Rµ (θ) + log pu (x)pu (x)dµ0 = − log pu (x)dµ0 .
X X pu (x)
The expression on the RHS of (3.4) is the Kullback-Leibler divergence KL(pθ µ0 , µu )
that is used in statistics for measuring the divergence between pθ µ0 and µu .
8For many important statistical models in machine learning the condition 1-1 map does
not hold and we refer to [AJLS2017] for a general treatment.
MACHINE LEARNING 21
The Kullback-Leibler divergence, unlike the expected log-likelihood func-

tion, can be defined on the space P(X ) of all probability measures on X . It
is a quasi-distance, i.e., it satisfies the following properties:
(3.5) KL(µ, µ0 ) ≥ 0 and KL(µ, µ0 ) = 0 iff µ = µ0 .
Thus a maximizer of the expected log-likelihood function minimizes the KL-
divergence. This justifies the choice of the expected risk function RµLu .
(3) It is important to find quasi-distance functions on P(Ω) that satisfying
certain natural statistical requirement. This problem has been considered
in information geometry, see [Amari2016, AJLS2017] for further reading.
(4) Traditionally in statistics people consider only measures that can be
expressed as a density function w.r.t. a given (dominant) measure. This
assumption holds in classical situations, when we consider only finite di-
mensional families of probability measures. Currently in machine learning
one also uses infinite dimensional family of probability measures on X that
cannot be dominated by any measure on X , for examples, the infinite di-
mensional family of posterior distributions of Dirichlet processes which are
used in clustering.
• A popular nonparametric technique for estimating density functions on
X = Rn using empirical data Sn ∈ X n is the kernel density estimation
(KDE) [Tsybakov2009, p. 2]. For understanding the idea of KDE we shall
consider only the case X = R and µ0 = dx. Let
Z t
F (t) = pu (x)dx
−∞
be the corresponding cumulative distribution function. Consider the empir-
ical distribution function
n
1X
F̂Sn (t) = 1(xi ≤t) .
n
i=1
By the strong law of large numbers we have
a.s.
lim FSn (t) = F (t).
n→∞
How can we estimate the density pu ? Note that for sufficiently small
h > 0 we can write an approximation
F (t + h) − F (t − h)
pu (t) ∼
= .
2h
Replacing F by F̂Sn we define the Rosenblatt estimator
F̂Sn (t + h) − F̂Sn (t − h)
(3.6) p̂R
Sn (t) := ,
2h
which can be rewritten in the following form
n n
1 X 1 X xi − t
(3.7) p̂R
Sn (t) = 1 (t−h≤xi ≤t+h) = K0 ( )
2nh nh h
i=1 i=1
where K0 (u) := 21 1(−1≤u≤1) . A simple generalization of the Rosenblatt

estimator is given by
n
PR 1 X xi − t
(3.8) p̂Sn (t) = K( )
nh h
i=1
R
where K : R → R is an integrable function satisfying K(u)du = 1. Such a
function K is called kernel and the parameter h is called bandwidth of the
kernel density estimator (3.8), also called the Parzen-Rosenblatt estimator.
To measure the accuracy of the estimator p̂PSnR we use a trick, namely

instead of using the L2 -estimation
Z
ˆPR
M SE(fSn ) := |p̂PSnR (x) − pu (x)|2 dx,
R
we consider M SE(fˆP R , x0 ) of fˆSPnR w.r.t. a given point x0 ∈ R, averaging

over the population of all possible data Sn ∈ Rn :
(3.9) M SE(fˆP R , x0 ) := Epn [(p̂P R (x0 ) − pu (x0 ))2 ]dSn .
u Sn
Note that the RHS measures the accuracy of p̂PSnR (x0 ) probably w.r.t. Sn ∈
Rn . This is an important concept of accuracy in the presence of uncertainty.
It has been proved that under certain condition on the kernel function K
and the infinite dimensional statistical model P of densities the M SE(fˆP R , x0 )
converges to zero uniformly on R as h goes to zero [Tsybakov2009, Theorem
1.1, p. 9].
Remark 3.2. In this Subsection we discuss two popular models of machine
learning for density estimation using ERM principle, which works under cer-
tain conditions. We postpone important Bayesian model of machine learning
and stochastic approximation method for finding minimizer of the expected
risk function using i.i.d. data to later parts of our course.
3.2. Statistical models and frameworks for clustering. Clustering is
the process of grouping similar objects x ∈ X together. There are two
possible types of grouping: partitional clustering, where we partition the
objects into disjoint sets; and hierarchical clustering, where we create a
nested tree of partitions. To formalize the notion of similarity we introduce
a quasi-distance function on X . That is, a function d : X × X → R+ that is
symmetric, satisfies d(x, x) = 0 for all x ∈ X .
A popular approach to clustering starts by defining a cost function over
a parameterized set of possible clusterings and the goal of the clustering
algorithm is to find a partitioning (clustering) of minimal cost. Under this
paradigm, the clustering task is turned into an optimization problem. The
function to be minimized is called the objective function, which is a function
G from pairs of an input (X , d), and a proposed clustering solution C =
(C1 , · · · , Ck ), to positive real numbers. Given G, the goal of a clustering
algorithm is defined as finding, for a given input (X , d), a clustering C
MACHINE LEARNING 23
so that G((X, d), C) is minimized. In order to reach that goal, one has
to apply some appropriate search algorithm. As it turns out, most of the
resulting optimization problems are NP-hard, and some are even NP-hard
to approximate.
Example 3.3. The k-means objective function is one of the most popular
clustering objectives. In k-means the data is partitioned into disjoint sets
C1 , · · · , Ck where each Ci is represented by a centroid µi := µi (Ci ). It
is assumed that the input set X is embedded in some larger metric space
(X 0 , d) and µi ⊂ X 0 . We define µi as follows
X
µi (Ci ) := arg min0 d(x, µ).
µ∈X
x∈Ci
The k-means objective function Gk is defined as follows

k X
X
(3.10) Gk ((X, d), (C1 , · · · , Ck )) = d(x, µi (Ci )).
i=1 x∈Ci
The k-means objective function G is used in digital communication tasks,

where the members of X may be viewed as a collection of signals that have to
be transmitted. For further reading on k-means algorithms, see [SSBD2014,
p. 313].
Remark 3.4. The above formulation of clustering is deterministic. We

consider also more complicated probabilistic clustering, where the output is a
function assigning to each domain point x ∈ X , a vector (p1 (x), · · · , pk (x)),
where pi (x) = P [x ∈ Ci ] is the probability that x belongs to cluster Ci .
3.3. Statistical models and frameworks for dimension reduction

and manifold learning. A central cause of the difficulties with unsuper-
vised learning is the high dimensionality of the random variables being mod-
eled. As the dimensionality of input x grows, any learning problem signifi-
cantly gets harder and harder. Handling high-dimensional data is cumber-
some in practice, which is often referred to as the curse of dimensionality.
Hence various methods of dimensionality reduction are introduced. Dimen-
sion reduction is the process of taking data in a high dimensional space and
mapping it into a new space whose dimension is much smaller.
• Classical (linear) dimension reduction methods. Given original data
Sn := {xi ∈ Rd | i ∈ [1, n]} we want to embed it into Rm , m < d, then we
would like to find a linear transformation W ∈ Hom(Rd , Rm ) such that
W (Sn ) := {W (xi )} ⊂ Rm .
To find the “best” transformation W = W(Sn ) we define an error function

on the space Hom(Rd , Rm ) and solve the associated optimization problem.
Example 3.5. A popular linear method for dimension reduction is called

Principal Component Analysis (PCA), which has another name SVD (sin-
gular value decomposition.) Given Sm ⊂ Rd , we use a linear transforma-
tion W ∈ Hom(Rd , Rn ), where n < d, to embed Sm into Rd . Then, a
second linear transformation U ∈ Hom(Rn , Rd ) can be used to (approxi-
mately) recover Sm from its compression W (Sm ). In PCA, we search for
W and U to be a minimizer of the following reconstruction error function
RSn : Hom(Rd , Rn ) × Hom(Rn , Rd ) → R
m
X
(3.11) R̂Sn (W, U ) = ||xi − U W (xi )||2
i=1
where ||, || denotes the quadratic norm.
Exercise 3.6. ([SSBD2014, Lemma 23.1, p.324]) Let (W, U ) be a minimizer
of R̂Sm defined in (3.11). Show that U is an orthogonal embedding and
W ◦ U = Id|Rn .
Let Homg (Rd , Rn ) denotes the set of all orthogonal projections from Rd
to Rn and Homg (Rn , Rd ) the set of all orthogonal embeddings from Rm
to Rd . Let F ⊂ Homg (Rd , Rn ) × Homg (Rn , Rd ) be the subset of all pairs
(W, U ) of transformations such that W ◦ U = Id|Rn . Exercise 3.6 implies
that any minimizer (W, U ) of R̂Sm is an element of F.
Exercise 3.7. ([SSBD2014, Theorem 3.23, p. 325]) Let C(Sm ) ∈ End(Rd )
be defined as follows
m
X
C(Sm )(v) := hxi , vixi .
i=1
Rd
Assume that ξ1 , · · · , ξd ∈ are eigenvectors of C(Sm ) with eigenvalues
λ1 ≥ · · · ≥ λd ≥ 0. Show that any (W, U ) ∈ F with W (xj ) = 0 for all
j ≥ m + 1 is a solution of (3.11).
Thus a PCA problem can be solved using linear algebra method.
• Manifold learning and autoencoder. In real life data are not concentrated
on a linear subspace of Rd but around a submanifold M ⊂ Rd . The current
challenge in ML community is that to reduce representation of data in Rd
using all the data in Rd but only use only data concentrated around M . For
that purpose we use autoencoder, which is a non-linear analogue of PCA.
In an auto-encoder we learn a pair of functions: an encoder function
ψ : Rd → Rk , and a decoder function ϕ : Rk → Rd . The goal of the learning
process is to P find a pair of functions (ψ, ϕ) such that the reconstruction
error
n
X
RSn (ψ, ϕ) := ||xi − ϕ(ψ(xi ))||2
i=1
is small. We therefore must restrict ψ and ϕ in some way. In PCA, we
constrain k < d and further restrict ψ and ϕ to be linear functions.
MACHINE LEARNING 25
Remark 3.8. Modern autoencoders have generalized the idea of an encoder

and a decoder beyond deterministic functions to stochastic mappings pstoch :
X → Y, (x, y) 7→ p(y|x). For further reading I recommend [SSBD2014],
[GBC2016] and [Bishop2006, §12.2, p.570].
3.4. Statistical model and framework for reinforcement learning. A
reinforcement learning agent interacts with its environment in discrete time
steps. At each time t, the agent receives an observation ot , which typically
includes the reward rt . It then chooses an action at from a set A of available
actions, which is subsequently sent to the environment. The environment
moves to a new state st+1 in a set S of available states and the reward rt+1
associated with the transition ot+1 := (st , at , st+1 ) is determined. The goal
of a reinforcement learning agent is to collect as much reward as possible.
The agent can (possibly randomly) choose any action as a function of the
history. The uncertainly in reinforcement learning is expressed in terms
of a transition probability P r[s0 |s, a] - distribution over destination states
s0 = δ(s, a) and in terms of a reward probability P r[r0 |s, a] - distribution
over rewards returned r0 = r(s, a). Thus the mathematical model of rein-
forcement learning is a Markov decision process. For further reading, see
[MRT2012, chapter 14].
3.5. Conclusion. In this lecture we learned mathematical models for un-
supervised learning, where for each empirical data Sn ∈ X n we choose an
empirical risk/error function R̂Sn on a space of possible hypotheses as a
quasi-distance between a hypothesis and the true (desired) hypothesis. In
some case (e.g. in parametric density estimation) we can interpret this em-
pirical risk function to be derived from an expected risk/error function as
in the models of supervised learning. The main problem is to show the con-
vergence of minimizers of R̂Sn to the desired hypothesis as n goes to zero
and Sn are i.i.d. by an unknown measure on X n .
4. Fisher metric and maximum likelihood estimator

In the last lecture we considered several mathematical models in unsuper-
vised learning. The most important problem among them is the problem of
density estimation, which is also a problem in generative models of super-
vised learning and an important problem of classical statistics. The error
function in density estimation problem can be defined as the expected log-
likelihood function, which combining with the ERM principle leads to the
well known maximum likelihood estimator. The popularity of this estima-
tor stems from its asymptotic accuracy, also called consistency, which holds
under mild conditions. Today we shall study MLE using the Fisher metric,
the associated MSE function and the Cramér-Rao inequality. We also clarify
the relation between the Fisher metric and the Kullback-Leibler divergence.
4.1. The space of all probability measures and total variation norm.
We begin today lecture with our investigation of natural geometry of P(X )
for an arbitrary measurable space (X , Σ). This geometry induces the Fisher
metric on any statistical model P ⊂ P(X ) satisfying a mild condition.
Let us fix some notations. Recall that a signed finite measure µ on X is
a function µ : Σ → R which satisfies all axioms of a measure except that µ
needs not take non-negative value. Now we set
M(X ) := {µ : µ a finite measure on X },
S(X ) := {µ : µ a signed finite measure on X }.
It is known that S(X ) is a Banach space whose norm is given by the total
variation of a signed measure, defined as
n
X
kµkT V := sup |µ(Ai )|
i=1
where the supremum is taken over all finite partitions X = A1 ∪˙ . . . ∪A

˙ n with
disjoint sets Ai ∈ Σ(X ) (see e.g. [Halmos1950]). Here, the symbol ∪˙ stands
for the disjoint union of sets.
Let me describe the total variation norm using the Jordan decomposition
theorem for signed measures, which is an analogue of the decomposition
theorem for a measurable function. For a measurable function φ : X →
[−∞, ∞] we define φ+ := max(φ, 0) and φ− := max(−φ, 0), so that φ± ≥ 0
are measurable with disjoint support, and
(4.1) φ = φ+ − φ− |φ| = φ+ + φ− .
Similarly, by the Jordan decomposition theorem, each measure µ ∈ S(X ) can
be decomposed uniquely as
(4.2) µ = µ + − µ− with µ± ∈ M(X ), µ+ ⊥ µ− .
That is, there is a Hahn decomposition X = X+ ∪X˙ − with µ+ (X− ) =
µ− (X+ ) = 0 (in this case the measures µ+ and µ− are called mutually
singular). Thus, if we define
|µ| := µ+ + µ− ∈ M(X ),
then (4.2) implies
(4.3) |µ(A)| ≤ |µ|(A) for all µ ∈ S(X ) and A ∈ Σ(X ),
so that
kµkT V = k |µ| kT V = |µ|(X ).
In particular,
P(X ) = {µ ∈ M(X ) : kµkT V = 1}.
Next let us consider important subsets of dominated measures and equiv-
alent measures in the Banach space S(X ) which are most frequently used
subsets in statistics and ML.
Given a measure µ0 ∈ M(X ), we let
S(X , µ0 ) := {µ ∈ S(X ) : µ is dominated by µ0 }.
MACHINE LEARNING 27
By the Radon-Nikodym theorem, we may canonically identify S(X , µ0 )

with L1 (X , µ0 ) by the correspondence
(4.4) ıcan : L1 (X , µ0 ) −→ S(X , µ0 ), φ 7−→ φ µ0 .
Observe that ıcan is an isomorphism of Banach spaces, since evidently
Z
kφkL1 (X ,µ0 ) = |φ| dµ0 = kφ µ0 kT V .
X
Example 4.1. Let Xn := {ω1 , · · · , ωn } be a finite set of n elementary events.
Let δωi denote the Dirac measure concentrated at ωi . Then
Xn
S(Xn ) = {µ = xi δωi | xi ∈ R} = Rn (x1 , · · · , xn )
i=1
and
Xn
M(Xn ) = { xi δωi | xi ∈ R≥0 } = Rn≥0 .
i=1
For µ ∈ M(Xn ) of the form
k
X
µ= ci δi , ci > 0
i=1
ci . Thus the space L1 (Xn , µ) with the total variation
P
we have ||µ||T V =
norm is isomorphic to Rk with the l1 -norm. The space P(Xn ) with the
induced total variation topology
n
P is homeomorphic to a (n − 1)-dimensional
simplex {(c1 , · · · , cn ) ∈ R+ | i ci = 1}.
Exercise 4.2. ([JLS2017]) For any countable family of signed measures
{µn ∈ S(X )} show that there exists a measure µ ∈ M(X ) dominating all
measures µn .
Remark 4.3. On (possibly infinite dimensional) Banach spaces we can do
analysis, since we can define the notion of differentiable mappings. Let V
and W be Banach spaces and U ⊂ V an open subset. Denote by Lin(V, W )
the space of all continuous linear map from V to W . A map φ : U → W is
called differentiable at x ∈ U , if there is a bounded linear operator dx φ ∈
Lin(V, W ) such that
kφ(x + h) − φ(x) − dx φ(h)kW

(4.5) lim = 0.
h→0 khkV
In this case, dx φ is called the (total) differential of φ at x. Moreover, φ is
called continuously differentiable or shortly a C 1 -map, if it is differentiable
at every x ∈ U , and the map dφ : U → Lin(V, W ), x 7→ dx φ, is continuous.
Furthermore, a differentiable map c : (−ε, ε) → W is called a curve in W.
A map φ from an open subset Θ of a Banach space V to a subset X of a
Banach space W is called differentiable, if the composition i ◦ φ : Θ → W is
differentiable.
4.2. Fisher metric on a statistical model. Given a statistical model

P ⊂ P(X ) we shall show that P is endowed with a nice geometric structure
induced from the Banach space (S(X ), ||, ||T V ). Under a mild condition this
implies the existence of the Fisher metric on P . Then we shall compare the
Fisher metric and the Kullback-Leibler divergence.
We study P by investigating the space of functions on P (which is a linear
infinite dimensional vector space) and by investigating its dual version: the
space of all curves on P . The tangent fibration of P describes the first order
approximation of the later space.
Definition 4.4. ([AJLS2017, Definition 3.2, p. 141]) (1) Let (V, k · k) be a
Banach space, X ⊂ V an arbitrary subset and x0 ∈ X . Then v ∈ V is called
a tangent vector of X at x0 , if there is a curve c : (−ε, ε) → X ⊂ V such
that c(0) = x0 and ċ(0) = v.
(2) The tangent (double) cone Cx X at a point x ∈ X is defined as the
subset of the tangent space Tx V = V that are tangent to a curve lying in
X . The tangent space Tx X is the linear hull of the tangent cone Cx X .
(3) The tangent cone fibration CX (resp. the tangent fibration T X ) is
the union ∪x∈X Cx X (resp. ∪x∈X Tx X ) is a subset of V × V and therefore is
endowed with the induced topology.
Exercise 4.5. (cf. [AJLS2018, Theorem 2.1]) Let P be a statistical model.
Show that any v ∈ Tξ P is dominated by ξ. Hence the logarithmic represen-
tation of v
log v := dv/dξ
1
is an element of L (X , ξ).
Example 4.6. Assume that a statistical model P consists of measures dom-
inated by a measure µ0 and therefore P is regarded as a family of density
functions on X , namely
(4.6) P = {f · µ0 | f ∈ L1 (X , µ0 )}.
Then a tangent vector v ∈ Tξ P has the form v = f˙(0)·µ0 and its logarithmic
representation is expressed as follows
dv f˙(0) d
(4.7) log v = = = log f (t).
dξ f (0) dt |t=0
Next we want to put a Riemannian metric on P i.e., to put a positive
quadratic form g on each tangent space Tξ P . By Exercise 4.5, the logarith-
mic representation log(Tξ P ) of Tξ P is a subspace in L1 (X , ξ). The space
L1 (X , ξ) does not have a natural metric but its subspace L2 (X , ξ) is a Hilbert
space.
Definition 4.7. (1) A statistical model P that satisfies
(4.8) log(Tξ P ) ⊂ L2 (X , ξ)
for all ξ ∈ P is called almost 2-integrable.
MACHINE LEARNING 29
(2) Assume that P is an almost 2-integrable statistical model. For each

v, w ∈ Cξ P the Fisher metric on P is defined as follows
Z
(4.9) g(v, w) := hlog v, log wiL2 (X ,ξ) = log v · log w dξ.
X
(3) An almost 2-integrable statistical model P is called 2-integrable, if the

function v 7→ |v|g is continuous on CP .
Since Tξ P is a linear hull of Cξ P , the formula (4.9) extends uniquely to
a positive quadratic form on Tξ P , which is also called the Fisher metric.
Example 4.8. Let P ⊂ P(X ) be a 2-integrable statistical model that is
parameterized by a differentiable map p : Θ → P, θ 7→ pθ µ0 , where Θ is an
open subset in Rn . It follows from (4.7) that the Fisher metric on P has the
following form
Z
∂v pθ ∂w pθ
(4.10) g|p(θ) (dp(v), dp(w)) = · pθ dµ0 ,
X pθ pθ
for any v, w ∈ Tθ Θ.
Remark 4.9. (1) The Fisher metric has been defined by Fisher in 1925 to
characterize “information” of a statistical model. One of most notable ap-
plications of the Fisher metric is the Cramér-Rao inequality which measures
our ability to have a good density estimator in terms of geometry of the
underlying statistical model, see Theorem 4.12 below.
(2) The Fisher metric gp(θ) (dp(v), dp(v)) of a parametrized statistical
model P of dominated measures in Example 4.8 can be obtained from the
Taylor expansion of the Kullback-Leibler divergence I(p(θ), p(θ + εv)), as-
suming that log pθ is continuously differentiable in all partial derivative in θ
up to order 3. Indeed we have
Z
pθ (x)
I(p(θ), p(θ + εv)) = pθ (x) log dµ0
X p θ+εv (x)
Z
(4.11) = −ε pθ (x)∂v log pθ (x)dµ0
X
Z
2
(4.12) −ε pθ (x)(∂v )2 log pθ (x)dµ0 + O(ε3 ).
X
Since logθ (x) is continuously differentiable in θ up to order 3, we can apply

differentiation under the integral sign, see e.g. [Jost2005, Theorem 16.11, p.
213] to (4.11), which then must vanish, and integration by part to (4.12).
Hence we obtain
I(p(θ), p(θ + εv)) = ε2 gp(θ) (dp(v), dp(v)) + O(ε3 )
what is required to prove.
4.3. MSE and Cramér-Rao inequality. Set ϕl : l ◦ ϕ for any l ∈ (Rn )∗

and
L2ϕ (P, X ) := {σ̂ : X → P | ϕl ◦σ̂ ∈ L2 (X , ξ) for all ξ ∈ P and for all l ∈ (Rn )∗ }.
If σ̂ ∈ L2ϕ (P, X ), then the following L2 -risk function, also called MSE, is
well-defined for any l, k ∈ (Rn )∗
(4.13) M SEξϕ [σ̂](l, k) := Eξ [(ϕl ◦ σ̂ − ϕl ◦ ξ) · (ϕk ◦ σ̂ − ϕk ◦ ξ)].
Thus the function M SEξϕ [σ̂](l, l) on P is the expected risk of the quadratic
instantaneous loss function
Ll : X × P → R, Ll (x, ξ) = |ϕl ◦ σ̂(x) − ϕl ◦ ξ|2 .
Next we define the mean value ϕσ̂ of a ϕ-estimator ϕ ◦ σ̂ as a Rn -valued
function on P :
Z
(4.14) hϕσ̂ (ξ), li := Eξ (ϕl ◦ σ̂) = ϕl ◦ σ̂ dξ
X
for any l∗ ∈ (Rn )∗ .
Definition 4.10. (1) The difference
(4.15) bϕ n P
σ̂ := ϕσ̂ − ϕ ∈ (R )
will be called the bias of the estimator σ̂ w.r.t. the map ϕ.
(2) Given an estimator σ̂ ∈ L2ϕ (P, X ) the estimator σ̂ will be called ϕ-
unbiased, if ϕσ̂ = ϕ, equivalently, bϕ
σ̂ = 0.
Using the mean value ϕσ̂ , we define the variance of σ̂ w.r.t. ϕ as the
derivation of ϕ ◦ σ̂ from its mean value ϕσ̂ . We set for all l ∈ (Rn )∗
(4.16) Vξϕ [σ̂](l) := Eξ [(ϕl ◦ σ̂ − ϕlσ̂ ) · (ϕl ◦ σ̂ − ϕlσ̂ )].
The RHS of (4.16) is well-defined, since σ̂ ∈ L2ϕ (P, X ). It is a quadratic
form on (Rn )∗ and will be denoted by Vξϕ [σ̂].
Exercise 4.11. ([JLS2017]) Prove the following formula
(4.17) M SEξϕ [σ̂](l, k) = Vξϕ [σ̂](l, k) + hbϕ ϕ
σ̂ (ξ), li · hbσ̂ (ξ), ki
for all ξ ∈ P and all l, k ∈ (Rn )∗ .
By Proposition 3.3 in [JLS2017], since P is 2-integrable, the function
ϕlσ̂ := hϕσ̂ (ξ), li is differentiable, i.e., there is differential dϕlσ̂ ∈ Tξ∗ P for any
ξ ∈ P such that ∂v ϕlσ̂ (ξ) = dϕlσ̂ (v) for all v ∈ Tξ P . Here for a function f
on P we define ∂v f (ξ) := f˙(c(t)) where c(t) ⊂ P is a curve with c(0) = ξ
and ċ(0) = v. The differentiability of ϕlσ̂ is proved by differentiation under
integral
Z Z
l l
(4.18) ∂v ϕσ̂ = ∂v (ϕ ◦ σ̂ dξ) = (ϕl ◦ σ̂(x) − Eξ (ϕl ◦ σ̂)) · log v dξ,
X X
see [AJLS2017, JLS2017] for more detail.
MACHINE LEARNING 31
Recall that the Fisher metric on any 2-integrable statistical model is non-
degenerate. 9 We now regard kdϕlσ̂ k2g−1 (ξ) as a quadratic form on (Rn )∗ .
Theorem 4.12 (The Cramér-Rao inequality). (cf. [AJLS2017, JLS2017])

Let P be a finite dimensional 2-integrable statistical model with non-degenerate
Fisher metric, ϕ a Rn -valued function on P and σ̂ ∈ L2ϕ (P, X ) a ϕ-estimator.
Then for all l ∈ (Rn )∗ we have
Vξϕ [σ̂](l, l) − kdϕlσ̂ k2g−1 (ξ) ≥ 0.
If σ̂ is unbiased then its MSE is equal to its variance. In this case the
Cramér-Rao inequality asserts that we can never construct an exact estima-
tor.
Outline of the proof of the Cramér-Rao inequality. Since P is 2-integrable
and finite dimensional, the logarithmic representation log(Tξ P ) := {log v| v ∈
Tξ P } is a closed subspace of the Hilbert space L2 (X , ξ). Denote by Πlog(Tξ P )
the orthogonal projection of L2 (X , ξ) to log(Tξ P ) and by ∇g f the gradient
of a function f on P w.r.t. the Fisher metric g. To prove the Cramér-Rao
inequality it suffices to show the following geometric identity
(4.19) Πlog(Tξ P ) (ϕl ◦ σ̂ − Eξ (ϕl ◦ σ̂)) = log(∇g ϕlσ̂ ) ∈ L2 (X , ξ),
since the square of the L2 -norm of the term in the RHS of (4.19) is equal to
||dϕlσ̂ ||2g−1 and the square of the L2 -norm of the term in the LHS is equal to
Vξϕ [σ̂](l, l).
Next, we reduce the proof of equality (4.19) to the proof of the following
equality for all v ∈ Tξ P
(4.20) hϕl ◦ σ̂ − Eξ (ϕl ◦ σ̂), log viL2 = h∇g ϕlσ̂ (ξ), vig
which is obtained easily from (4.18).
Remark 4.13. The Cramér-Rao inequality in Theorem 4.12 is non-parametric,
i.e. P is not assumed to be parameterized by a smooth manifold, as in
Janssen’s version of Cramér-Rao inequality [Janssen2003]. The condition
of 2-integrability of P is an adaptation of the 2-integrability condition of
parametrized measure models in [AJLS2017, JLS2017], which is equivalent
to the differentiability condition in [Janssen2003], see [JLS2017] for com-
ments and history of the Cramér-Rao inequality.
Example 4.14. Assume that ϕ is a differentiable coordinate mapping, i.e.,
there is a differentiable parameterization p from an open subset Θ of Rn
such that ϕ ◦ p = Id. Assume that σ̂ is an unbiased estimator. Then the
terms involving bσ̂ := bϕ l
σ̂ vanishes. Since ϕσ̂ = ϕ we have ||dϕ ||g−1 (ξ) =
9In literature, e.g., [Amari2016, AJLS2017, Borovkov1998], one considers the Fisher
metric on a parametrized statistical model, i.e., the metric obtained by pull-back the
Fisher metric on P via the parameterization map p : Θ → P . This “parameterized”
Fisher metric may be degenerate.
||dϕlσ̂ ||2g−1 (ξ) . Hence the Cramér-Rao inequality in Theorem 4.12 becomes
the well-known Cramér-Rao inequality for an unbiased estimator
(4.21) Vξ := Vξ [σ̂] ≥ g−1 (ξ).
4.4. Efficient estimators and MLE.
Definition 4.15. Assume that P ⊂ P(X ) is a 2-integrable statistical model
and ϕ : P → Rn is a feature map. An estimator σ̂ ∈ L2ϕ (P, X ) is called effi-
cient, if the Cramér-Rao inequality for σ̂ becomes an equality, i.e., Vξϕ [σ̂] =
||dϕσ̂ ||2g−1 (ξ) for any ξ ∈ P .
Theorem 4.16. Let P be a 2-integrable statistical model parameterized by

a differentiable map p : Θ → P(X ), θ 7→ pθ µ0 , where Θ is an open subset of
Rn , and ϕ : P → Rn a differentiable coordinate mapping, i.e., p ◦ ϕ = Id.
Assume that the function p(x, θ) := pθ (x) has continuous partial derivatives
up to order 3. If σ̂ : X → P is an unbiased efficient estimator then σ̂ is a
maximum likelihood estimator (MLE), i.e.,
(4.22) ∂v log p(θ, x)|θ=ϕ◦σ̂(x) = 0
for all x ∈ X and all v ∈ Tθ Θ.
Proof. Assume that σ̂ is efficient. Since ϕ is a coordinate mapping, we obtain
from (4.19)
log(∇g ϕlσ̂ )|ξ=σ̂(x) = ϕl ◦ σ̂ − Eσ̂(x) (ϕl ◦ σ̂ = hl, bϕ
σ̂ (x)i = 0
since σ̂ is unbiased. Comparing the LHS of the above equality with the
LHS of (4.22) for dp(v) = ∇g dϕl = ∇g dϕlσ̂ we obtain immediately Theorem
4.16.
4.5. Consistency of MLE. Assume that P1 = P ⊂ P(X ) is a 2-integrable

statistical model that contains an unknown probability measure µu govern-
ing distribution of random instance xi ∈ X . Then Pn = {µn | µ ∈ P } ⊂
P(X n ) is a 2-integrable statistical model containing probability measure µnu
that governs the distribution of i.i.d. of random instances (x1 , · · · , xn ) ∈ X n .
Denote by gn the Fisher metric on the statistical model Pn . The map
λn : P1 → Pn , µ 7→ µn , is a 1-1 map, it is the restriction of the differentiable
map, also denoted by λn
λn : S(X ) → S(X n ), λn (µ) = µn .
It is easy to see that for any v ∈ Tµ P we have
(4.23) gn (dλn (v), dλn (v)) = n · g1 (v, v).
This implies that the lower bound in the Cramér-Rao inequality (4.21) for
unbiased estimators σ̂n∗ := λ−1 n
n ◦ σ̂n : (X ) → P converges to zero and there
is a hope that MLE is asymptotically accurate as n goes to infinity. Now
we shall give a concept of a consistent sequence ϕ-estimators that formalizes
MACHINE LEARNING 33
the notion of asymptotically accurate sequence of estimators σ̂k∗ : X k → P1

and using it to examine MLE. 10.
Definition 4.17. (cf. [IH1981, p. 30], [Borovkov1998, Definition 1, p. 53])
Let P ⊂ P(X ) be a statistical model and ϕ a Rn -valued function on P .
A sequence of ϕ-estimators σ̂k∗ : (X )k → P → Rn is called a consistent
sequence of ϕ-estimators for the value ϕ(µu ) if for all δ > 0 we have
(4.24) lim µku ({x ∈ X k : |ϕ ◦ σ̂k∗ (x) − ϕ(µu )| > δ}) = 0.
k→∞
Under quite general conditions on the density functions of a statistical

model P ⊂ P(X ) of dominated measures, see e.g. [IH1981, Theorem 4.3, p.
36] the sequence of MLE’s is consistent.
4.6. Conclusion. In this lecture we derive a natural geometry on a statis-

tical model P ⊂ P(X ) regarding it as a subset in the Banach space S(X )
with the total variation norm. Since P is non-linear, we linearize estimator
σ̂k∗ : X k → Pk = λk (P ) by composing it with a map ϕk := λ−1
k ◦ϕ : Pk → R .
n
Then we define the MSE and variance of ϕk -estimator ϕk ◦ σ̂k , which can be
estimated using the Cramér-Rao inequality. It turns out that the efficient
unbiased estimator w.r.t. MSE is MLE. The notion of a ϕ-estimator allows
to define the notion of a consistent sequence of ϕ-estimators that formalizes
the notion of asymptotically accurate estimators.
5. Consistency of a learning algorithm

In the last lecture we learned the important concept of a consistent se-
quence of ϕ-estimators, which formalizes the notion of the asymptotic accu-
racy of a sequence of estimators, and applied this concept to MLE.
In this lecture we extend the concept of a consistent sequence of estimators
to the notion of consistency of a learning algorithm in a unified learning
model that encompasses models for density estimation and discriminative
models of supervised learning. The concept of consistency of a learning
algorithm formalizes the notion of learnability of a learning machine. In
particular we relate this notion with the problem of overfitting. 11 Then we
shall apply this concept to examine the success of ERM algorithm in binary
classification problems.
5.1. Consistent learning algorithm and its sample complexity.

In a unified learning model (Z, H, L, P ) we are given a measurable space
Z (e.g. Z = X × Y in classification and regression problems), a hypothesis
class H (e.g., the domain Θ in the density estimation problem in Subsection
10the notion of a consistent sequence of estimators that is asymptotically accurate has

been suggested by Fisher in [Fisher1925]
11In ML community, one says that a learning algorithm is consistent if it may commit
no error on the example of the training data [MRT2012, p.5].
3.1) together with an instantaneous loss function L : Z × H → R+ and a

statistical model P ⊂ P(Z). We define the expected risk function as follows
RL : H × P → R+ , (h, µ) 7→ Eµ L(z, h).
A learning algorithm is a map
[
A: Z n → H, S 7→ hS
n
where S is distributed by some unknown µn ∈ Pn = λn (P ), cf. (2.1).
Example 5.1. In a unified learning model (Z, H, L, P ) there is always a
learning algorithm using ERM, if H is compact. For an element Sn =
(z1 , · · · , zn ) ∈ Z n we denote by [Sn ] the subset {z1 , · · · , zn } ⊂ Z. For
n L L
Pn ∈ Z we define the empirical risk R̂Sn : H → R by the formula: R̂Sn (h) =
S
z∈[Sn ] L(z, h). The ERM algorithm Aerm for (Z, H, L, P ) is defined as
follows
(5.1) Aerm (Sn ) := argh∈H min R̂SLn (h).
Observe that argh∈H min R̂SLn (h) may not exist if H is not compact. In
this case we denote by Aerm (Sn ) any hypothesis that satisfies the inequality
R̂SL (Aerm (Sn )) − inf h∈H R̂SLn (h) = O(n−k ), where k ≥ 1 depends on the
computational complexity of defining Aerm . In other words, Aerm is only
asymptotically ERM. 12
Note that the ERM algorithm Aerm does not depend on P but the knowl-
edge of P is important for defining and understanding the expected loss
function RµL . In general we expect a close relationship between H, a repre-
sentation of a discriminative model, and P , a representation of a generative
model.
L
Recall that Rµ,H = inf h∈H RµL (h), see (2.10).
Definition 5.2. A learning algorithm A in a model (Z, H, L, P ) is called
consistent, if for any ε ∈ (0, 1) and for every probability measure µ ∈ P ,
(5.2) lim µn {S ∈ Z n : |RµL (A(S)) − Rµ,H
L
| ≥ ε} = 0
n→∞
(2) A learning algorithm A is called uniformly consistent, if (5.2) converges
to zero uniformly on P , i.e., for any (ε, δ) ∈ (0, 1)2 there exists a number
mA (ε, δ) such that for any µ ∈ P and any m ≥ mA (ε, δ) we have
(5.3) µm {S ∈ Z m : |RµL (A(S)) − Rµ,H
L
| ≤ ε} ≥ 1 − δ.
If(5.3) holds we say that A predicts with accuracy ε and confidence 1 − δ
using m samples.
We characterize the uniform consistency of a learning algorithm A via the
notion of the sample complexity function of A.
12Vapnik considered ERM algorithm also for the case that R̂L may not reach infimum,
Sn
using slightly different language than ours. In [SSBD2014] the authors assumed that a
minimizer of R̂SLn always exists.
MACHINE LEARNING 35
Definition 5.3. Let A be an algorithm on (Z, H, L, P ) and mA (ε, δ) the

minimal number m0 ∈ R+ ∪ ∞ such that (5.3) holds for any m ≥ m0 . Then
the function mA : (ε, δ) 7→ mA (ε, δ) is called the sample complexity function
of algorithm A.
Clearly a learning algorithm A is uniformly consistent if and only if mA
takes value in R+ . Furthermore, A is consistent if and only if the sample
function of A on the sub-model (Z, H, L, µ) takes values in R+ for all µ ∈ P .
Example 5.4. Let (Z, H, L, P ) be a unified learning model. Denote by
πn : Z ∞ → Z n the map (z1 , · · · , z∞ ) 7→ (z1 , · · · , zn ). A sequence {S∞ ∈
Z ∞ } of i.i. instances distributed by µ ∈ P is called overfitting, if there exist
ε ∈ (0, 1) such that for all n we have
(5.4) |RµL (Aerm (πn (S∞ )) − Rµ,H
L
| ≥ ε.
Thus Aerm is consistent, if and only if the set of all overfitting sequences
S∞ ∈ Z ∞ has µ∞ -zero measure 13, equivalently, if (5.2) holds, for any µ ∈ P .
In Example 2.11 we showed the existence of a measure µf on Z = X × Y
such that any sequence S∞ ∈ Z ∞ distributed by µ∞ f is overfitting. Hence
the unified learning model in Example 2.11) for any P containing µf is not
consistent using Aerm .
The following simple Lemma reduces a proof of the uniform consistency
of Aerm to the proof of the convergence in probability of RSL (h) to RµL (h)
(the weak law of large numbers) that is uniform on H.
Lemma 5.5. Assume that for any (ε, δ) ∈ (0, 1)2 there exists a function
mH (ε, δ) taking value in R+ such that for all m ≥ mH (ε, δ) for all µ ∈ P
and for all h ∈ H we have
(5.5) µm {S ∈ Z m : |R̂SL (h) − RµL (h)| < ε} ≥ 1 − δ
then Aerm is uniformly consistent.
Proof. For the simplicity of the exposition we assume first that Aerm (Sn ) =
arg minh∈H RSLn (h). The argument for the general case can be easily adapted
from this simple case.
Given m ≥ mH (ε/2, δ/2), µ ∈ P and hε ∈ H such that RµL (hε ) ≤ Rµ,H
L +ε
we have
µm {S ∈ Z m : |RµL (Aerm (S)) − Rµ,H | ≤ 2ε} ≥
µm {S ∈ Z m : RµL (Aerm (S)) ≤ RµL (hε ) + ε} ≥
ε ε
µm {S ∈ Z m : RµL (Aerm (S)) ≤ R̂SL (hε ) + & |RSL (hε ) − RµL (hε )| < } ≥
2 2
δ
µn {S ∈ Z m : RµL (Aerm (S)) ≤ R̂SL (hε ) + ε} − ≥
2
n n L L ε δ
µ {S ∈ Z : |Rµ (Aerm (S)) − R̂S (A(S))| ≤ } − ≥ 1 − δ
2 2
13we refer to [AJLS2017, p. 293] and [Bogachev2007, p. 188] for definition of µ∞
since R̂SL (A(S)) < R̂SL (hε ). This completes the proof of Lemma 5.5.
Theorem 5.6. (cf. [SSBD2014, Corollary 4.6, p.57]) Let (Z, H, L, P(Z))
be a unified learning model. If H is finite and L(Z × H) ⊂ [0, c] 63 ∞ then
the ERM algorithm is uniformly consistent.
Proof. By Lemma 5.5, it suffices to find for each (ε, δ) ∈ (0, 1) × (0, 1) a
number mH (ε, δ) such that for all m ≥ mH (ε, δ) and for all µ ∈ P(Z) we
have
\
µm {S ∈ Z m : |R̂SL (h) − RµL (h)| ≤ ε} ≥ 1 − δ.

(5.6)
h∈H
In order to prove (5.6) it suffices to establish the following inequality
X
µm {S ∈ Z m | |R̂SL (h) − RµL (h)| > ε} < δ.

(5.7)
h∈H
Since #H < ∞, it suffices to find mH (ε, δ) such that when m ≥ mH (ε, δ)
each summand in RHS of (5.7) is small enough. For this purpose we shall
apply the well-known Hoeffding inequality, which specifies the rate of con-
vergence in the week law of large numbers, see Subsection 11.2.
To apply Hoeffding’s inequality to the proof of Theorem 5.6 we observe
that for each h ∈ H
{θih (z) := L(h, z) ∈ [0, c]}
are i.i.d. R-valued random variables on Z. Furthermore we have for any
h ∈ H and S = (z1 , · · · , zm )
m
L 1 X h
R̂S (h) = θi (zi ),
m
i=1
RµL (h)
= θ̄h .
Hence the Hoeffding inequality implies
(5.8) µm {S ∈ Z m : |R̂SL (h) − RµL (h)| > ε} ≤ 2 exp(−2mε2 c−2 ).
Now pluging
log(2#(H)/δ)
m ≥ mH (ε, δ) :=
2ε2 c−2
in (5.8) we obtain (5.7). This completes the proof of Theorem 5.6.

Definition 5.7. The function mH : (0, 1)2 → R defined by the requirement
that mH (ε, δ) is the least number for which (5.6) holds is called the sample
complexity of a (unified) learning model (Z, H, L, P ).
Remark 5.8. (1) We have proved that the sample complexity of the algo-
rithm Aerm is upper bounded by the sample complexity mH of (Z, H, L, P ).
(2) The definition of the sample complexity mH in our lecture is different
from the definition of the sample complexity mH in [?, Definition 3.1, p. 43],
which is equivalent to the notion of the sample complexity mA of a learning
algorithm in our lecture.
MACHINE LEARNING 37
5.2. Uniformly consistent learning and VC-dimension. In this sub-

section we shall examine sufficient and necessary conditions for the existence
of a uniformly consistent learning algorithm on a unified learning model with
infinite hypothesis class H ⊂ Y X . First we prove a version of No-Free-Lunch
theorem which asserts that there is no uniformly consistent learning algo-
rithm on a learning model with a very large hypothesis class and a very large
statistical model. Denote by PX (X × Y) the set of all probability measures
(Γf )∗ (µX ) ∈ P(X × Y) where f ∈ Y X and µX ∈ P(X ).
Theorem 5.9. Let X be an infinite domain set, Y = {0, 1} and L(0−1) the
0-1 loss function. Then there is no uniformly consistent learning algorithm
on a unified learning model (X × Y, H, L(0−1) , PX (X × Y)).
Proof. To prove Theorem 5.9 it suffices to show that mA (ε, δ) = ∞ for

(ε, δ) = (1/8, 1/8) and any learning algorithm A. Assume the opposite, i.e.,
mA (1/8, 1/8) = m < ∞. Then we shall find µ(m) ∈ PX (X × Y) such that
Equation (5.3) violates for m, µ(m) and (ε, δ) = (1/8, 1/8).
To describe the measure µ(m) we need some notations. For a subset
C[k]
C[k] ⊂ X of k-elements let µX ∈ P(X ) be defined by
C[k] #(B ∩ C[k])

(5.9) µX (B) := for any B ⊂ X .
k
For a map f : X → Y we set
C[k] C[k]
µf := (Γf )∗ µX ∈ PX (X × Y).
Lemma 5.10. Assume that X , Y are finite sets and #X ≥ n + 1. For

f ∈ Y X set µf := µX f ∈ PX (X × Y). Then for any learning algorithm
A : S 7→ AS , any f ∈ Y X we have
Z Z
X 1 n
(5.10) Rµ(0−1) (AS ) d(µnf )(S)dµY
Y X (f ) ≥ (1 − )(1 − )
Y X (X ×Y)n
f
#Y #X
Proof of Lemma 5.10. We set for S ∈ (X × Y)n
P ri : (X × Y)n → X , (x1 , y1 ), · · · , (xn , yn ) 7→ xi ∈ X ,
n
[
XS := P ri (S).
i+1
Note that S is distributed by µnf means that S = {(x1 , f (x1 )), · · · , (xn , f (xn )),
so S is essentially distributed by the uniform probability measure (µX n
X ) . Let
us compute and estimate the double integral in the LHS of (5.10) using (2.9)
and the Fubini theorem.

1 X AS (x)

EµY X Eµnf Rµ(0−1) (A S ) = E Y X E(µX )n (1 − δ f (x) )
YX
f
#X µY X X
x∈X
1 X AS (x)

≥ EµY X E(µX )n (1 − δf (x) )
#X Y X X
x6∈XS
1 X
AS (x)

= E(µX )n EµY X (1 − δf (x) )
#X X
YX
x6∈XS
1 1
= E(µX )n #[X \ XS ] · 1 −
#X X #Y
since #[X \XS ]≥#X −n 1 n
(5.11) ≥ (1 − )(1 − ).
#Y #X
This completes the proof of Lemma 5.10.
Continuation of the proof of Theorem 5.9. It follows from Lemma 5.10
C[2m]
that there exists f ∈ Y X such that, denoting µ := µf , we have
Z Z
1
(5.12) Rµ(0−1) (AS ) dµm = Rµ(0−1) (AS ) d(µm )(S) ≥ .
(X ×Y)m (C[2m]×Y)m 4
(0−1)
Since 0 ≤ Rµ ≤ 1 we obtain from (5.12)
1 1
µm {S ∈ (X × Y)m | Rµ(0−1) A(S) ≥ } > .
8 8
This implies that (5.3) does not hold for (ε, δ) = (1/8, 1/8), for any m and
C[m]
µ(m) = µf . This proves Theorem 5.9.
Remark 5.11. In the proof of Theorem 5.9 we showed that if there is a
subset C ⊂ X of size 2m and the restriction of H to C is the full set of
functions in {0, 1}C then (5.3) does not hold for any learning algorithm A,
m ∈ N and (ε, δ) = (1/8, 1/8). In other words we cannot predict a hypothesis
in H = Y X of a learning model (X , H, L(0−1) , PX (X × Y)) with accuracy ε
and confidence 1 − δ using sample of size m if #X ≥ 2m. This motivates
the following Definition.
Definition 5.12. (1) A hypothesis class H ⊂ {0, 1}X shatters a finite subset
C ⊂ X if #H|C = 2#C .
(2)The VC-dimension of a hypothesis class H ⊂ {0, 1}X , denoted by
V C dim(H), is the maximal size of a set C ⊂ X that can be shattered by
H. If H can shatter sets of arbitrarily large size we say that H has infinite
VC-dimension.
Example 5.13. Let H be the class of intervals in the real line, namely,
H = {1(a,b) : a < b ∈ R},
where 1(a,b) : R → {0, 1} is the indicate function of the interval (a, b). Take
the set C = {1, 2}. Then, H shatters C, since all all the functions in the set
MACHINE LEARNING 39
{1, 2}(0,1) can be obtained as the restriction of some function from H to C.

Hence V C dim(H) ≥ 2. Now take an arbitrary set C = {c1 < c2 < c3 } and
the corresponding labeling (1, 0, 1). Clearly this labeling cannot be obtained
by an interval: Any interval h(a,b) that contains c1 and c3 (and hence labels
c1 and c3 with the value 1) must contain c2 (and hence it labels c2 with 0 ).
Hence H does not shatter C. We therefore conclude that V C dim(H) = 2.
Note that H has infinitely many elements.
Exercise 5.14 (VC-Threshold functions). Consider the hypothesis class
F ⊂ {−1, 1}R of all threshold functions sgnb : R → R, where b ∈ R, defined
by
sgnb (x) := sgn(x − b)
Show that V C dim(F) = 1.
In Remark 5.11 we observed that the finiteness of V C dim H is a necessary
condition for the existence of a uniformly consistent learning algorithm on
a unified learning model (X , H, L(0−1) , PX (X × Y)). In the next section we
shall show that the finiteness of V C dim H is also a sufficient condition for
the uniform consistency of Aerm on (X , H, L0−1 , PX (X × Y)).
5.3. Fundamental theorem of binary classification.
Theorem 5.15 (Fundamental theorem of binary classification). A learning
model (X , H ⊂ {0, 1}X , L(0−1) , P(X × {0, 1})) has a uniformly consistent
learning algorithm, if and only if V C dim(H) < ∞.
Outline of the proof. Note that the “only if” assertion of Theorem 5.15
follows from Remark 5.11. Thus we need only to prove the “if” assertion. By
Lemma 5.5 it suffices to show that if V C dim(H) = k < ∞ then mH (ε, δ) <
∞ for all (ε, δ) ∈ (0, 1)2 . In other words we need to find a lower bound for
the LHS of (5.5) in terms of the VC-dimension, which is an upper bound of
the RHS of (5.5), when ε ∈ (0, 1) and m is sufficiently large. This shall be
done in three steps.
In step 1, setting Z = X ×Y, and omitting superscript of the risk function

R, we use the Markov inequality to obtain
Eµm |Rµ (h) − RS (h)|
(5.13) µm {S ∈ Z m : |Rµ (h) − R̂S (h)| < a} ≤
a
for any a > 0 and any h ∈ H.
In step 2 we define the growth function ΓH (m) : N → N and use it to

upper bound the RHS of (5.13). Namely we shall prove the following
q
4 + log ΓH (2m)
(5.14) Eµm (sup |Rµ (h) − R̂S (h)| ≤ √ )≥1−δ
h∈H δ 2m
for every µ ∈ P(X × Y) and any δ ∈ (0, 1). The proof of (5.14) is delicate
and can be found in [SSBD2014, p. 76-77].
Definition 5.16 (Growth function). Let F ⊂ Y X be a class of functions

with finite target space Y. The growth function ΓF assigned to F is then
defined for all n ∈ N as
ΓF (n) := max #F|Σ .
Σ⊂X | #Σ=n
We also set Γ(0) = 1.
Example 5.17. Consider the set F : {signb | b ∈ R} of all threshold func-

tions. Given a set of distinct points {x1 , · · · , xn } = Σ ⊂ R, there are n + 1
functions in F|Σ corresponding to n + 1 possible ways of placing b relative
to the xi s. Hence, in this case ΓF (n) ≥ n + 1.
Exercise 5.18. Show that ΓF (n) = n + 1 for the set F of all threshold
functions.
In step 3 we use the following Vapnik-Chervonenski-Lemma, also known

as Sauer’s Lemma, whose proof can be found in [SSBD2014, p. 74-75].
Lemma 5.19. Let H ⊂ {0, 1}X be a hypothesis class with V C dim(H) =

d < ∞. Then, for all n ∈ N we have
d
X n
ΓH (n) ≤ .
i
i=0
In particular, if n > d + 1 then ΓH (n) ≤ (en/d)d .
It follows from Lemma 5.19 that for any (ε, δ) ∈ (0, 1)2 there exists m
such that
q
4 + log ΓH (2m)
√ <ε
δ 2m
and therefore by (5.14) for any (ε, δ) ∈ (0, 1)2 the value mH (ε, δ) is finite.
This completes the proof of Theorem 5.15.
5.4. Conclusions. In this lecture we define the notion of the (uniform) con-
sistency of a learning algorithm A on a unified learning model (Z, H, L, P )
and characterize this notion via the sample complexity of A. We relate
the consistency of the ERM algorithm with the uniform convergence of the
law of large numbers over the parameter space H and use it to prove the
uniform consistency of ERM in the binary classification problem (X , H ⊂
{0, 1}X , L(0−1) , P(X × {0, 1}). We show that the finiteness of the VC-
dimension of H is a necessary and sufficient condition for the existence of
a uniform consistent learning algorithm on a binary classification problem
(X × {0, 1}, H ⊂ {0, 1}X , L(0−1) , PX (X × {0, 1}).
MACHINE LEARNING 41
6. Generalization ability of a learning machine and model

selection
In the last lecture we measured the consistency of a learning algorithm A
in a unified learning model (Z, H, L, P ) via the sample complexity function
mA : (0, 1)2 → R+ ∪ ∞. The sample complexity mA (ε, δ) is the less number
of samples that A requires in order to make a prediction with ε accuracy
and (1 − δ) confidence. In 1984 Valiant suggested a PAC-model of learning,
which corresponds to the notion of uniform (w.r.t. P ) consistency of A,
which has moreover to be efficiently computable, i.e. the function mA (ε, δ)
must be polynomial in ε−1 and δ −1 . Furthermore Valiant also requires
that A is efficiently computable, which can be expressed in terms of the
computational complexity of A, see [SSBD2014, Chapter 8] for discussion
on running time of A.
Thus, given a learning machine (Z, H, L, P, A), where A is a learning
algorithm, the generalization ability of (Z, H, L, P, A) is measured in terms
of the family of sample complexities {mA,µ (ε, δ)| µ ∈ P } of the learning
machines (Z, H, L, µ ∈ P, A) and the computational complexity of A. In
the previous lecture, Lemma 5.5, we gave upper bounds for the sample
complexity mAerm (ε, δ) in terms of the sample complexity mH (ε/2, δ/2) of
the learning model. Then we showed that in a binary classification problem
the sample complexity mH takes finite values if and only if the VC-dimension
of H is finite.
Today we shall discuss two further methods of upper bounds for the sam-
ple complexities mH and mAerm of some important learning models. Then
we discuss the problem of learning model selection.
6.1. Covering number and sample complexity. In the binary classifi-
cation problem of supervised learning the VC-dimension is a combinatorial
characterization of the hypothesis class H, which carries no topology, since
the domain X and the target space Y are discrete. The expected zero-
one loss function is therefore the preferred choice of a risk function. In
[CS2001] Cucker-Smale estimated the sample complexity mH of a discrim-
inative model for a regression problem with the MSE as the expected loss
function.
Before stating Cucker-Smale’s results we introduce necessary notations.
• X is a topological space and ρ denotes a Borel measure on X × Rn .
• Let Cn (X ) be the Banach space of continuous bounded Rn -valued func-
tions on X with the norm 14
||f ||C 0 = sup ||f (x)||.
x∈X
• For f ∈ Cn (X ) let fY ∈ Cn (X × Rn ) denote the function
fY (x, y) := f (x) − y.
14In [CS2001, p.8] the authors used the L -norm, but they considered only the sub-
∞
space of continuous functions
Then M SEρ (f ) = Eρ (||fY ||2 ), see (2.2) and (2.3).

• For a function g ∈ Cn (X × Rn ) denote by Vρ (g) its variance, see (4.16),
Vρ (g) = Eρ (||g − Eρ (g)||2 ) = Eρ (||g||2 ) − ||Eρ g||2 .
• For a compact hypothesis class H ⊂ Cn (X ) define the following quan-
tities
(6.1) M SEρ,H (f ) := M SEρ (f ) − min M SEρ (f ),
f ∈H
which is called the estimation error of f , or the sample error of f [CS2001],

and
Vρ (H) := sup Vρ (fY ),
f ∈H
N (H, s) := min{l ∈ N| there exists l disks in H with radius s covering H}.

• For S = (z1 , · · · , zm ) ∈ Z m := (X × Y)m , where zi = (xi , yi ), denote by
fS the minimizer of the function M SES : H → R such that
m
1 X
M SES (f ) := ||f (xi ) − yi ||2 .
m
i=1
The existence of fS follows from the compactness of H and the continuity
of the functional M SES on H.
Theorem 6.1. ([CS2001, Theorem C]) Let H be a compact subset of C(X ) :=
C1 (X ). Assume that for all f ∈ H we have |f (x) − y| ≤ M ρ-almost every-
where, where ρ is a probability measure on X × R. Then for all ε > 0
mε 2
ε − 1 2
(6.2) ρm {S ∈ Z m | M SEρ,H (fS ) ≤ ε} ≥ 1 − N (H, )2e 8(4Vρ (H)+ 3 M ε) .
16M
Theorem 6.1 implies that for any n < ∞ the ERM algorithm is consistent
on the unified learning model (X , H ⊂ Cn (X ), L2 , PB (X × Rn )), where L2
is the quadratic loss function, and PB (X × Rn ) denotes the space of Borel
measures on the topological space X × Rn , if H is compact. To increase
the accuracy of the estimate in (6.2) we need to decrease the radius of the
covering balls and therefore increase the number of the covering ball.
The proof of Theorem 6.1 is based on Lemma 5.5. To prove that the
sample complexity of Aerm is bounded it suffices to prove that the sample
complexity mH of the learning model (X , H ⊂ C(X ), L2 , PB (X × Rn )) is
bounded.
Outline of the proof of the Cucker-Smale theorem. The strategy of the

proof of Theorem 6.1 is similar to that of Fundamental Theorem of binary
classification (Theorem 5.15).
In the first step we prove the following Lemma, which gives a lower bound
on the rate of the convergence in probability of empirical risk M SES (f ) to
the expected risk M SEρ (f ) for a given f ∈ H.
MACHINE LEARNING 43
Lemma 6.2. ([CS2001, Theorem A, p.8]) Let M > 0 and f ∈ C0 (X ) such

that |f (x) − y| ≤ M ρ-a.e.. Then for all ε > 0 we have
mε2
(− 1 M 2 ε) )
m m 2(σ 2 + 3
ρ {S ∈ Z : |M SEρ (f ) − M SES (f )| ≤ ε} ≥ 1 − 2e
where σ2 = Vρ (fY2 ).
Lemma 6.2 is a version of the inequality (5.8), for which we used the
Hoeffding inequality. The Hoeffding inequality does not involve the variance
and Cucker-Smale used the Bernstein inequality instead of the Hoeffding
inequality, see Appendix 11.
In the second step, we reduce the problem of estimating upper bound for
the sample complexity mH to the problem of estimating upper bound for
the sample complexities mDj , where {Dj | j ∈ [1, l]} is a cover of H, and
using the covering number. Namely we have the following easy inequality
ρm {S ∈ Z m | sup |M SEρ (f ) − M SES (f )| ≥ ε} ≤
f ∈H
l
X
(6.3) ρm {S ∈ Z m | sup |M SEρ (f ) − M SES (f )| ≥ ε}.
j=1 f ∈Dj
In the last third step we proof the following

Lemma 6.3. ([CS2001, Proposition 3, p. 12]) Let f1 , f2 ∈ C(X ). If |fj (x)−
y| ≤ M on a set U ⊂ Z of full measure for j = 1, 2 then for any S ∈ U m
we have
|M SES (f1 ) − M SES (f2 )|| ≤ 4M ||f1 − f2 ||C0 .
Lemma 6.3 implies that for all S ∈ U m
sup |M SEρ (f ) − M SES (f )| ≥ 2ε =⇒ |M SEρ (fj ) − M SES (fj )| ≥ ε.
f ∈Dj
Combining the last relation with (6.3), we derive the following desired upper
estimate for the sample complexity mH .
Proposition 6.4. Assume that for all f ∈ H we have |f (x)−y| ≤ M ρ-a.e..
Then for all ε > 0 we have
mε 2
m m ε (− 2 1 2
ρ {z ∈ Z : sup |M SEρ (f )−M SEz (f )| ≤ ε} ≥ 1−N (H, )2e 4(2σ + 3 M ε)
f ∈H 8M
where σ 2 = supf ∈H Vρ (fY2 ).
This completes the proof of Theorem 6.1.
Exercise 6.5. Let L2 denote the instantaneous quadratic loss function in
(2.12). Derive from Theorem 6.1 an upper bound for the sample complexity
mH (ε, δ) of the learning model (X , H ⊂ Cn (X ), L2 , PB (X ×Rn )), where H is
compact and PB (X × Rn ) is the space of Borel measures on the topological
space X × Rn .
Remark 6.6. If the hypothesis class H in Theorem 6.1 is a convex subset in

H then Cucker-Smale got an improved estimation of the sample complexity
mAerm [CS2001, Theorem C∗ ].
6.2. Rademacher complexities and sample complexity. Rademacher
complexities are more sophisticated complexities that can be used in upper
bounds for a “half” of the sample complexity mH of a learning model but
they are sufficient for estimating upper bounds of the sample complexity
mAerm of the ERM algorithm.
The Rademacher complexity of a learning model (Z, H, L, µ) is defined as
the Rademacher complexity of the family GH L of functions
(6.4) {gh : Z → R, gh (z) = L(z, h)| h ∈ H}.

Definition 6.7 (Rademacher complexity). The empirical Rademacher com-
plexity of G w.r.t. a sample S = (z1 , · · · , zn ) ∈ Z n is defined as follows
n
1X
R̂S (G) := E(µZ2 )n [sup σi g(zi )],
Z2 g∈G n i=1
where {σi ∈ Z2 | i ∈ [1, n]} and µZZ22 is the counting measure on Z2 , see (5.9).
If S is distributed according to a probability measure µn on Z n , then the
Rademacher complexity of G w.r.t. µ are given by
Rn,µ (G) := Eµn [RS (G)].
The Rademacher n-complexity Rn (Z, H, L, µ) (resp. the Rademacher em-
pirical n-complexity RS (Z, H, L)) is defined to be the complexity Rn,µ (GHL)
L L
(resp. the empirical complexity R̂S (GH )), where GH is the family associated
to the model (Z, H, L, µ) by (6.4).
Example 6.8. Let us consider a learning model (X ×Z2 , H ⊂ (Z2 )X , L(0−1) , µ).
For a sample S = {(x1 , y1 ), · · · , (xm , ym )} ∈ (X × Z2 )m denote by P r(S)
the sample (x1 , · · · , xm ) ∈ X m . We shall show that
(0−1) 1
(6.5) R̂S (GH ) = R̂P r(S) (H).
2
Using the identity
1
L(0−1) (x, y, h) = 1 − δyh(x) = (1 − yi h(xi ))
2
we compute
m
(0−1) 1 X
R̂S (GH ) = E(µZ2 )m [sup σi δyh(x
i
i)
]
Z2 h∈H m
i=1
m
1 X 1 − yi h(xi )
= E(µZ2 )m [sup σi ]
Z2 h∈H m 2
i=1
MACHINE LEARNING 45
E Z σi =0 m
(µ 2 )m 1 1 X
Z2
= E(µZ2 )m [sup −σi yi h(xi )]
2 Z2 h∈H m i=1
m
1 1 X 1
= E(µZ2 )m [sup σi h(xi )] = R̂P r(S) (H)
2 Z2 h∈H m 2
i=1
which is required to prove.
We have the following relation between the empirical Rademacher com-
plexity and the Rademacher complexity, using the McDiarmid concentration
inequality, see (11.4) and [MRT2012, (3.14), p.36]
r
n n L L ln(2/δ)
(6.6) µ {S ∈ Z |Rn,µ (GH ) ≤ RS (GH ) + } ≥ 1 − δ/2.
2m
Theorem 6.9. (see e.g. [SSBD2014, Theorems 26.3, 26.5, p. 377- 378])
Assume that (Z, H, L, µ) is a learning model with |L(z, h)| < c for all z ∈ Z
and all h ∈ H. Then for any δ > 0 and any h ∈ H we have
r
n n L L L 2 ln(2/δ)
(6.7) µ {S ∈ Z | Rµ (h) − RS (h) ≤ Rn,µ (GH ) + c } ≥ 1 − δ,
n
r
n n 2 ln(4/δ)
(6.8) µ {S ∈ Z | RµL (h) − RSL (h) ≤ L
RS (GH ) + 4c } ≥ 1 − δ.
n
(6.9) r
n n 2 ln(8/δ)
µ {S ∈ Z |RµL (Aerm (S)) − RµL (h) ≤ L
2RrS (GH ) + 5c } ≥ 1 − δ.
δ
(6.10) Eµn (RµL (Aerm (S)) − Rµ,H

L L
) ≤ 2Rrn,µ (GH ).
It follows from (6.10), using the Markov inequality, the following bound
for the sample complexity mAerm in terms of Rademacher complexity
2RL L
n,µ (GH )
(6.11) µn {S ∈ Z n | RµL (Aerm ) − Rµ,H
L
≤ } ≥ 1 − δ.
δ
Remark 6.10. (1) The first two assertions of Theorem 6.9 give an upper
bound of a “half” of the sample complexity mH of a unified learning model
(Z, H, L, µ) by the (empirical) Rademacher complexity Rn (G) of the associ-
ated family G. The last assertion of Theorem 6.9 is derived from the second
assertion and the Hoeffding inequality.
(2) For the binary classification problem (X ×{0, 1}, H ⊂ {0, 1}X , L(0−1) , P(X ×
{0, 1}) there exists a close relationship between the Rademacher complexity
and the growth function ΓH (m), see [MRT2012, Lemma 3.1, Theorem 3.2,
p. 37] for detailed discussion.
6.3. Model selection. The choice of a right prior information in machine

learning is often interpreted as the choice of a right class H in a learning
model (Z, H, L, P ) which is also called a model selection. A right choice of
H should make balance between the approximation error and the estimation
error of H defined in the error decomposition of H.
6.3.1. Error decomposition. We assume that the maximum domain of the

expected loss function RµL is a subspace HL,µ ⊃ H, given L and a probability
measure µ ∈ P .
We define the Bayes risk of the learning problem RµL on the maximal
domain HL,µ as follows
L
Rb,µ := inf RµL (h)
h∈HL,µ
L
Recall that Rµ,H := inf h∈H RµL (h) quantify the optimal performance of a
learner in H. Then we decompose the difference between the expected risk
of a predictor h ∈ H and the Bayes risk as follows:
(6.12) RµL (h) − Rb,µ
L
= (RµL (h) − Rµ,H
L L
) + (Rµ,H L
− Rb,µ ).
The first term in the RHS of (6.12) is called the estimation error of h, cf.
(6.1), and the second term is called the approximation error. If h = Aerm (S)
is a minimizer of the empirical risk R̂SL , then the estimation error of Aerm (S)
is also called the sample error [CS2001, p. 9].
The approximation error quantifies how well the hypothesis class H is
suited for the problem under consideration. The estimation error measures
how well the hypothesis h performs relative to best hypotheses in H. Typi-
cally, the approximation error will decrease when enlarging H but the sample
error will increase as demonstrated in No-Free-Lunch Theorem 5.9, because
P should be enlarged as H will be enlarged.
Example 6.11. (cf. [Vapnik2000, p. 19], [CS2001, p. 4, 5]) Let us
compute the error decomposition of a discriminative model (X × R, H ⊂
RX , L2 , PB (X × R) for regression problem. Let π : X × R → X denote
the natural projection. Then the measure ρ, the push-forward measure
π∗ (ρ) ∈ P(X ) and the conditional probability measure ρ(y|x) on each fiber
π −1 (x) = R are related as follows
Z Z Z
ϕ(x, y)dρ = ( ϕ(x, y)dρ(y|x))dπ∗ (ρ)
X ×R X R
for ϕ ∈ L1 (ρ),
similar to the Fubini theorem, see Subsection 10.2.5.
Let us compute the Bayes risk Rb,µ L for L = L . The maximal subspace
2
HL,µ where the expected loss RµL is well defined is the space L2 (X , π∗ (ρ)).
We claim that the regression function of ρ, see Exercise 2.7,
Z
rρ (x) := Eρ (i2 (Y )|X = x) = ydρ(y|x)
R
MACHINE LEARNING 47
minimizes the M SEπ∗ (ρ) defined on the space L2 (X , π∗ (ρ)). Indeed, for any
f ∈ L2 (X , π∗ (ρ)) we have
Z
(6.13) M SEπ∗ (ρ) (f ) = (f (x) − rρ (x))2 dπ∗ (ρ) + M SEπ∗ (ρ) (rρ ).
X
The equation (6.13) implies that the Bayes risk Rb,π L is M SEπ∗ (ρ) (rρ ). It
∗ (ρ)
follows that the approximation error of a hypothesis class H is equal
Z
M SEπ∗ (ρ) (fmin ) = (fmin − rρ (x))2 dπ∗ (ρ) + M SEπ∗ (ρ) (rρ ),
X
where fmin is a minimizer of MSE in H. Since M SEπ∗ (ρ) (rρ ) is a constant,

if H is compact, fmin exists and satisfies the condition
(6.14) fmin = arg min d(g, rρ ),

g∈H
where d is the L2 -distance on L2 (X , π∗ (ρ)).
6.3.2. Validation and cross-validation. An important empirical approach in

model selection is validation and its refinement - (k-fold) cross-validation.
Validation is used for model selection as follows. We first train different
algorithms (or the same algorithm with different parameters) on the given
training set S. Let H := {h1 , · · · , hr } be the set of all output predictors of
the different algorithms. Now, to choose a single predictor from H we sample
a fresh validation set S 0 ∈ Z m and choose the predictor hi that minimizes
the error over the validation set.
The basic idea of cross-validation is to partition the training set S =
(z1 , · · · , zn ) ∈ Z n into two sets S = S1 ∪ S2 where S1 ∈ Z k and S2 ∈ Z n−k .
The set S1 is used for training each of the candidate models, and the second
set S2 is used for deciding which of them yields the best results.
The n-cross validation is a refinement of cross-validation by partition of
the training set into n-subsets and use one of them for testing the and repeat
the procedure (n − 1)-time for other testing subsets.
6.4. Conclusion. In this lecture we use several complexities of a learning

model (Z, H, L, P ) for obtaining upper bounds of the sample complexity of
the ERM algorithm Aerm . Among them Rademacher complexities are the
most sophisticated ones that measure the capacity of a hypothesis class on a
specific sample, which can be used to bound the difference between empirical
and expected error, and thus the excess generalization error of empirical risk
minimization. To find an ideal hypothesis class H for a learning problem
we have to take into account the error decomposition of a learning model
and the resulting bias-variance trade-off and use empirical cross validation
methods.
7. Support vector machines

In this lecture we shall consider a class of simple supervised learning
machines for binary classification problems and apply results in the previ-
ous lectures on consistent learning algorithms. Our learning machines are
(V × Z2 , Hlin , L, P(V × Z2 ), A) where V is a real Hilbert space, Hlin consists
of linear classifiers, defined below, L is a (0 − 1) loss function (resp. regu-
larized (0 − 1) loss function) and A is a hard SVM algorithm (resp. a soft
SVM algorithm), which we shall learn in today lecture. The original SVM
algorithm is the hard SVM algorithm, which was invented by Vapnik and
Chervonenkis in 1963. The current standard incarnation (soft margin) was
proposed by Cortes and Vapnik in 1993 and published in 1995.
7.1. Linear classifier and hard SVM. For (w, b) ∈ V × R we set

(7.1) f(w,b) (x) := hw, xi + b.
Definition 7.1. A linear classifier is a function sign f(w,b) : V → Z2 ,
x 7→ signf(w,b) (x) ∈ {−1, 1} = Z2 .
+ −1
We identify each linear classifier with the half space H(w,b) := signf(w,b) (1) =
−1 −1
f(w,b) (R≥0 ) ⊂ V and set H(w,b) := f(w,b) (0) ⊂ V . Note that each hyperplane
+
H(w,b) ⊂ V defines H(w,b) up to a reflection of V around H(w,b) and therefore
defines the affine function f(w,b) up to a multiplicative factor λ ∈ R∗ . Denote
by HA (V ) the set of of all hyperplanes in the affine space V . Then Hlin is
a double cover of HA (V ) with the natural projection π : Hlin → HA (V )
defined above.
Definition 7.2. A training sample S = (x1 , y1 ), · · · , (xm , ym ) ∈ (V ×
+
{±1})m is called separable, if there is a half space H(w,b) ⊂ V that cor-
+
rectly classifies S, i.e. for all i ∈ [1, m] we have xi ∈ H(w,b) iff yi = 1. In
other words, the linear classifier signf(w,b) is a minimizer of the empirical
risk function RS0−1 : Hlin → R associated to the zero one loss function L(0−1) .
+
Remark 7.3. (1) A half space H(w,b) correctly classifies S if and only if the
(0−1)
empirical risk function R̂S (f(w,b) ) = 0 and sign f(w,b) is a linear classifier
+
associated to H(w,b) .
(2) Write S = S+ ∪ S− where
S± := {(x, y) ∈ S| y = ±1}.
Let P r : (V × {±1})m → V m denote the canonical projection. Then S
is separable if and only if there exists a hyper-plane H(w,b) that separates
[Pr(S+ )] and [P r(S− )], where recall that [(x1 , · · · , xm )] = ∪m
i=1 {xi } ⊂ V . In
this case we say that H(w,b) correctly separates S.
(3) If a training sample S is separable then the separating hyperplane
is not unique, and hence there are many minimizers of the empirical risk
MACHINE LEARNING 49
(0−1)
function R̂S . Thus, given S, we need to find a strategy for selecting one
of these ERM’s, or equivalently for selecting a separating hyperplane H(w,b) ,
+
since the associated half-space H(w,b) is defined by H(w,b) and any training
value (xi , yi ). The standard approach in the SVM framework is to choose
H(w,b) that maximizes the distance to the closest points xi ∈ [P r(S)]. This
approach is called the hard SVM rule. To formulate the hard SVM rule we
need a formula for the distance of a point to a hyperplane H(w,b) .
Lemma 7.4 (Distance to a hyperplane). Let V be a real Hilbert space and
H(w,b) := {z ∈ V | hz, wi + b = 0}. The distance of a point x ∈ V to H(w,b)
is given by
|hx, wi + b|
(7.2) ρ(x, H(w,b) ) := inf ||x − z|| = .
z∈H(w,b) ||w||
Proof. Since H(w,b) = H(w,b)/λ for all λ > 0, it suffices to prove (7.2) for the
case ||w|| = 1 and hence we can assume that w = e1 . Now formula (7.2)
follows immediately, noting that H(e1 ,b) = H(e1 ,0) − be1 .
Let H(w,b) separate S = {(x1 , y1 ), · · · , (xm , ym )} correctly. Then we have
yi = sign(hxi , wi + b),
=⇒ |hxi , wi + b| = yi (hx, wi + b).
Hence, by Lemma 7.4, the distance between H(w,b) and S is
mini yi (hxi , wi + b)
(7.3) ρ(S, H(w,b) ) := min ρ(xi , H(w,b) ) = .
i ||w||
The distance ρ(S, H(w,b) ) is also called the margin of a hyperplane H(w,b)
w.r.t. S. The hyperplanes, that are parallel to the separating hyperplane
and passing through the closest points on the negative or positive sides are
called marginal.
Denote by HS the subset of π(Hlin ) = HA (V ) that consists of hyperplanes
separating S. Set
(7.4) A∗hs (S) := arg max ρ(S, H(w,b)0 ).
H(w,b)0 ∈HS
Now we define a map Ahs : ∪m (V × Z2 )m → Hlin by letting Ahs (S) ∈

π −1 A∗hs (S)
to be the linear classifier that correctly classifies S.
Definition 7.5. A hard SVM is a learning machine (V × Z2 , Hlin , L(0−1) ,
P(V × Z2 ), Ahs ).
The domain of the optimization problem in (7.4) is HS , which is not easy
to determine. So we replace this problem by another optimization problem
over a larger convex domain as follows.
Lemma 7.6. For S = {(x1 , y1 ), · · · , (xm , ym )} we have
+
(7.5) Ahs (S) = H(w,b) where (w, b) = arg max min yi (hw, xi i + b).
(w,b):||w||≤1 i
Proof. If H(w,b) separates S then ρ(S, H(w,b) ) = mini yi (hw, xi i + b). Since
the constraint ||w|| ≤ 1 does not effect on H(w,b) , which is invariant under a
positive rescaling, (7.3) implies that
(7.6) max min yi (hw, xi i + b) ≥ max ρ(S, H(w,b)0 ).
(w,b):||w||≤1 i H(w,b)0 ∈HS
Next we observe that if H(w,b) 6∈ HS then

min yi (hw, xi i + b) < 0.
i
Combining this with (7.6) we obtain

max min yi (hw, xi i + b) = max min yi (hw, xi i + b).
(w,b):||w||≤1 i (w,b):H(w,b) ∈HS i
This completes the proof of Lemma 7.6.

A solution Ahs (S) of the equation (7.5) maximizes the enumerator of the
far RHS of (7.3) under the constraint ||w|| ≤ 1. In the Proposition below we
shall show that Ahs (S) can be found as a solution to the dual optimization
problem of minimizing the dominator of the RHS (7.3) under the constraint
that the enumerator of the far RHS of (7.3) has to be fixed.
Proposition 7.7. A solution to the following optimization problem, which
is called Hard-SVM,
(7.7) (w0 , b0 ) = arg min{||w||2 : yi (hw, xi i + b) ≥ 1 for all i}
w,b
produces a solution (w, b) := (w0 /||w0 ||, b0 /||w0 ||) of the optimization prob-
lem (7.5).
Proof. Let (w0 , b0 ) be a solution of (7.7). We shall show that (w0 /||w0 ||, b0 /||w0 ||)
is a solution of (7.5). It suffices to show that the margin of the hyperplane
H(w0 ,b0 ) is greater than or equal to the margin of the hyperplane associated
to a (and hence any) solution of (7.5).
Let (w∗ , b∗ ) be a solution of Equation (7.5). Set
γ ∗ := min yi (hw∗ , xi i + b∗ )
i
which is the margin of the hyperplane H(w∗ ,b∗ ) by (7.3). Therefore for all i
we have
yi (hw∗ , xi i + b∗ ) ≥ γ ∗
or equivalently
w∗ b∗
yi∗ (h , xi i + ) ≥ 1.
γ∗ γ∗
∗ ∗
Hence the pair ( wγ ∗ , γb ∗ ) satisfies the condition of the quadratic optimization
problem in (7.7). It follows that
w∗ 1
||w0 || ≤ || || = ∗ .
γ∗ γ
MACHINE LEARNING 51
Hence for all i we have

w0 b0 yi (hw0 , xi i + b0 ) 1
yi ( + )= ≥ ≥ γ∗.
||w0 || ||w0 || ||w0 || ||w0 ||
This implies that the margin of H(w0 ,b0 ) satisfies the required condition. This
completes the proof of Proposition 7.7.
Remark 7.8. (1) The optimization problem of (7.4) is a specific instance
of quadratic programming (QP), a family of problems extensively studied in
optimization. A variety of commercial and open-source solvers are available
for solving convex QP problems. It is well-known that there is a unique
solution of (7.4).
(2) In practice, when we have a sample set S of large size, then S is not
separable, thus the application of hard SVM is limited.
Exercise 7.9. (1) Show that the vector w0 of the solution (w0 , b0 ) in (7.7)
of the SVM problem is a linear combination of the training set vectors
x1 , · · · , xm .
(2) Show that xi lies on the marginal hyperplanes hw0 , xi + b0 = ±1.
A vector xi appears in the linear expansion of the weight vector w0 in
Exercise 7.9 is called a support vector.
7.2. Soft SVM. Now we consider the case when the sample set S is not sep-
arable. There are at least two possibilities to overcome this difficulty. The
first one is to find a nonlinear embedding of patterns into a high-dimensional
space. To realize this approach we use a kernel trick that embeds the pat-
terns in an infinite dimensional Hilbert space space, which we shall learn in
the next lecture. The second way is to seek a predictor sign f(w,b) such that
−1
H(w,b) = f(w,b) (0) still has maximal margin in some sense. More precisely,
we shall relax the hard SVM rule (7.7) by replacing the constraint
(7.8) yi (hw, xi i + b) ≥ 1
by the relaxed constraint
(7.9) yi (hw, xi i) + b ≥ 1 − ξi
where ξi ≥ 0 are called the slack variables. The slack variables are commonly
used in optimization to define relaxed versions of some constraints. In our
case a slack variable ξi measures the distance by which vector xi violates
the original inequality in the LHS of (7.8).
The relaxed hard SVM rule is called the soft SVM rule.
Definition 7.10. The soft SVM algorithm Ass : (Rd × Z2 )m → Hlin with
slack variables {ξ ∈ Rm
≥0 } is defined as follows
Ass (S) = sign f(w0 ,b0 ) (S)

where (w0 , b0 ) satisfies the following equation with ξ = (ξ1 , · · · , ξm )

1
(7.10) (w0 , b0 , ξ) = arg min(λ||w||2 + ||ξ||l1 )
w,b,ξ m
(7.11) s. t. ∀i, yi (hw, xi i + b) ≥ 1 − ξi and ξi ≥ 0.
In what follows we shall show that the soft SVM algorithm Ass is a so-
lution of a regularized loss minimization rule, which is a refinement of the
ERM rule.
Digression Regularized Loss Minimization (RLM) is a learning algorithm

on a learning model (Z, H, L, P ) in which we jointly minimize the empirical
risk R̂SL and a regularization function. Formally, a regularization function
is a mapping R : H → R and the regularized loss minimization rule is a
map Arlm : Z n → H such that Arlm (S) is a minimizer of the empirical
regularized loss function R̃SL := R̂SL + R : H → R.
As the ERM algorithm works under certain condition, the RLM algorithm
also works under certain conditions, see e.g. [SSBD2014, Chapter 13] for
detailed discussion.
The loss function for the soft SVM learning machine is the hinge loss
function Lhinge : Hlin × (V × {±1}) → R defined as follows
(7.12) Lhinge (h(w,b) , (x, y)) := max{0, 1 − y(hw, xi + b)}.
Hence the empirical hinge risk function is defined as follows for S =
{(x1 , y1 ) · · · , (xm , ym )}
m
1 X
RShinge (h(w,b) ) = max{0, 1 − yi (hw, xi i + b}.
m
i=1
Lemma 7.11. The Equation (7.10) with constraint (7.11) for Ass is equiv-
alent to the following regularized risk minimization problem, which does not
depend on the slack variables ξ:
Ass (S) = arg min λ||w||2 + RShinge (f(w,b) ) ∈ Hlin .

(7.13)
f(w,b)
Proof. Let us fix (w0 , b0 ) and minimize the RHS of (7.10) under the con-
straint (7.11). It is straightforward to see that ξi = Lhinge (w, b), (xi , yi ) .

Using this and comparing (7.10) with (7.13), we complete the proof of
Lemma 7.11.
From Lemma 7.11 we obtain immediately the following

Corollary 7.12 (Definition). A soft SVM is a learning machine (V ×
Z2 , Hlin ,
Lhinge , P(V × Z2 ), Arlm ).
MACHINE LEARNING 53
Remark 7.13. The hinge loss function Lhinge enjoys several good properties
that justify the preference of Lhinger as a loss function over the zero-one loss
function L(0−1) , see [SSBD2014, Subsection 12.3, p. 167] for discussion.
7.3. Sample complexities of SVM.
Exercise 7.14. Prove that V C dim Hlin = dim V + 1.
From the Fundamental Theorem of binary classification 5.15 and Exercise
(7.14) we obtain immediately the following
Proposition 7.15. The binary classification problem (V ×Z2 , Hlin , L(0−1) , P(V ×
Z2 )) ha a uniformly consistent learning algorithm if and only if V is finite
dimensional.
In what follow we shall show the uniform consistent of hard SVM and
soft SVM replacing the statistical model P(Rd × Z2 ) by a smaller class.
Definition 7.16. ([SSBD2014, Definition 15.3]) Let µ be a distribution on
V × Z2 . We say that µ is separable with a (γ, ρ)-margin if there exists
(w∗ , b∗ ) ∈ V × R such that kwk = 1 and such that
µ{(x, y) ∈ V × Z2 | y(hw∗ , xi + b∗ ) ≥ γ and ||x|| ≤ ρ} = 1.
Similarly, we say that µ is separable with a (γ, ρ)-margin using a homoge-
neous half-space if the preceding holds with a half-space defined by a vector
(w∗ , 0).
Definition 7.16 means that the set of labeled pairs (x, y) ∈ V × Z2 that
satisfy the condition
y(hw∗ , xi + b∗ ) ≥ γ and ||x|| ≤ ρ
has a full µ-measure, where µ is a separable measure on with (γ, ρ)-margin.
Remark 7.17. (1) We regard an affine function f(w,b) : V → R as a linear
function fw0 : V 0 → R where V 0 = he1 i⊗R ⊕ V , i.e., we incorporate the
bias term b of f(w,b) in (7.1) into the term w as an extra coordinate. More
precisely we set
w0 := be1 + v and x0 := e1 + x.
Then
f(w,b) (x) = fw0 (x0 ).
Note that the natural projection of the zero set fw−1 0
0 (0) ⊂ V to V is the zero
set H(w,b) of f(w,b) .
(2) By Remark 7.17 (1), we can always assume that a separable measure
with (γ, ρ)-margin is a one that uses a homogeneous half-space by enlarging
the instance space V .
Denote by P(γ,ρ) (V ×Z2 ) the subset of P(V ×Z2 ) that consists of separable
measures with a (γ, ρ)-margin using a homogeneous half-space. Using the
Rademacher complexity, see [SSBD2014, Theorem 26.13, p. 384], we have
the following estimate of the sample complexity of the learning machine

(V × Z2 , Hlin , L(0−1) , P(γ,ρ) (V × Z2 ), Ahs ).
Theorem 7.18. ([SSBD2014, Theorem 15.4, p. 206]) Let µ ∈ P(γ,ρ) (V ×

Z2 ). Then we have
r r
m m (0−1) 4(ρ/γ)2 2 log(2/δ)
µ {S ∈ (V × Z2 ) | Rµ (Ahs (S)) ≤ + } ≥ 1 − δ.
m m
Denote by Pρ (V × Z2 ) the set of of probability measures on V × Z2 whose

support lies in B(0, ρ) × Z2 where B(0, ρ) is the ball of radius ρ centered at
the origin of V . Now we shall examine the sample complexity of the soft
SVM learning machine (V ×Z2 , Hlin , Lhinge , Pρ (V ×Z2 ), Ass ). The following
theorem is a a consequence of Lemma 7.11 and a general result on the sample
complexity of RLM under certain conditions, see [SSBD2014, Corollary 13.8,
p. 179].
Theorem 7.19. ([SSBD2014, Corollary 15.7, p. 208]) Let µ ∈ Pρ (V × Z2 ).

Then for every r > 0 we have

Eµm Rµ(0−1) Ass (S) ≤ Eµm Rµhinge Ass (S)
r
8ρ2 r2
(7.14) ≤ min Rµhinge (hw ) + .
w∈B(0,r) m
Exercise 7.20. Using the Markov inequality, derive from Theorem 7.19 an
upper bound for the sample complexity of the soft SVM.
Theorem 7.19 and Exercise 7.20 imply that we can control the sample
complexity of a soft SVM algorithm as a function of the norm of the under-
lying Hilbert space V , independently of the dimension of V . This becomes
highly significant when we learn classifiers h : V → Z2 via embeddings into
high dimensional feature spaces.
7.4. Conclusion. In this section we consider two classes of learning ma-

chines for binary classification. The first class consists of learning models
(V ×Z2 , Hlin , L(0−1) , P(V ×Z2 )) with finite VC-dimension iff and only if V is
finite dimensional. If µ ∈ P(V × Z2 ) is separable with (γ, ρ)-margin then we
can upper bound the sample complexity of the hard SVM algorithm Ahs for
(V × Z2 , Hlin , L(0−1) , µ) using the ration ρ/γ and Rademacher’s complexity.
The second class consists of learning machines (V × Z2 , Hlin , Lhinge , Pρ (V ×
Z2 ), Ass ). The soft SVM algorithm Ass is a solution of a regularized ERM
and therefore we can apply here general results on sample complexity of
regularized ERM algorithms.
MACHINE LEARNING 55
8. Kernel based SVMs

In the previous lecture we considered the hypothesis class Hlin of linear
classifiers. A linear classifier sign f(w,b) correctly classifies a training sam-
ple S ⊂ (V × {±1})m iff the zero set H(w,b) of f(w,b) separates the subsets
[P r(S− )] and [P r(S+ )]. By Radon’s theorem any set of distinct (d+2) points
in Rd can be partitioned into two subsets that cannot be separated by a hy-
perplane in Rd . Thus it is reasonable to enlarge the hypothesis class Hlin
by adding polynomial classifiers. Now we observe that any (polynomial)
function f on Rd can be regarded as the restriction of a new coordinate
function y on Rd × R(y) to the image of the graph Γf (Rd ) ⊂ Rd × R of
f , i.e., f (x) = y(Γf (x)). However, the computational complexity of SVM
with learning by polynomial embedding {(x, f (x))| x ∈ Rd } may be compu-
tationally expensive. The common solution to this concern is kernel based
learning. The term “kernels” is used in this context to describe inner prod-
ucts in the feature space. Namely we are interested in classifiers of the
form
sign h̃ : X → Z2 , h̃(x) := hh, ψ(x)i,
where h is an element in a Hilbert space W , ψ : X → W is a “feature map”
and the kernel function Kϕ : X × X → R is defined
(8.1) Kψ (x, y) := hψ(x), ψ(y)i.
We shall see that to solve the hard SVM optimization problem (7.7) for
h ∈ W it suffices to learn K. This kernel trick requires less computational
complexity than the one for learning ψ : X → W .
8.1. Kernel trick. It is known that a solution of a hard SVM can be ex-
pressed as a linear combination of support vectors (Exercise 7.9). If the
number of support vectors is less than the dimension of the instance space,
then this property simplifies the search for a solution of the hard SVM.
Below we shall show that this property is a consequence of the Represen-
ter Theorem concerning solutions of a special optimization problem. The
optimization problem we are interested in is of the following form:

(8.2) w0 = arg min f hw, ψ(x1 )i, · · · , hw, ψ(xm )i + R(kwk)
w∈W
where xi ∈ X , w and ψ(xi ) are elements of a Hilbert space W , f : Rm → R

is an arbitrary function and R : R+ → R is a monotonically non-decreasing
function. The map ψ : X → W is often called the feature map, and W is
called the feature space.
The following examples show that the optimization problem for the hard
(resp. soft) SVM algorithm is an instance of the optimization problem (8.2).
Example 8.1. Let S = {(x1 , y1 ), · · · , (xm , ym ) ∈ (V × Z2 )m .
(1) Plugging in Equation (8.2)
R(a) := a2 ,

0 if yi (ai ) ≥ 1 for all i
f (a1 , · · · , am ) :=
∞ otherwise
we obtain Equation (7.7) of hard SVM for homogeneous vectors (w, 0), re-
placing ai by hw, xi i. The general case of non-homogeneous solutions (w, b)
is reduced to the homogeneous case by Remark 7.17.
(2) Plugging in Equation (8.2)
R(a) := λa2 ,
m
1 X
f (a1 , · · · , am ) := max{0, 1 − yi ai }
m
i=1
we obtain Equation (7.13) of soft SVM for a homogeneous solution Ass (S),
identifying Ass (S) with its parameter (w, 0), S with {(x1 , y1 ) · · · (xm , ym )}
and replacing ai with hw, xi i.
Theorem 8.2 (Representer Theorem). Let ψ : X → W be a feature mapping
from an instance space X to a Hilbert space W and w0 a solution of (8.2).
Then the projection of w0 to the subspace hψ(x1 ), · · · , ψ(xm )i⊗R in W is
also a solution of (8.2).
Proof. Assume that w0 is a solution of (8.2). Then we can write
m
X
w0 = αi ψ(xi ) + u
i=1
where hu, ψ(xi )i = 0 for all i. Set w̄0 := w0 − u. Then

(8.3) kw̄0 k ≤ kw0 k
and since hw̄0 , ψ(xi )i = hw0 , ψ(xi )i we have

(8.4) f hw̄0 , ψ(x1 )i, · · · , hw̄0 , ψ(xm )i = f hw0 , ψ(x1 )i, · · · , hw0 , ψ(xm )i .
From (8.3), (8.4) and taking into account the monotonicity of R, we conclude
that w̄0 is also a solution of (8.2). This completes the proof of Theorem
8.2.
The Representer Theorem implies that it suffices to find a solution of

Equation (8.2) in a finite dimensional subspace W1 ⊂ W . In what follows
we shall describe a method to solve the minimization problem of (8.2) on
W1 , which is called the kernel trick.
Let
• K : X × X → R, K(x, x0 ) := hψ(x), ψ(x0 )i be a kernel function,
• G = (GPij ) := K(xi , xj ) - a Gram matrix,
• w0 = m i=1 αj ψ(xj ) ∈ W1 - a solution of Equation (8.2).
Then α = (α1 , · · · , αm ) is a solution of the following minimization prob-
lem
MACHINE LEARNING 57
v
m
X m
X u m
uX
(8.5) arg minm f αj Gj1 , · · · , αj Gjm +R t αi αj Gji .
α∈R
j=1 j=1 i,j=1
Pm
Recall that the solution w0 = i=1 αj ψ(xj ) of the hard (resp. soft) SVM
optimization problem, where (α1 , · · · , αm ) is a solution of (8.5), produces a
“nonlinear” classifier ŵ0 : X → Z2 associated to as follows
ŵ0 (x) := sign w0 (x)
where
m
X m
X
(8.6) w0 (x) := hw0 , ψ(x)i = αj hψ(xj ), ψ(x)i = αj K(xj , x).
i=1 j=1
To compute (8.6) we need to know only the kernel function K and not the
mapping ψ, nor the inner product h, i on the Hilbert space W .
This motivates the following question.
Problem 8.3. Find a sufficient and necessary condition for a kernel func-
tion, also called a kernel, K : X × X → R such that K can be written as
K(x, x0 ) = hψ(x), ψ(x0 )i for a feature mapping ψ : X → W , where W is a
real Hilbert space.
Definition 8.4. If K satisfies the condition in Problem 8.3 we shall say
that K is generated by a (feature) mapping ψ. The target Hilbert space is
also called a feature space.
8.2. PSD kernels and reproducing kernel Hilbert spaces.
8.2.1. Positive semi-definite kernel.

Definition 8.5. Let X be an arbitrary set. A map K : X × X → R is
called positive semi-definite kernel (PSD kernel) iff for all x1 , · · · , xm the
Gram matrix Gij = K(xi , xj ) is positive semi-definite.
Theorem 8.6. A kernel K : X × X → R is induced by a feature map to a
Hilbert space if and only if it is positive semi-definite.
Proof. 1) First let us prove the “only if” assertion of Theorem 8.6. Assume
that K(x, x0 ) = hψ(x), ψ(x0 )i for a mapping ψ : X → W , where W is a
Hilbert space. Given m points x1 , · · · , xm ∈ X we consider the subspace
Wm ⊂ W generated by ψ(x1 ), · · · , ψ(xm ). Using the positive definite of the
inner product on Wm , we conclude that the Gram matrix Gij = K(xi , xj )
is positive semi-definite. This proves the “only if” part of Theorem 8.6
2) Now let us prove the “if” part. Assume that K : X × X → R is positive
semi-definite. For each x ∈ X let Kx ∈ RX be the function defined by
Kx (y) := K(x, y).
Denote by
N (f )
X
X
W := {f ∈ R | f = ai Kxi , ai ∈ R and N (f ) < ∞}.
i=1
Then W is equipped with the following inner-product

X X X
h αi Kxi , βj Kyj i := αi βj K(xi , yj ).
i j i,j
The PSD property of K implies that the inner product is positive semi-
definite, i.e.
X X
h αi Kxi , αi Kxi i ≥ 0.
i j
Since the inner product is positive semi-definite, the Cauchy-Schwarz in-
equality implies for f ∈ W and x ∈ X
(8.7) hf, Kx i2 ≤ hf, f ihKx , Kx i.
Since for all x, y we have Ky (x) = K(y, x) = hKy , Kx i , it follows that for
all f ∈ W we have
(8.8) f (x) = hf, Kx i.
Using (8.8), we obtain from (8.7) for all x ∈ X
|f (x)|2 ≤ hf, f iK(x, x).
This proves that the inner product on W is positive definite and hence W
is a pre-Hilbert space. Let H be the completion of W . The map x 7→ Kx
is the desired mapping from X to H. This completes the proof of Theorem
8.6.
Example 8.7. (1) (Polynomial kernels). Assume that P is a polynomial in
one variable with non-negative coefficients. Then the polynomial kernel of
the form K : Rd × Rd → R, (x, y) 7→ P (hx, y)i is a PSD kernel. This follows
from the observations that if Ki : X × X → R, i = 1, 2, are PDS kernes then
(K1 +K2 )(x, y) := K1 (x, y)+K2 (x, y) is a PDS kernel, and (K1 ·K2 )(x, y) :=
K1 (x, y) · K2 (x, y) is a PDS kernel. In particular, K(x, y) := (1 + hx, yi)2 is
a PSD kernel.
(2)(Exponential kernel). For any γ > 0 the kernel K(x, y) := exp γ ·
hx, yi is a PDS kernel, since it is the limit of a polynomials in hx, yi with
non-negative coefficients.
Exercise 8.8. (1) Show that the Gaussian kernel K(x, y) := exp − γ2 ||x −
y||2 ) is a PDS kernel.
(2) Let X = B(0, 1) - the open ball of radius 1 centered at the origin
0 ∈ Rd . Show that K(x, y) := (1 − hx, yi)−p is a PDS kernel for any p > 0.
MACHINE LEARNING 59
8.2.2. Reproducing kernel Hilbert space. Given a PSD kernel K : X ×X → R

there exist many feature maps ϕ : X → W such that K is generated by a
feature map ϕ : X → W . Indeed, if K(x, x) = hϕ(x), ϕ(x)i then K(x, x) =
he ◦ ϕ(x), e ◦ ϕ(x)i for any isometric embedding e : W → W 0 . However,
there is a canonical choice for the feature space, a so-called reproducing
kernel Hilbert space.
Definition 8.9 (Reproducing kernel Hilbert space). Let X be an instance
set and H ⊂ RX a real Hilbert space of functions on X with the unique
vector space structure such that for x ∈ X the evaluation map
evx : H → R , evx (f ) := f (x)
is a linear map. 15 Then H is called a reproducing kernel Hilbert space
(RKHS) on X if for all x ∈ X the linear map evx is bounded i.e.,
sup evx (f ) < ∞.
f ∈B(0,1)⊂H
Remark 8.10. Let H be a RKHS on X and x ∈ X . Since evx is bounded,
by the Riesz representation theorem there is a function kx ∈ H so that
f (x) = hf, kx i for all f ∈ H. Then the kernel
K(x, y) := hkx , ky i
is a PSD kernel. K is called the reproducing kernel of H.
Thus every RKHS H on X produces a PSD kernel K : X × X → R.
Conversely, Theorem 8.11 below asserts that every PSD kernel reproduces
a RKHS H.
Theorem 8.11. Let K : X × X → R be a PSD kernel. There there exists
a unique RKHS H such that K is the reproducing kernel of H.
Proof. By Theorem 8.6, given a PSD kernel K : X × X → R, there exists a
RKHS
H := {Kx | Kx (y) = K(x, y) for all x, y ∈ X }
such that
(8.9) ∀x, y ∈ X we have K(x, y) = hKx , Ky i.
From (8.9) we conclude that the evaluation map evx : H → R, evx (Ky ) =
K(y, x) is a linear bounded map for all x, y, since
p
||evx || = max evx (Ky ) = max K(y, x) ≤ K(x, x).
||Ky ||=1 ||Ky ||=1
Hence H is a RKHS.
To show the uniqueness of a RKHS H such that K is the reproducing
kernel of H we assume that there exists another RKHS H0 such that for all
x, y ∈ X there exist kx , ky ∈ H0 with the following properties
K(x, y) = hkx , ky i and f (x) = hf, kx i for all f ∈ H.
15In other words, the vector structure on H is induced from the vector structure on R
via the evaluation map.
We define a map g : H → H0 by setting g(Kx ) = kx . It is not hard to see

that g is an isometric embedding. To show that g extends to an isometry
it suffices to show that the set kx is dense in H0 . Assume the opposite, i.e.
there exists f ∈ H0 such that hf, kx i = 0 for all x. But this implies that
f (x) = 0 for all x and hence f = 0. This completes the proof of Theorem
8.11.
8.3. Kernel based SVMs and their generalization ability.
8.3.1. Kernel based SVMs. Let K : X × X → R be a PSD kernel. Denote
by H(K) the RHKS of functions on X that produces K. Each function
h ∈ H(K) defines a binary classifier
sign h : X → Z2 .
Denote by Klin the set of all binary classifiers sign h where h ∈ H(K). Using
the Representer Theorem 8.2 and Example 8.1 (1), we replace the algorithm
Ahs of a hard SVM by a kernel based algorithm.
Definition 8.12. A kernel based hard SVM is a learning machine
(X ×Z2 , Klin , L(0−1) , P(X ×Z2 ), Ahk ), where for S = {(x1 , y1 ), · · · , (xm , ym )}
∈ (X × Z2 )m and any x ∈ X we have
m
X
Ahk (S)(x) = sign αj K(xj , x) ∈ Z2 ,
i=1
and α := (α1 , · · · , αm ) is the solution of the following optimization problem
(8.10) v
m m
u m
X X uX

α = arg min f αj K(xj , x1 ), · · · , αj K(xj , xm ) +R(t αi αj K(xi , xj )) ,
j=1 j=1 i,j=1
where R and f are defined in Example 8.1(1).

Using the Representer Theorem 8.2 and Example 8.1 (2) , we replace the
algorithm Ass of a soft SVM by a kernel based algorithm.
Definition 8.13. A kernel based soft SVM is a learning machine
(X ×Z2 , Klin , Lhinge , P(X ×Z2 ), Ask ), where for S = {(x1 , y1 ), · · · , (xm , ym )}
∈ (X × Z2 )m and any x ∈ X we have
m
X
Ask (S)(x) = sign αj K(xj , x) ∈ Z2 ,
i=1
and α := (α1 , · · · , αm ) is the solution of the following optimization problem
(8.11) v
m m
u m
X X uX

α = arg min f αj K(xj , x1 ), · · · , αj K(xj , xm ) +R(t αi αj K(xi , xj )) ,
j=1 j=1 i,j=1
where R and f are defined in Example 8.1(2).

MACHINE LEARNING 61
8.3.2. Generalization ability of kernel based SVMs.

• The advantage of working with kernels rather than directly optimizing in
the feature space is that in some situations the dimension of the feature space
is extremely large while implementing the kernel function is very simple
and in many case the computational time complexity of solving (8.10) is a
polynomial on the variable of the size of xi , i ∈ [1, m], see [SSBD2014, p.
221-223].
• The upper bound for the sample complexity of hard SVM in Theorem
7.18 is also valid for the sample complexity of the kernel based hard SVM
[SSBD2014, Theorem 26.3, p. 384] after adapting the condition of separa-
bility with (γ, ρ)-margin of a measure µ ∈ P(X × Z2 ) in terms of the kernel
function.
• The upper bound for the sample complexity of soft SVM in Theorem
7.19 is also valid for the sample complexity of the kernel based soft SVM,
after adapting the support condition a measure µ ∈ P(X × Z2 ) in terms of
the kernel function.
8.4. Conclusion. In this section we learn the kernel trick, which simplifies
the algorithm of solving hard SVM and soft SVM optimization problem,
using embedding of patterns into a Hilbert space. The kernel trick is based
on the theory of RKHS. The main difficulty of the kernel method is that
we still have no general method of selecting a suitable kernel for a concrete
problem. Another open problem is to improve the upper bound for sample
complexity of SVM algorithm, i.e., to find new conditions on µ ∈ P such that
the sample complexity of Ahk , Ask which is computed w.r.t. µ is bounded.
9. Neural networks
In the last lecture we examined kernel based SVMs which are generaliza-
tions of linear classifiers, which are also called perceptrons.
Today we shall examine other generalizations of linear classifiers which
are artificial neural networks, shortened as neural networks (otherwise, non-
artificial neural networks are (called) biological neural networks). The idea
behind neural networks is that many neurons can be joined together by
communication links to carry out complex computations. Neural networks
achieve outstanding performance on many important problems in computer
vision, speech recognition, and natural language processing.
Note that under “a neural network” one may think of a computing de-
vice, a learning model, or a hypothesis class of a learning model, a class of
(sequences of) multivariable functions.
In today lecture we shall investigate several types of neural networks,
their expressive power, i.e., the class of functions that can be realized as
elements in a hypothesis class of a neural network. In the next lecture we
shall discuss the current learning algorithm -stochastic gradient descend -
on neural networks.
9.1. Neural networks as computing devices. A neural network has a

graphical representation for multivariate functions of multi-variables hV,E,σ,w :
Rn → Rm or a sequence of multivariate functions of multi-variables {hiV,E,σ,w :
Rn → Rm | i ∈ N}.
• The quadruple (V, E, σ, w) consists of
+ the network graph (V, E), also called the underlying graph of the net-
work, where V is the set of nodes n, also called neurons, and E is the set of
directed edges connecting nodes of
+ σ - a family of functions σn : R → R, also called the activation function
of neuron n. Usually σn = σ is independent of n. Most common activation
functions are:
- the sign function σ(x) = sign(x),
- the threshold function σ(x) = 1R+ (x),
- the sigmoid function σ(x) := 1+e1−x , which is a smooth approximation to
the threshold function;
+ w : E → R - the weight function of the network.
• The networks architecture of a neural network is the triple G = (V, E, σ).
• A weight w : E → R endows each neuron n with a computing instruction of

type “ input-output”. The input I(n) of a neuron n is equal to P the weighted
sum of the outputs of all the neurons connected to it: I(n) = w(n0 n)O(n0 ),
where n0 n ∈ E is a directed edge and O(n0 ) is the output of the neuron n0 in
the network.
• The output O(n) of a neuron n is obtained from the input I(n) as follows:
O(n) = σ(I(n)).
MACHINE LEARNING 63
• The i-th input nodes give the output xi . If the input space is Rn then
we have n + 1 input-nodes, one of them is the “constant” neuron, whose
output is 1.
• There is a neuron in the hidden layer that has no incoming edges. This
neuron will output the constant σ(0).
• A feedforward neural network (FNN) has underlying acyclic directed
graph. Each FNN (E, V, w, σ) represents a multivariate multivariable func-
tion hV,E,σ,w : Rn → Rm which is obtained by composing the computing
instruction of each neuron on directed paths from input neurons to output
neurons. For each architecture (V, E, σ) of a FNN we denote by
HV,E,σ = {hV,E,σ,w : w ∈ RE }
the underlying hypothesis class of functions from the input space to the
output space of the network.
• A recurrent network (RNN) has underlying directed graph with a cycle.
By unrolling cycles in a RNN in time r, a RNN defines a map r : N+ :→
{F N N } such that [r(n)] ⊂ [r(n + 1)], where [r(n)] is the underlying graph
of r(n), see [GBC2016, §10.2, p. 368] and [Graves2012, §3.2, p. 22]. Thus a
RNN can be regarded as a sequence of multivariate multivariable functions
which serves as in a discriminative model for supervised sequence labelling.
Digression The goal of supervised sequence labelling is to assign sequences

of labels, drawn from a label space, to sequences of input data. For example,
one might wish to transcribe a sequence of acoustic features with spoken

words (speech recognition), or a sequence of video frames with hand gestures
(gesture recognition). Although such tasks commonly arise when analysing
time series, they are also found in domains with non-temporal sequences,
such as protein secondary structure prediction.
If the sequences are assumed to be independent and identically distributed,
we recover the basic framework of pattern classification, only with sequences
in place of patterns (of course the data-points within each sequence are not
assumed to be independent). In practice this assumption may not be the
case.
Example 9.1. A multilayer perceptron (MLP) is a type of FNN that has

vertices arranged in a disjoint union of layers V = ∪nl=0 Vi such that every
edge in E connects nodes in neighboring layers Vi , Vi+1 . The depth of the
M LP is m. V0 is called the input layer, Vn is called the output layer, the
other layer is called hidden.
Remark 9.2. Neural networks are abstraction of biological neural networks,

the connection weights w represent the strength of the synapses between the
neurons and the activation function σ is usually an abstraction representing
the rate of action potential firing in the cell. In its simplest form, this
function is binary, that is, either the neuron is firing or not. We can consider
activation function as a filter of relevant information, or introducing the non-
linearlity in regression problems.
In the remaining of today lecture we consider only FNNs. In particular

under NN’s we mean FNNs.
9.2. The expressive power of neural networks. In this section we want

to address the following
Question 9.3. What type of functions can be implemented using neural

networks.
First we consider representation of Boolian functions by neural networks.
Proposition 9.4 (Representation of Boolean functions). ( [SSBD2014, Claim

20.1, p. 271]) Every Boolean function f : Zd2 → Z2 can be represented exactly
by a feedforward neural network HV,E, sign with a single hidden layer contain-
ing at most 2d + 1 neurons and with the activation function σ(x) = sign(x).
Proof. Let (V, E) be a two-layer FNN with #V0 = d + 1, #V1 = 2d + 1,

#V2 = 1 and E consist of all possible edges between adjacent layers. As
before Z2 = {±1}. Now let f : Zd2 → Z2 . Let ui ∈ f −1 (1) ⊂ Zd2 and
k := #f −1 (1). Set
gi (x) := sign(hx, ui i − d + 1).

MACHINE LEARNING 65
Then {gi | i ∈ [1, k]} are linear classifiers and therefore can be implemented
by the neurons in V1 . Now set
k
X
f (x) := sign( gi (x) + k − 1)
i=1
which is also a linear classifier. This completes the proof of Proposition
9.4
In general case we have the following Universal Approximation Theorem,
see e.g. [Haykin2008, p. 167].
Theorem 9.5. Let ϕ be a nonconstant, bounded and monotone increas-
ing continuous function. For any m ∈ N, ε > 0 and any function F ∈
C0 ([0, 1]m ) there exists an integer m1 ∈ N and constants ai , bj , wij where
i ∈ [1, m1 ], j ∈ [1, m] such that
m1
X Xm
f (x1 , · · · , xm ) := αi ϕ( wij xj + bi )
i=1 j=1
for all (x1 , · · · , xm ) ∈ [0, 1]m we have

|F (x1 , · · · , xm ) − f (x1 , · · · , xm )| < ε.
9.3. Sample complexities of neural networks. A learning model (X ×
Y, HV,E,σ , L, P ) is called a neural network if HV,E,σ is a neural network and
HV,E,σ ⊂ Y X .
9.3.1. Neural networks for binary classification problem. In neural networks

with hypothesis class HV,E,σ for binary classification problems one often
choose the activation function σ to be the sign function, and the loss function
L to be L(0−1) .
Proposition 9.6. ([SSBD2014, Theorem 20.6, p. 274]) Let HV,E,sign be a
MLP. The VC-dimension of HV,E,sign is O(|E| log |E|).
Outline of the proof If H := HV,E,sign consists of exactly one percep-
tron, then Proposition 9.6 is valid since in this case V C dim H = |E in | =
O(|E| log |E|), where E in denotes the set of directed edges coming into the
perceptron.
We want to reduce the proof for the general case of a neural network to
the case of a single perceptron, using the known VC-dimension of a single
perceptron. Let m := V C dim(H). Using
(9.1) ΓH (m) = 2m ,
to prove Proposition 9.6, it suffices to show that
(9.2) ΓHV,E,σ (m) ≤ (em)|E| ,
since log2 (em) < 4 log(E) by (9.2) and ((9.1)).
Let V0 , · · · , VT be the layers of (E, V ). For t ∈ [1, T ] denote by Ht the

neural network HWt ,Et ,sign where Wt consists of inputs neurons in Vt−1 and
output neurons in Vt and Et consists of edges of H that connect Vt−1 with
Vt . Now we decompose
H = HT ◦ · · · ◦ H1 .
Lemma 9.7 (Exercises). (1) ([SSBD2014, Exercise 4, p. 282]) Let F1 ⊂ Z X
and F2 ⊂ Y Z . Set H := F2 ◦ F1 . Then ΓH (n) ≤ ΓF2 (n)ΓF1 (n).
(2) ([SSBD2014, Exercise 3, p. 282]) Let Fi be a set of function from X
to Yi for i = 1, 2. Then ΓF1 ×F2 (n) ≤ ΓF1 (n)ΓF2 (n).
By Lemma 9.7 (1) we have
ΓH (m) ≤ ΠTt=1 ΓHt (m).
Next we observe that
Ht = Ht,1 × · · · × Ht,|Vt | .
Each neuron ni on Vt has dt,i heading edges presenting the number of the
inputs for the linear classifier ni . Hence V C dim Ht,i = dt,i < m − 1. By
Lemma 9.7 and by Vapnik-Chervonenski-Sauer-Lemma we have
|Vt | em dt,i
ΓH (m) ≤ ΠTt=1 Πi=1 ( ) < (em)|E| ,
dt,i
which completes the proof of Proposition 9.6.
It follows from Proposition 9.6 that the sample complexity of the ERM
algorithm for (V ×Z2 , HV,E,sign , L(0−1) , P(V ×Z2 )) is finite. But the running
time for ERM algorithm in a neural network HV,E,sign is non-polynomial and
therefore it is impractical to use it [SSBD2014, Theorem 20.7, p. 276]. The
solution is to use the stochastic gradient descend, which we shall learn in
the next lecture.
9.3.2. Neural networks for regression problem. In neural networks with hy-
pothesis class HV,E,σ for regression problems one often chooses the activa-
tion function σ to be the sigmoid function σ(a) := (1 + e−a )−1 , and the loss
function L to be L2 , i.e., L(x, y, hw ) := 21 ||hw (x) − y||2 for hw ∈ HV,E,σ and
x ∈ X = Rn , y ∈ Y = Rm .
9.3.3. Neural networks for generative models in supervised learning. In gen-
erative models of supervised learning we need to estimate the conditional
distribution p(t|x). In many regression problems p is chosen as follows
[Bishop2006, (5.12), p. 232]
β −β
(9.3) p(t|x) = N (t|y(x, w), β −1 ) = √ exp (t − y(x, w)),
2π 2
where β is unknown parameter and y(x, w) is the expected value of t. Thus
the learning model is of the form (X , H, L, P ) where H := {y(t, x, w)} pa-
rameterized by a parameter w, β, and a statistical model P is a subset of
MACHINE LEARNING 67
P(X ), since the joint distribution µ(x, y) is completely defined by µX and

the conditional distribution µ(y|x).
Now assume that X = (x1 , · · · , xn ) are i.i.d. by µ ∈ P along with labels
(t1 , · · · , tn ). Then (9.3) implies
n
n n βX
(9.4) − log p(t|X, w, β) = − log β + log(2π) + |tn − y(xn , w)|2 .
2 2 2
i=1
As in the density estimation problem we want to minimize the LHS of (9.3).
Leaving β = const we minimize first the β -independent component of the
loss function
n
1X
(9.5) LS (w) = |y(xn , w)2 − tn |2 .
2
i−1
Once we know a solution wM L of the equation minimizing LS (w), the value

of β can be found by the following formula
n
1 1X
(9.6) = |y(xi , wM L ) − ti |2 .
βM L n
i=1
9.4. Conclusion. In this lecture we considered learning machines whose

hypothesis class consisted of functions or sequence of functions that can
be graphical represented by neural networks. Neural networks have good
expressive power and finite VC-dimension in binary classification problems
but the ERM algorithm in these networks has very high computational com-
plexity and therefore they are unpractical.
10. Appendix 1: Some basic notions in probability theory

Basis objects in probability theory (and mathematical statistics) are mea-
surable spaces (X , Σ), where Σ is a σ-algebra of subsets of a space X . A
countably additive measure µ on Σ is called a probability measure if µ ≥ 0
and µ(X ) = 1.
For this Appendix I use [Bogachev2007] as my main reference on measure
theory, the book [JP2003] for a clear and short exposition of probability
theory, and [Kallenberg2002] for a modern treatment of probability theory.
I also use [Borovkov1998] for its clear exposition of probability theory and
applications in statistics.
10.1. Dominating measures and the Radon-Nikodym theorem. Let
µ and ν be countably additive measures on a measurable space (X , Σ)
(i) The measure ν is called absolutely continuous with respect to µ (or domi-
nated by µ) if |ν|(A) = 0 for every set A with |µ|(A) = 0. Notation: ν << µ.
(ii) The measure ν is called singular with respect to µ, if there exists a set
Ω ∈ Σ such that
|µ|(Ω) = 0 and |ν|(X \ Ω) = 0.
Notation: ν ⊥ µ.
Theorem 10.1. (cf. [Bogachev2007, Theorem 3.2.2, p. 178]) Let µ and

ν be two finite measures on a measurable space (X , Σ). The measure ν
is dominated by the measure µ precisely when there exists a µ-integrable
function f such that ν is given by
Z
(10.1) ν(A) = f dµ
A
for each A ∈ Σ.
We denote ν by f µ for µ, ν, f satisfying the equation (10.1). The function
f is called the (Radon-Nikodym) density (or the Radon-Nikodym derivative)
of ν w.r.t. µ. The function f is denoted by dν/dµ.
10.2. Conditional expectation, conditional probability (measure)
and joint distribution. The important notion of conditional probabil-
ity in the case of uncountable measurable space (X , Σ) is defined via the
notion of conditional expectation. There are two reasons for this approach.
Firstly conditional probability can be obtained from conditional expectation:
P (A|B) = EP (·|B) 1A . Secondly, in general case, we can treat conditional ex-
pectation as a function satisfying certain axioms ([Bogachev2007, Chapter
10, p. 339].
10.2.1. Conditional expectation.
Definition 10.2. Let f ∈ L1 (µ). A conditional expectation of f with respect
to the σ-algebra B and the measure µ is a B-measurable µ-integrable function
EBµ f such that
Z Z
(10.2) gf dµ = gEB
µ f dµ
X X
for every bounded B-measurable function g.
A conditional expectation of an individual integrable function f is defined
as the conditional expectation of the corresponding class in L1 (µ).
Remark 10.3. 1) The defining equality (10.2) is equivalent to the following
relationship obtained by the substitution g = 1B :
Z Z
(10.3) f dµ = EBµ f dµ ∀B ∈ B.
X B
This follows from the fact that every bounded B-measurable function is
the uniform limit of simple B-measurable functions.
2) In the case where only one measure µ is given, for simplification of
notation and terminology one uses:
EB f := EB
µ f.
In the probabilistic literature one uses the notation
E(f |B) := EB
µ f.
MACHINE LEARNING 69
3) If Y is an integrable function on a probability space (X , µ) and B =

η −1 (Σ0 ) is generated by a measurable function (or mapping) η : (X , Σ) →
(X 0 , Σ0 ) 16 then one uses the notation
E(Y |η) := Eσ(η) Y,
where σ(η) is the sub-σ-algebra generated by η.
Theorem 10.4. ([Bogachev2007, Theorem 10.1.5, p. 341]. Suppose that µ
is a probability measure. To every function f ∈ L1 (µ), one can associate a
B-measurable function EB f such that
(1) EB f is a conditional expectation of f with respect to B;
(2) EB f = f µ-a.e. for every B-measurable µ-integrable function f ;
(3) EB f ≥ 0 µ-a.e. if f ≥ 0 µ-a.e.;
(4) if a sequence of µ-integrable functions fn converges monotonically de-
creasing or increasing to a µ-integrable function f , then EB fn → EB f µ-a.e.;
(5) For every p ∈ [1, +∞], the mapping EB defines a continuous linear op-
erator with norm 1 on the space Lp (µ). In addition, EB is the orthogonal
projection of L2 (µ) to the closed linear subspace generated by B-measurable
functions.
Remark 10.5. (1) In [JP2003, Definition 23.5, p. 200] the author choose the
last part of Property (5) of Theorem 10.4, namely the orthogonal projection
to L2 (µ), as the definition of conditional expectation, using some extension
arguments [JP2003, Lemma 23.1, Theorem 23.4, p. 201, 202].
(2) In the case B is generated by a mapping η to Rn the conditional
expectation of an integrable random variable ξ can be computed as follows
(see e.g. [Bogachev2007, p. 345])
E(ξ|η) := Eσ(η) (ξ) = f (η)
where f is a function on Rn that is defined by
Z
1
f (x) = lim ξdP.
r→0 P (η ∈ B(x, r)) {η∈B(x,r)}
(3) Generalizing the previous formula in (2), we express the conditional

expectation for arbitrary B = σ(η), where η : X → X 0 , using the Radon-
Nykodim theorem as in [AJLS2015, (3.2)]
dη∗ (Y µ)
(10.4) Eσ(η) µ
µ Y := η∗ Y := .
dη∗ (µ)
10.2.2. Conditional measure and conditional probability. The conditional mea-

sure (or conditional probability in the case of probability measure) of A ∈ Σ
w.r.t. B, is defined as follows
(10.5) µ(A|B) := Eµ (1A |B).
16One can take η to be the identity mapping, and Σ0 = B.
In probabilistic literature one omits µ in (10.5) and writes instead

P (A|B) := µ(A|B).
If B = ξ −1 (Σ0 ) where ξ : (X , Σ) → (X 0 , Σ0 ) is a measurable map, one uses
the notation
P (A|ξ) := µ(A|ξ) := µ(A|B).
Remark 10.6. (1) By (10.5) and (10.4), for every A ∈ Σ there exists a
function ζA : X 0 → R such that
µ(A|ξ)(x) = ζA (ξ(x))
for all x ∈ X . Then one sets
(10.6) µB (A|ξ = x) := ζA (x).
The RHS of (10.6) is called the measure of A under conditioning ξ = x.
(2) Let π : (X , Σ) → (Y, Σ0 ) be a measurable map. For any A ∈ Σ and
B ∈ B = π −1 (Σ0 ) we have
Z
(10.7) µ(A ∩ B) = µB (A|x)dµ(x).
B
Since µB (A|x)
is a B-measurable function on X for each A, by [Bogachev2007,
Theorem 2.12.3, p. 144] on B-measurable functions, we have for any E ∈ Σ0
Z
−1
(10.8) µ(A ∩ π (E)) = µy (A) dπ∗ (µ)(y)
E
where µy (A) := µB (A, x) is a Σ0 -measurable function.

10.2.3. Regular conditional measure.
Definition 10.7. [Bogachev2007, Definition 10.4.1, p. 357]) Suppose we
are given a sub-σ-algebra B ⊂ Σ. A function
µB (·, ·) : Σ × X → R
is called a regular conditional measure on Σ w.r.t. B if
(1) for every x ∈ X the function A 7→ µB (A, x) is a measure on Σ,
(2) for every A ∈ Σ the function x 7→ µB (A, x) is B-measurable and µ-
integrable,
(3) For all A ∈ Σ, B ∈ B the following formula for joint probability holds,
cf (10.7)
Z
(10.9) µ(A ∩ B) = µB (A, x)d|µ|(x).
B
Remark 10.8. (cf. [Bogachev2007, Definition 10.4.2, p. 358]) In the case B

is generated by a measurable map π : (X , Σ) → (Y, Σ0 ) as in Remark 10.6(2),
it is more convenient to parameterize regular conditional measures by points
of the space Y. We say that a system of regular conditional measures µy ,
y ∈ Y, is a function on Σ × Y such that for every fixed y it is a measure on
MACHINE LEARNING 71
Σ, for every fixed A ∈ Σ is measurable w.r.t. Σ0 and |µ| ◦ π −1 -integrable,

and for all A ∈ Σ and E ∈ Σ0 we have
Z
−1
(10.10) µ(A ∩ π (E)) = µy (A) dπ∗ (|µ|)(y).
E
10.2.4. Regular conditional probability and joint distribution.

Definition 10.9. ([Borovkov1998, Definition 2, §20, p. 107]) Assume that a
(regular) 17 conditional probability P (x ∈ B|y) for each y ∈ Y is absolutely
continuous w.r.t. some measure µ in X, i.e.,
Z
(10.11) P (B|η = y) = f (x|y)dµ(x).
B
Then the density f (x|y) is called the conditional density of ξ subject to the
condition that η = y.
Theorem 10.10. ([Borovkov1998, Theorem 2, §20, p. 108]) If the joint
distribution of ξ and η in X × Y has density function f (x, y) w.r.t. the
product of measures µ ∈ P(X) and λ ∈ P(Y) then the function
Z
f (x, y)
f (x|y) := where q(y) = f (x, y)dµ(x)
q(y)
is the conditional density of ξ given η = y and the function q(y) is the
density of η w.r.t. the measure λ.
10.2.5. Conditional measures and transitional measures. Regular conditional
measures in Definition 10.7 are examples of transitions measures for which
we shall have a generalized version of Fubini theorem (Theorem 10.12)
Definition 10.11. ([Bogachev2007, Definition 10.7.1, p. 384]) Let (X1 , B1 )
and (X , B2 ) be a pair of measurable spaces. A transition measure for this
pair is a function P (·|·) : X1 × B2 → R with the following properties:
(i) for every fixed x ∈ X1 the function B 7→ P (x|B) is a measure on B2 ;
(ii) for every fixed B ∈ B2 the function x 7→ P (x|B) is measurable w.r.t.
B1 .
In the case where transition measures are probabilities in the second ar-
gument, they are called transition probabilities.
Theorem 10.12. Let P (·|·) be a transition probability for spaces (X1 , B1 )
and (X2 , B2 ) and let ν be a probability measure on B1 . Then there exists a
unique probability measure µ on (X1 × X2 , B1 ⊗ B2 ) with
Z
(10.12) µ(B1 × B2 ) = P (x|B2 )dν(x) for all B1 ∈ B1 , B2 ∈ B2 .
B1
17
Borovkov did not use the terminology “regular conditional probability”, instead he
use the terminology “conditional distribution: and “conditional density ”.
In addition, given any f ∈ L1 (µ) for ν- a.e. x1 ∈ X1 the function x2 7→

f (x1 , x2 ) on X2 is measurable w.r.t. the completed σ-algebra (B2 )P (x1 |·) and
P (x1 |·)-integrable, the function
Z
x1 7→ f (x1 , x2 )dP (x1 |x2 )
X2
is measurable w.r.t. (B1 )ν , and ν-integrable, and one has
Z Z Z
(10.13) f (x1 , x2 )dµ(x1 , x2 ) = f (x1 , x2 )dP (x1 |x2 )dν(x1 ).
X1 ×X2 X1 X2
11. Appendix 2: Concentration-of-measure inequalities

In probability theory, the concentration of measure is a property of a
large number of variables, such as in laws of large numbers. Concentration-
of-measure inequalities provide bounds on the probability that a random
variable X deviates from its mean, median or other typical value by a given
amount. Very roughly speaking, the concentration of measure phenomenon
can be stated in the following simple way: “A random variable that depends
in a smooth way on many independent random variables (but not too much
on any of them) is essentially constant”. For the proofs of concentration-of
measure inequalities in this Appendix we refer to [Lugosi2009]. I also recom-
mend [Ledoux2001, Shioya2016] for more advanced treatment the concentration-
of-measure theory.
11.1. Markov’s inequality. For any nonnegative random variable X :

(X , µ) → (R+ , dt), and any t > 0 we have
Eµ X
(11.1) µ{x ∈ X | X(x) > t} ≤ .
t
11.2. Hoeffding’s inequality. ([Hoeffding1963]) Let θ = (θ1 , · · · , θn ) be a
sequence of i.i.d. R-valued random variables on Z and µ ∈ P(Z). Assume
that Eµ (θi (z)) = θ̄ for all i and µ{z ∈ Z| [ai ≤ θi (z) ≤ b]} = 1. Then for
any ε > 0 we have
m
1 X −2mε2
µm {z ∈ Z m :

(11.2) θi (zi ) − θ̄ > ε} ≤ 2 exp ,
m (b − a)2
i=1
where z = (z1 , · · · , zm ).
11.3. Bernstein’s inequality. Let θ be a R-valued random variable on a
probability space (Z, µ) with the mean Eµ (θ) = θ̄ and variance σ 2 = Vµ (θ).
If |ξ − Eµ (ξ)| ≤ M then for all ε > 0 we have
m
m m
1 X −mε2
(11.3) µ {z ∈ Z : θi (zi ) − θ̄ > ε} ≤ 2 exp ),
m
i=1
2(σ 2 + 13 M ε
where z = (z1 , · · · , zm ).
MACHINE LEARNING 73
11.4. McDiarmid’s inequality. (or Bounded Differences or Hoeffding/Azuma

Inequality). Let X1 , · · · , Xm ∈ X are i.i.d. by a probability measure µ. As-
sume that f : X m → R satisfies the following property form some c > 0.
For all i ∈ [1, m] and for all x1 , · · · , xm , x0i ∈ X we have
|f (x1 , · · · , xm ) − f (x1 , · · · , xi−1 , x0i , xi+1 , · · · , xm )| ≤ c.
Then we have for all δ ∈ (0, 1)
r
m m ln(2/δ)
(11.4) µ {S ∈ X | |f (S) − Eµm f (S)| ≤ c } ≥ 1 − δ.
m
References
[Amari2016] S. Amari, Information Geometry, Springer, 2016.
[AJLS2018] N. Ay, J. Jost, H. V. Lê and L. Schwachhöfer, Parametrized measure
models, Bernoulli vol. 24 Nr 3 (2018), 1692-1725, arXiv:1510.07305.
[AJLS2015] N. Ay, J. Jost, H. V. Lê, and L. Schwachhöfer, Information geometry
and sufficient statistics, Probability Theory and related Fields, 162 (2015), 327-
364, arXiv:1207.6736.
[AJLS2017] N. Ay, J. Jost, H. V. Lê and L. Schwachhöfer, Information Geometry,
Springer, 2017.
[Bishop2006] C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
[Bogachev2007] V. I. Bogachev, Measure theory I, II, Springer, 2007.
[Borovkov1998] A. A. Borovkov, Mathematical statistics, Gordon and Breach Science
Publishers, 1998.
[CS2001] F. Cucker and S. Smale, On mathematical foundations of learning, Bulletin
of AMS, 39(2001), 1-49.
[Fisher1925] R. A. Fisher, Theory of statistical estimation, Proceedings of the Cambridge
Philosophical Society, 22(1925), 700-725.
[GBC2016] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT, 2016.
[Ghahramani2013] Z. Ghahramani, Bayesian nonparametrics and the probabilistic ap-
proach to modelling. Philosophical Transactions of the Royal Society A 371 (2013),
20110553.
[Graves2012] A. Graves, Supervised Sequence Labelling with Recurrent Neural Net-
works, Springer, 2012.
[JLS2017] J. Jost, H. V. Lê and L. Schwachhöfer, The Cramér-Rao inequality on
singular statistical models, arXiv:1703.09403.
[Halmos1950] P.R. Halmos, Measure theory, Van Nostrand 1950.
[Haykin2008] S. Haykin Neural Networks and Learning Machines, 2008.
[Hoeffding1963] W. Hoeffding, Probability inequalities for sums of bounded random
variables, J. Am. Stat. Assoc., 58(301):13-30, 1963.
[IH1981] I. A. Ibragimov and R. Z. Has’minskii, Statistical Estimation: Asymptotic
Theory, Springer, 1981.
[Janssen2003] A. Janssen, A nonparametric Cramér-Rao inequality, Statistics & Proba-
bility letters, 64(2003), 347-358.
[Jost2005] J. Jost, Postmodern Analysis, Springer, 2005.
[JP2003] J. Jacod and P. Protter, Probability Essentials, Springer, 2. edition, 2004.
[Kallenberg2002] O. Kallenberg, Foundations of modern Probability, Springer, 2002.
[Kullback1959] S. Kullback, Information theory and statistics, John Wiley and Sons,
1959.
[Ledoux2001] M. Ledoux, The concentration of measure phenomenon, AMS, 2001.
[Lugosi2009] G. Lugosi, Concentration of measure inequalities - lecture notes, 2009, avail-
able at http://www.econ.upf.edu/∼lugosi/anu.pdf.
[LC1998] E. L. Lehmann and G. Casella, Theory of Point Estimation, Springer, 1998.

[MRT2012] M. Mohri, A. Rostamizadeh, A. Talwalkar, Foundations of Machine
Learning, MIT Press, 2012.
[Murphy2012] K. P. Murphy, Machine Learning: A Probabilistic Perspective, MIT
Press, 2012.
[RN2010] S. J. Russell and P. Norvig, Artificial Intelligence A Modern Approach,
Prentice Hall, 2010.
[Shioya2016] T. Shioya, Metric Measure Geometry Gromov’s Theory of Convergence and
Concentration of Metrics and Measures, EMS, 2016.
[SSBD2014] S. Shalev-Shwart, and S. Ben-David, Understanding Machine Learning:
From Theory to Algorithms, Cambridge University Press, 2014.
[Sugiyama2016] M. Sugiyama, Introduction to Statistical Machine Learning, Elsevier,
2016.
[Tsybakov2009] A. B. Tsybakov, Introduction to Nonparametric Estimation, Springer,
2009.
[Valiant1984] L. Valiant, A theory of the learnable, Communications of the ACM, 27,
1984.
[Vapnik1998] V. Vapnik, Statistical learning theory, John Willey and Sons, 1998.
[Vapnik2000] V. Vapnik, The nature of statistical learning theory, Springer, 2000.
[Vapnik2006] V. Vapnik, Estimation of Dependences Based on Empirical Data, Springer,
2006.

Mathematical Foundations of Machine Learning: (NMAG 469, FALL TERM 2018-2019)

Uploaded by

Mathematical Foundations of Machine Learning: (NMAG 469, FALL TERM 2018-2019)

Uploaded by

MATHEMATICAL FOUNDATIONS OF MACHINE

Date: December 7, 2018.

5.3. Fundamental theorem of binary classification 39

Machine learning is an interdisciplinary field in the intersection of math-

What is the mathematical model of learning? How to quantify the diffi-

1. Learning, machine learning and artificial intelligence

1.1. Learning, inductive learning and machine learning. To start our

deductive learning. Of course we also learn inductively in school by search-

In mathematical theory of machine learning, or more general, in mathe-

which the desired function f should minimize. If the measurements gener-

learning problem is an interpolation problem. But in general one expects

1.2. A brief history of machine learning. Machine learning was born

Machine learning, reorganized as a separate field, started to flourish in

2according to Dhar, V. (2013). “Data science and prediction”. Communications of the

such tasks can be unbounded as in OCR, text classification, or speech recog-

ν << µ. By Radon-Nykodym theorem, see Appendix, Subsection 10.1, we

Main types of machine learning are supervised, unsupervised and rein-

1.4. Basic questions in mathematical foundations of machine learn-

1.5. Conclusion. Machine learning is automatized learning, whose perfor-

Finally I recommend some sources for further reading.

• F. Cucker and S. Smale, On mathematical foundations of learning,

2. Statistical models and frameworks for supervised learning

• A domain set X (also called an input space) consists of elements, whose

• An output space Y, also called a label set, consists of possible features

• A training data is a sequence S = {(x1 , y1 ), · · · , (xn , yn )} ∈ (X × Y)n of

• A hypothesis space H ⊂ Y X of possible predictors h : X → Y.

• The aim of a learner is to find a best prediction rule A that assigns a

find a rule, more precisely, an algorithm

where L : X × Y × H → R is an instantaneous loss function that measure

(2.3) L(x, y, f ) := d(y, f (x)).

Here d : Y × Y is a non-negative function that vanishes at the diagonal

• The main question of learning theory is to find necessary and sufficient

Remark 2.3. (1) In our discriminative model of supervised learning we

(3) The quasi-distance function d : Y × Y → R induces a quasi-distance

Thus it suffices to consider only Rµ (f ), if S is a sequence of i.i.d. observables.

coincides with the error of estimation in (1.1).

Let us compute the expected 0-1 loss function for a mapping f ∈ H = Y X

2.2. Generative model of supervised learning. In many cases a dis-

Digression. Conditional probability is one of most basic concepts in prob-

We may also wish to estimate the joint distribution µ := µX ×Y of i.i.d.

instantaneous loss function L, and hence a minimizing sequence {hi ∈ H}

i2 : RY → RX ×Y : [i2 (f )](x, y) := f (y).

RµL (h) = Eµ |Y − h(x)|2 = |i2 (Y ) − i1 (h)|2L2 (X ×R,µ) .

Remark 2.9. Generative models give us more complete information of the

under which the RHS of (2.16) for a sequence of hS , instead of h, converges

2.4. Conclusion. In this lecture we learn discriminative model of super-

further condition on H and L the ERM principle does not work.

3. Statistical models and frameworks for unsupervised

• In the parametric density estimation problem we assume that P ⊂ P(X )

where µu = pu µ0 . Given a data Sn = (x1 , · · · , xn ) ∈ X n , by (3.1), the

The Kullback-Leibler divergence, unlike the expected log-likelihood func-

where K0 (u) := 21 1(−1≤u≤1) . A simple generalization of the Rosenblatt

To measure the accuracy of the estimator p̂PSnR we use a trick, namely

we consider M SE(fˆP R , x0 ) of fˆSPnR w.r.t. a given point x0 ∈ R, averaging

The k-means objective function Gk is defined as follows

The k-means objective function G is used in digital communication tasks,

Remark 3.4. The above formulation of clustering is deterministic. We

3.3. Statistical models and frameworks for dimension reduction

To find the “best” transformation W = W(Sn ) we define an error function

Example 3.5. A popular linear method for dimension reduction is called

Remark 3.8. Modern autoencoders have generalized the idea of an encoder

4. Fisher metric and maximum likelihood estimator

where the supremum is taken over all finite partitions X = A1 ∪˙ . . . ∪A

By the Radon-Nikodym theorem, we may canonically identify S(X , µ0 )

kφ(x + h) − φ(x) − dx φ(h)kW

4.2. Fisher metric on a statistical model. Given a statistical model

(2) Assume that P is an almost 2-integrable statistical model. For each

(3) An almost 2-integrable statistical model P is called 2-integrable, if the

Since logθ (x) is continuously differentiable in θ up to order 3, we can apply

4.3. MSE and Cramér-Rao inequality. Set ϕl : l ◦ ϕ for any l ∈ (Rn )∗

Theorem 4.12 (The Cramér-Rao inequality). (cf. [AJLS2017, JLS2017])

Theorem 4.16. Let P be a 2-integrable statistical model parameterized by

4.5. Consistency of MLE. Assume that P1 = P ⊂ P(X ) is a 2-integrable

the notion of asymptotically accurate sequence of estimators σ̂k∗ : X k → P1

Under quite general conditions on the density functions of a statistical

4.6. Conclusion. In this lecture we derive a natural geometry on a statis-