AI701: Foundations of Artificial Intelligence

Week 2
Agenda

• Introduction to linear regression

• Logistic regression fundamentals

• Gradient descent (batch/stochastic)

• Underfitting and overfitting

• Introduction to regularization
Linear Regression
The discovery of Ceres
1801: Astronomer Piazzi discovered Ceres
Made 19 observations of location before it was obscured by the sun

Time Right ascension Declination

Jan 01,20:43:17.8 50.91 15.24
Jan 02,20:39:04.6 50.84 15.30
... ... ...
Feb 11, 18:11:58.2 53.51 18.43

Where and when it will be observed again?

Gauss's triumph
September 1801: Gauss took Piazzi’s data and created a model of
Ceres’s orbit
Makes prediction

December 7, 1801: Ceres located within 1/2 degree of Gauss’s prediction,

much more accurate than other astronomers
Method: Least squares linear regression
Linear regression framework

Design decisions:

Which predictors are possible? hypothesis class

How good is a predictor? Loss function
How do we compute the best predictor? Optimization algorithm
Hypothesis class: which predictors?
4

3
f (x) = 1 + 0.57x
2

y
f (x) = 2 + 0.2x
1
f (x) = w1 + w2x
0
0 1 2 3 4 5

Vector notation:
weight vector w = [w1, w2] feature extractor φ(x) = [1, x] feature vector

f w (x) = w ·φ(x) score

f w (3) = [1, 0.57] ·[1, 3] = 2.71

Hypothesis class:
F = { f w : w ∈R2}
Loss function: how good is apredictor?
4

training data Dtrain 3

f w (x) = w ·φ(x) x y 2

y
w = [1, 0.57] 1 1 residual
φ(x) = [1, x] 1
2 3
4 3 0
0 1 2 3 4 5

Loss(x, y, w) = (f w (x) − y)2 squared loss

Loss(1, 1, [1, 0.57]) = ([1, 0.57] ·[1, 1] − 1)2 = 0.32
Loss(2, 3, [1, 0.57]) = ([1, 0.57] ·[1, 2] − 3)2 = 0.74
Loss(4, 3, [1, 0.57]) = ([1, 0.57] ·[1, 4] − 3)2 = 0.08

1
TrainLoss(w) = |D train| Σ(x,y)∈Dtrain Loss(x, y,w)
TrainLoss([1, 0.57]) = 0.38
Visualizing Loss Function
w
1 2
TrainLoss(w) = |D | Σ ( f w (x) − y) min TrainLoss(w)
train (x,y)∈D train
Gradient Descent
Optimization algorithm: how to compute best?
Goal: min w TrainLoss(w)

Definition: gradient

The gradient ∇w TrainLoss(w) is the direction that increases the

training loss the most.
Computing the gradient
Objective function:

Gradient (use chain rule):

Gradient descent example
Linear Classification
Linear classification framework
3

2
training data decision boundary
[2, 0] input 1
x1 x2 y

x2
example 0
0 2 1 learning algorithm
example
f classifier
-1
-2 0 1
example
1 -1 -1 -2
-1 label -3
-3 -2 -1 0 1 2 3

x1

Design decisions:
Which classifiers are possible? hypothesisclass
How good is a classifier? loss function
How do we compute the best classifier? optimization algorithm
An example linear classifier
3

x2
-1

-2

-3
-3 -2 -1 0 1 2 3

x1
f ([0, 2]) = sign([−0.6, 0.6] ·[0, 2]) = sign(1.2) = 1
f ([−2, 0]) = sign([−0.6, 0.6] ·[−2, 0]) = sign(1.2) = 1
f ([1, −1]) = sign([−0.6, 0.6] ·[1, −1]) = sign(−1.2) = −1

Decision boundary: x such that w ·φ(x) = 0

Hypothesis class: which classifiers?
3

1
φ(x) = [x1, x2]

x2
0
f (x) = sign([−0.6, 0.6] ·φ(x)) -1

f (x) = sign([0.5, 1] ·φ(x)) -2

-3
-3 -2 -1 0 1 2 3

x1

General binary classifier:

f w (x) = sign(w ·φ(x))
Hypothesis class:
F = { f w : w ∈R2}
Loss function: how good is aclassifier?
3

2
training data Dtrain
1

f w (x) = w ·φ(x) x1 x2 y

x2
0
w = [0.5, 1] 0 2 1 -1
φ(x) = [x1, x2]
-2 0 1
-2
1 -1 -1
-3
-3 -2 -1 0 1 2 3

x1

Loss0-1(x, y, w) = 1[f w (x) y] zero-one loss

Loss([0, 2], 1, [0.5, 1]) = 1[sign([0.5, 1] ·[0, 2]) ƒ= 1] = 0
Loss([−2, 0], 1, [0.5, 1]) = 1[sign([0.5, 1] ·[−2, 0]) ƒ= 1] = 1
Loss([1, −1], −1, [0.5, 1]) = 1[sign([0.5, 1] ·[1, −1]) ƒ= −1] = 0

TrainLoss([0.5, 1]) = 0.33

Score and margin
3

Predicted label: f w (x) = sign(w ·φ(x))

x2
0

-1
Target label: y
-2

-3
-3 -2 -1 0 1 2 3

x1
Definition: score

The score on an example (x, y) is w·φ(x), how confident we are in

predicting +1.

Definition: margin

The margin on an example (x, y) is (w ·φ(x))y, how correct we are.

Zero-one loss rewritten
4

Loss(x,y, w)
3

0
-3 -2 -1 0 1 2 3

margin (w ·φ(x))y
Optimization algorithm: how to compute best?
Goal: min w TrainLoss(w)
To run gradient descent, compute the gradient:
∇w TrainLoss(w) = 1
|D train| Σ (x,y)∈D train ∇Loss0-1(x, y,w)

∇w Loss0-1(x, y, w) = ∇1[(w ·φ(x))y ≤ 0]

4
Loss(x,y, w)

2
Gradient is zero almost everywhere!
1

0
-3 -2 -1 0 1 2 3

margin (w ·φ(x))y
Hinge loss
4

Loss(x,y, w)
3
Loss0-1
2
Losshinge
1

0
-3 -2 -1 0 1 2 3

margin (w ·φ(x))y

Losshinge(x, y, w) = max{1 − (w ·φ(x))y, 0}

Another: logistic regression
Losslogistic(x, y, w) = log(1 + e−(w·φ(x))y )
4

Loss(x, y,w)
3
Loss0-1
2 Losshinge

1
Losslogistic

0
-3 -2 -1 0 1 2 3

margin (w ·φ(x))y
Intuition: Try to increase margin even when it already exceeds
1
Gradient of the hinge loss
4

Loss(x,y, w)
3

2 Losshinge
1

0
-3 -2 -1 0 1 2 3

margin (w ·φ(x))y
Hinge loss on training data
3

2
training data Dtrain
1

f w (x) = w ·φ(x) x1 x2 y

x2
0
w = [0.5, 1] 0 2 1 -1
φ(x) = [x1, x2]
-2 0 1
-2
1 -1 -1
-3
-3 -2 -1 0 1 2 3

x1

Losshinge(x, y, w) = max{1 − (w ·φ(x))y, 0}

Loss([0, 2], 1, [0.5, 1]) = max{1 − [0.5, 1] ·[0, 2](1), 0} = 0 ∇Loss([0, 2], 1, [0.5, 1]) = [0, 0]
Loss([−2, 0], 1, [0.5, 1]) = max{1 − [0.5, 1] ·[−2, 0](1), 0} = 2 ∇Loss([−2, 0], 1, [0.5, 1]) = [2, 0]
Loss([1, −1], −1, [0.5, 1]) = max{1 − [0.5, 1] ·[1, −1](−1), 0} = 0.5 ∇Loss([1, −1], −1, [0.5, 1]) = [1, −1]
TrainLoss([0.5, 1]) = 0.83 ∇TrainLoss([0.5, 1]) = [1, −0.33]
Regression vs Classification

Regression Classification

Prediction f w (x) score sign(score)

Relate to target y residual (score − y) margin (score y)
zero-one
squared
Loss functions hinge
absolute deviation
logistic
Algorithm gradient descent gradient descent
Stochastic Gradient Descent
Gradient descent is slow

Algorithm: gradient descent

Initialize w = [0, . . . , 0]
For t = 1, . . . , T :
w ← w − η∇w TrainLoss(w)

Problem: each iteration requires going over all training examples

Expensive when have lots of data!

Stochastic gradient descent

Algorithm: stochastic gradient descent

Initialize w = [0, . . . , 0]
For t = 1, . . . , T :
For (x, y) ∈ Dtrain:
w ← w − η∇wLoss(x, y, w)
Step size

Question: what should η be?

0 1
η
conservative, more stable aggressive, faster

Strategies:
• Constant: η = 0.1
• Decreasing:
GD vs SGD

gradient descent stochastic gradient descent

Key idea: stochastic updates

It’s not about quality, it’s about quantity.

Overfitting and Regularization
Minimizing the training loss
Hypothesis class:
f w (x) = w ·φ(x)
Training objective (loss function):
1
TrainLoss(w) = Σ Loss(x, y,w)
|Dtrain| (x,y)∈D
train

Optimization algorithm:
stochastic gradient descent

Is the training loss a good objective to optimize?

Rote Learning
Algorithm: rote learning

Training: just store Dtrain .

Predictor f (x):
If (x, y) ∈ Dtrain : return y.
Else: segfault.

Minimizes the objective perfectly (zero), but clearly bad...

Overfitting scenarios

Classification Regression
Overfitting
Overfitting – Possible reasons

• Too few training data

• Noise in the data

• The hypothesis space is too large

• The input space is high-dimensional

Overfitting vs. model complexity
• We talk of overfitting when High bias Low bias

Error
Low variance High variance
decreasing 𝐸𝑖𝑛 leads to
increasing 𝐸𝑜 𝑢 𝑡

• Major source of failure for

machine learning systems Out of sample error

𝐸𝑜𝑢𝑡
• Overfitting leads to bad
generalization
In sample error Overfitting
• A model can exhibit bad 𝐸𝑖𝑛
generalization even if it does not
overfit
Low Model complexity High
Overfitting vs. model complexity
Evaluation
Dtrain learning algorithm f

How good is the predictor f ?

Key idea: the real learning objective

Our goal is to minimize error on unseen future examples.

Don’t have unseen examples; next best thing:

Definition: test set

Test set Dtest contains examples not used for training.

Generalization
When will a learning algorithm generalizewell?

Dtrain Dtest
Approximation & estimation error
All predictors
F
Learning
f ∗ approx. error est. error
g fˆ

• Approximation error: how good is the hypothesis class?

• Estimation error: how good is the learned predictor relative to the potential of the
hypothesis class?
Effect of hypotheses class
All predictors
F
Learning
f∗ approx. error est. error
g fˆ

As the hypothesis class size increases...

Approximation error decreases because:
taking min over larger set
Estimation error increases because:

harder to estimate something more complex

How do we control the hypothesis class size?

Cure 1: Dimensionality
w ∈ Rd

Reduce the dimensionality d (number of features):

Controlling the dimensionality
Manual feature (template) selection:
• Add feature templates if they help
Remove feature templates if they don’t help

Automatic feature selection (beyond the scope of this class):

• Forward selection
• Boosting
• L 1 regularization

It’s the number of features that matters

Cure 2: Norm
Controlling the Norm
Regularized objective:
λ
min TrainLoss(w)+ ǁwǁ 2
w 2

Algorithm: gradient descent

Initialize w = [0, . . . , 0]
For t = 1, . . . , T :
w ← w − η(∇w TrainLoss(w)+λw)

Same as gradient descent, except shrink the weights towards zero by λ.

Controlling the Norm: Early Stopping
Algorithm: gradient descent

Initialize w = [0, . . . , 0]
For t = 1, . . . , T :
w ← w − η∇w TrainLoss(w)

Idea: simply make T smaller

Intuition: if have fewer updates, then ǁwǁ can’t get too big.

Lesson: try to minimize the training error, but don’t try too hard.
Regularization and bias-variance
The effects of the regularization procedure can be observed in the bias and variance
terms

• Regularization trades bias in order to considerably decrease the variance of the model

• Regularization strives for smoother hypothesis, thus reducing the opportunities to overfit

• The amount of regularization 𝜆 has to be chosen specifically for each type of regularizer

• Usually 𝜆 is chosen by cross-validation

How overfitting affects predictions

Predictive Underfitting Overfitting

Error

Error on Test Data

Error on Training Data

Model Complexity

Ideal Range
for Model Complexity
Regularization
• A method for automatically controlling the complexity of the learned
hypothesis

• Idea: penalize for largevaluesof thertaj

oIncorporate penalty into the cost function
oWorks well when we have a lot of features, each that contributes a bit to
predicting the label

• Can also address overfitting by eliminating features (either manually or via model
selection)
L2 Regularization
• Regularized linear regression objectivefunction:

model fit to data regularization

o λis the regularization parameter (λ
oNo regularization on ! 0!
Slide Credits

Percy Liang
Dorsa Sadigh
Mirko Mazzoleni
Ryan P. Adams
 Thank You

Mohamed bin Zayed

University of Artificial Intelligence
Masdar City
Abu Dhabi
United Arab Emirates

mbzuai.ac.ae