Introduction to

Supervised Machine
Learning: Regression
MC. Luis Angel Galaviz Alejos
MS. LUIS ANGEL GALAVIZ ALEJOS
Introduction
Artificial Intelligence
Data Science end to end
Get data Build Model Deploy Model
Machine Learning Types
Supervised Unsupervised Reinforced

Classification

Regression
Setup
1. Install Python Anaconda
2. Open Spyder by your OS file explorer
Coding Warmup – Problem #1
Problem #1 Sum of elements of an array
We have an vector/array u with N elements. Write a python program
which finds the sum of all its elements.
𝑢 = [ 𝑢1 , 𝑢2 , … , 𝑢𝑖 , … , 𝑢𝑛−1 , 𝑢𝑁 ] 𝑆𝑢𝑚 1, 1, 1, 1 =4
𝑁
𝑆𝑢𝑚([2, −1, 3, 5, 0]) = 9
𝑆𝑢𝑚 𝑢 = ෍ 𝑢𝑖
𝑖=1
𝑁 𝑆𝑢𝑚 3 =3
𝑆𝑢𝑚 𝑢 = 𝑢1 + ෍ 𝑢𝑖
𝑖=2
𝑆𝑢𝑚([]) = 0
𝑁

𝑆𝑢𝑚 𝑢 = 𝑢1 + 𝑢2 + ෍ 𝑢𝑖
𝑖=3

𝑆𝑢𝑚 𝑢 = 𝑢1 + 𝑢2 + … + 𝑢𝑖 + … + 𝑢𝑁−1 + 𝑢𝑁
Coding Warmup – Compare with numpy
Time Spent (ms)
N
Sum Numpy Sum *
10,000 1.037 0.0
50,000 3.989 0.0
100,000 7.933 0.0
200,000 14.962 0.0
300,000 21.936 0.0
400,000 29.924 0.0
* Please be aware that the fact that numpy spent 0 ms means that python’s time package do not has
enough resolution to be able to measure numpy’s sum elapsed time
Coding Warmup – Measuring Error
Problem #2 Measuring Error
We were doing an experiment on our electric engineering lab. According to our
Ohm’s law calculations the voltage should be 100 volts. We assume that our
calculated value is correct. We measured the voltage 10 times with two distinct
voltmeters. Which voltmeter is better?
# Red Yellow
1 104 100
2 103 106
3 97 95
4 103 104
5 97 100
6 102 94
7 98 101
8 98 100
9 100 99
10 101 100
Coding Warmup – Measuring Error
∆𝑦 = 𝑦𝑚𝑒𝑎𝑠𝑢𝑟𝑒𝑑 − 𝑦𝑐𝑎𝑙𝑐𝑢𝑙𝑎𝑡𝑒𝑑 𝑦𝑐𝑎𝑙𝑐𝑢𝑙𝑎𝑡𝑒𝑑 = 100 𝑣𝑜𝑙𝑡𝑠

# Red ∆𝒚𝟐 ∆𝒚 # Yellow ∆𝒚𝟐 ∆𝒚

1 104 16 4 1 100 0 0
2 103 9 3 2 106 36 6
3 97 9 3 3 95 25 5
4 103 9 3 4 104 16 4
5 97 9 3 5 100 0 0
6 102 4 2 6 94 36 4
7 98 4 2 7 101 1 1
8 98 4 2 8 100 0 0
9 100 0 0 9 99 1 1
10 101 1 1 10 100 0 0
Total 65 23 Total 115 23
Linear Regression
Linear Regression: The Line Equation

A linear equation is represented as 𝑓 𝑥 = 𝑎𝑥 + 𝑏

were:

𝑎 = 𝐿𝑖𝑛𝑒 𝑠𝑙𝑜𝑝𝑒
𝑏 = 𝐿𝑖𝑛𝑒 𝑖𝑛𝑡𝑒𝑟𝑐𝑒𝑝𝑡
𝑥 = 𝐼𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡 𝑣𝑎𝑟𝑖𝑎𝑏𝑙𝑒
𝑓 𝑥 = 𝑦 = 𝐷𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡 𝑣𝑎𝑟𝑖𝑎𝑏𝑙𝑒

Its easy to find 𝑎 and 𝑏 values if:

• We have two points or more of the given line
• We have one point and the line slope ‘a’

https://en.wikipedia.org/wiki/Linear_equation
Linear Regression: A Noisy Line
Imagine we have a real world process. Assume
that we are sure the process could be described by
a linear model. However, given measurement
errors we cannot observe 𝒇 𝒙 real values,
instead we observe it with an additional error:

𝑓 𝑥 = 𝑎𝑥 + 𝑏 + 𝐸𝑟𝑟𝑜𝑟

If Error is Gaussian distributed

𝑓 𝑥 = 𝑎𝑥 + 𝑏 + 𝑁(𝑚𝑒𝑎𝑛, 𝑣𝑎𝑟)

We are searching for 𝒇𝒑𝒓𝒆𝒅 (𝒙) which minimizes an

error measure
Linear Regression – Evaluating Error Functions
Mean Squared Error(𝑴𝑺𝑬) Mean Absolute Error (𝑴𝑨𝑬)
𝟏 𝑵 𝟏 𝑵
𝑴𝑺𝑬 = σ (𝒚 ෝ𝒊 )𝟐
−𝒚 𝑴𝑨𝑬 = σ ෝ𝒊
𝒚𝒊 − 𝒚
𝑵 𝒊=𝟏 𝒊 𝑵 𝒊=𝟏

# X 𝒇(𝒙) 𝒇𝒑𝒓𝒆𝒅 𝒙 ∆𝒚 ∆𝒚𝟐 ∆𝒚

1 0 0.442 0.8 -0.358 0.128 0.358
2 1 1.098 1.7 -0.602 0.362 0.602
3 2 2.179 2.6 -0.421 0.177 0.421
4 3 1.828 3.5 -1.672 2.794 1.672
5 4 3.458 4.4 -0.942 0.888 0.942
6 5 5.280 5.3 -0.020 0.000 0.020
7 6 6.470 6.2 0.270 0.073 0.270
8 7 6.511 7.1 -0.589 0.347 0.589
9 8 8.252 8 0.252 0.063 0.252
10 9 9.203 8.9 0.303 0.092 0.303
Total / N 0.493 0.543
Linear Regression – Optimization Surfaces
Mean Squared Error (𝑴𝑺𝑬) Mean Absolute Error (MAE)
𝟏 𝟏
𝑴𝑺𝑬 = σ𝑵 (𝒚
𝒊=𝟏 𝒊 − ෝ
𝒚 𝒊 ) 𝟐 𝑴𝑨𝑬 = σ𝑵 ෝ𝒊
𝒊=𝟏 𝒚𝒊 − 𝒚
𝑵 𝑵
Linear Regression – Optimization Contours
Mean Absolute Error (MAE) Mean Squared Error (𝑴𝑺𝑬)
𝟏 𝑵 𝟏
𝑴𝑨𝑬 = σ ෝ𝒊
𝒚𝒊 − 𝒚 𝑴𝑺𝑬 = σ𝑵 ෝ𝒊 )𝟐
𝒊=𝟏(𝒚𝒊 − 𝒚
𝑵 𝒊=𝟏 𝑵
Linear Regression: Graphical intuition

https://giphy.com/gifs/gradient-O9rcZVmRcEGqI/fullscreen
 Linear Regression – Least Squares Regression
We should minimize Residuals Squared Error
SE = σ𝑵 ෝ𝒊 )𝟐
𝒊=𝟏(𝒚𝒊 − 𝒚

Recall that
ෝ𝒊 = 𝒂𝒙𝒊 + 𝒃
𝒚
Substituting 𝑦ො𝑖
𝑵

𝒎𝒊𝒏𝒂,𝒃 (෍ 𝒚𝒊 − (𝒂𝒙𝒊 + 𝒃) 𝟐 )
𝒊=𝟏

We calculate MSE derivative. We have two variables 𝒂 and 𝒃, hence, two derivatives

𝝏(𝑺𝑬) 𝝏 𝟐
= 𝝏𝒂 σ𝑵
𝒊=𝟏 𝒚𝒊 − 𝒂𝒙𝒊 + 𝒃 ) = σ𝑵
𝒊=𝟏 𝟐(𝒚𝒊 − 𝒂𝒙𝒊 + 𝒃 )(−𝒙𝒊 )
𝝏𝒂

𝝏(𝑺𝑬) 𝝏 𝟐
= σ𝑵
𝒊=𝟏 𝒚𝒊 − 𝒂𝒙𝒊 + 𝒃 ) = σ𝑵
𝒊=𝟏 𝟐(𝒚𝒊 − 𝒂𝒙𝒊 + 𝒃 )(−𝟏)
𝝏𝒃 𝝏𝒃
Linear Regression – LSR Derivatives
𝝏(𝑴𝑺𝑬) 𝝏(𝑴𝑺𝑬)
In order to find the minimum, we set and to zero
𝝏𝒂 𝝏𝒂
𝝏(𝑴𝑺𝑬)
= 𝟎 = σ𝑵 𝑵 𝟐
𝒊=𝟏 𝟐(𝒚𝒊 − 𝒂𝒙𝒊 + 𝒃 )(−𝒙𝒊 ) = σ𝒊=𝟏 𝟐(−𝒙𝒊 𝒚𝒊 + 𝒂𝒙𝒊 + 𝒃𝒙𝒊 )
𝝏𝒂

− σ 𝒙𝒊 𝒚𝒊 + σ 𝒂𝒙𝟐𝒊 + σ 𝒃𝒙𝒊 = 𝟎

𝒂 σ 𝒙𝟐𝒊 + 𝒃 σ 𝒙𝒊 = σ 𝒙𝒊 𝒚𝒊

𝟏) 𝐚 σ 𝒙𝟐𝒊 + 𝒃 σ 𝒙𝒊 = σ 𝒙𝒊 𝒚𝒊

𝝏(𝑴𝑺𝑬)
= 𝟎 = σ𝑵
𝒊=𝟏 𝟐(𝒚𝒊 − 𝒂𝒙𝒊 + 𝒃 )(−𝟏) = σ 𝟐(− 𝒚𝒊 + 𝒂𝒙𝒊 + 𝒃)
𝝏𝒃

− ෍ 𝒚𝒊 + ෍ 𝒂𝒙𝒊 + ෍ 𝒃𝒙𝒊 = 𝟎

σ 𝒂𝒙𝒊 + σ 𝒃 = σ 𝒚𝒊

𝟐) 𝒂 ෍ 𝒙𝒊 + 𝒃𝑵 = ෍ 𝒚𝒊
Linear Regression – LSR Solve
We solve Eq 1 and Eq 2 Simultaneously

𝟏) 𝒂 σ 𝒙𝟐𝒊 + 𝒃 σ 𝒙𝒊 = σ 𝒙𝒊 𝒚𝒊
𝐂 = ෍ 𝒙𝟐𝒊 𝐃 = ෍ 𝒙𝒊 𝐄 = ෍ 𝒙𝒊 𝒚𝒊 𝐅 = ෍ 𝒚𝒊
𝟐) 𝒂 ෍ 𝒙𝒊 + 𝒃𝑵 = ෍ 𝒚𝒊

We multiply coefficient multiply both equations to resolve them

𝑏𝐷2 − 𝑏𝑁𝐶 = 𝐷𝐸 − 𝐶𝐹
𝑏 𝐷2 − 𝑁𝐶 = 𝐷𝐸 − 𝐶𝐹
𝟏)𝒂𝑪 + 𝒃𝑫 = 𝑬 𝟏)𝑫[𝒂𝑪 + 𝒃𝑫 = 𝑬] 𝐷𝐸 − 𝐶𝐹
𝑏=
𝟐) 𝒂𝑫 + 𝒃𝑵 = 𝑭 𝟐) − 𝑪[𝒂𝑫 + 𝒃𝑵 = 𝑭] 𝐷2 − 𝑁𝐶

σ 𝒙𝒊 σ 𝒙𝒊 𝒚𝒊 − σ 𝒙𝟐𝒊 σ 𝒚𝒊
𝑏=
σ 𝒙𝒊 σ 𝒙𝒊 − 𝑵 σ 𝒙𝟐𝒊

By a similar method you can derive 𝒂 solution. Try it as an exercise!

Linear Regression – LSR Solution
The solution for 𝒂 and 𝒃

𝑵 σ 𝒙 𝒊 𝒚 𝒊 − σ 𝒙 𝒊 σ 𝒚𝒊 σ 𝒙𝟐𝒊 σ 𝒚𝒊 − σ 𝒙𝒊 σ 𝒙𝒊 𝒚𝒊
𝒂= 𝟐 𝒃= 𝟐
𝑵 σ 𝒙𝟐𝒊 − (σ 𝒙𝒊 ) 𝑵 σ 𝒙𝟐𝒊 − (σ 𝒙𝒊 )

Recall that

𝟏 𝟏 𝟏 𝟏
෍ 𝒙𝒊 𝒚𝒊 = 𝒙𝒚 ഥ
෍ 𝒙𝒊 = 𝒙 ഥ
෍ 𝒚𝒊 = 𝒚 ෍ 𝒙𝟐𝒊 = 𝒙𝟐
𝑵 𝑵 𝑵 𝑵
1
ൗ 2
We multiply both solutions by 1 𝑁
ൗ𝑁2

ഥ𝒚
𝒙𝒚 − 𝒙 ഥ 𝒙𝟐 𝒚
ഥ−𝒙
ഥ 𝒙𝒚
𝒂= 𝒃=
𝒙𝟐 − 𝒙
ഥ𝟐 𝒙𝟐 − 𝒙
ഥ𝟐
Linear Regression – Problem #1
Learning 𝒇(𝒙) = 𝟐𝒙 + 𝟏 + 𝑵(𝟎, 𝟐)
Linear Regression – Problem #2
Learning 𝒇(𝒙) = 𝑺𝒊𝒏(𝒙) + 𝑵(𝟎, 𝟎. 𝟏)
K Nearest Neighbors
(KNN)
KNN – Graphical Intuition

https://www.youtube.com/watch?v=3lp5CmSwrHI
KNN – Graphical Intuition

https://www.youtube.com/watch?v=3lp5CmSwrHI
KNN – Simulated Data

https://www.youtube.com/watch?v=3lp5CmSwrHI