Linear

Regression
Notes
Linear Regression
Below, types of Linear Regression.

1. Simple Linear Regression

2. Multiple Linear Regression

3. Polynomial Linear Regression

4. Regularisation Linear Regression

* Regression Metrics

* Gradient Descent

* Bias Variance Trade-O

ff
1. Simple Linear Regression
Simple linear regression is a statistical method that
models the linear relationship between two variables
using a straight line equation.

Intuition

y = m x + b (m is slop and b is intercept)

Now, for nding a value of m and b we have two methods.

1. Closed form (Ordinary List Squared)

For info in scikitlearn LinearRegression() class.

2. Non closed form (Gradient Descent)

For info in scikitlearn SGDRegressor() class.

fi
1. Closed form (Ordinary List Squared)
Formula y = m x + b

Now, we will see how we can get m and b.

b = ȳ − m x̄ (x̄ is mean of x-axis values, ȳ is mean of y-

axis values and m is below)
n
∑i=1 (x i − x̄ )(yi − ȳ)
m= n
∑i=1 (x i − x̄ )2
2. Non closed form (Gradient Descent)
* It has been discussed in Gradient Descent discussion
which is below the Simple and Multiple Linear
Regression's discussion.
Regression Metrics
They provide a quantitative assessment of how well the
regression model is performing in terms of predicting the
dependent variable.

1. Mean Squared Error (MSE)

2. Root Mean Squared Error (RMSE)

3. Mean Absolute Error (MAE)

4. R-squared (Coe cient of Determination)

5. Adjusted R-squared
ffi
1. Mean Absolute Error (MAE)
The Mean Absolute Error (MAE) is a metric used to
measure the average absolute di erence between the
observed values and the values predicted by a model.
1 n
yi − ^yi
n∑
MAE =
i=1

Here, y is actual data point, ^y is predicted.

Advantages

Interpretability and Robustness to Outliers.

Disadvantages

Sensitivity to Outliers and Equal weight to all errors.

ff
2. Mean Squared Error (MSE)
The Mean Absolute Error (MAE) is a common metric used
to measure the average squared di erence between
predicted and actual values in regression analysis.
1 n
( )
2
^yi
n∑
MSE = yi −
i=1

Here, y is actual data point, ^y is predicted.

Advantages

Sensitivity Error and Mathematically Simplicity.

Disadvantages

Sensitive to outliers and Units of measurement.

ff
3. Root Mean Squared Error (RMSE)
It is derived from the Mean Squared Error (MSE) but
provides a measure of error.

1 n
( )
2
^yi
n∑
RMSE = yi −
i=1

Here, y is actual data point, ^y is predicted.

 4. R-squared (Coe cient of
Determination)
The R-squared score, also known as the coe cient of
determination, is a metric used to evaluate the goodness
of t of a regression model.

∑i=1 (yi − ^yi)

2
n

R2 = 1 −
∑i=1 (yi − y i)
n − 2

−
Here, y is actual data point, ^y is predicted. y is the mean
of the observed values.
fi
ffi
ffi
 5. Adjusted R-squared
The Adjusted R-squared (Adjusted R-squared) is a
modi ed version of the R-squared score that adjusts for
the number of predictors in a regression model. It
penalises the inclusion of irrelevant predictors that do
not improve the model's explanatory power.

(1 − R )(n − 1)
2
R2 = 1 −
n−k −1
Here, n is numbers of data points, k is the number of
independent variables in data point and R 2 is for
coe cient for determination.

Advantages

Penalty for Additional Predictors and Range of values.

ffi
fi
 2. Multiple Linear Regression
Multiple linear regression is an extension of simple linear
regression, allowing for the modelling of relationships
between a dependent variable and multiple independent
variables.

Equation

y = b0 + b1 x1 + b2 x 2 + … + bn xn + ε

Formula

y = Xb + ε

Here, y is the dependent variable, X is the matrix of

independent variables, b is the vector of coe cients
(including the intercept), ε is the error term.

Let’s see, how we can nd ε and b by formulas.

ε = (y − ^y)

Here y is actual or observed values and ^y is predicted

values of the target variable. Even ^y = Xb where X is the
matrix of independent variables, b is the vector of
coe cients (including the intercept).

b = (X T X )−1X T y

Here X is input matrix, b is vector of coe cients and

intercept and y is observed target value. It a
di erentiation of error function result.
ff
ffi
fi
ffi
ffi
Gradient Descent
Gradient descent is an iterative optimisation algorithm
used for nding the minimum of a function. The primary
goal of gradient descent is to minimise a cost function or
loss function. This function measures the di erence
between the model's predictions and the actual target
values.

How it works ?

Step 1: Start with initial values for the model parameters.

Step 2: In each iteration, the algorithm updates the

parameters in the direction opposite to the gradient of
the cost function.

Step 3: The process continues until the algorithm

converges to a minimum or a predetermined number of
iterations is reached.

What is Gradient ?

The gradient represents the direction of the steepest

ascent in the cost function.

How we will reach to minimum ? For that Learning Rate.

The learning rate determines the size of the steps taken

during each iteration. Choose learning rate according to
condition for less iteration.

When Iterations will stop ?

Here, we have two methods to do this thing.

fi
ff
1. Epochs

2. Until 0 is reached.

Below, the types of Gradient Descent.

1. Batch: Updates are made in each iteration.

2. Stochastic: Updates are made choosing any data point

at a time.

3. Mini-Batch: Updates are made using a small batch of

data points.
 1. Batch Gradient Discent

Updates are made in each iteration. For that we have to

nd b and m every iteration. One thing that iteration is
based on epoch or until we reach at 0. So we will nd b
and m like this. This is useful when we have convex
function or small dataset.

For Simple Linear Regression.

( ∂b )
∂J
bnew = bold − α

Here bnew is next iteration, bold is last iteration and α is

∂J
Learning Rate and is the partial derivative of the cost
∂b ∂J
function with respect to the intercept. would be
n ∂b
(y − ^yi ) here ^yi would be m X + b
∑
−2
i=1

( ∂m )
∂J
m new = mold − α

Here m new is next iteration, mold is last iteration and α is

∂J
Learning Rate and is the partial derivative of the
∂m ∂J
cost function with respect to the intercept. would be
n ∂m
(y − ^yi ) × X here ^yi would be m X + b
∑
−2
i=1

y is the dependent variable which the output we want to

predict.

x is the independent variable which is the input feature.

m is the slope of the line.

b is the intercept.
fi
fi
 For Multiple Linear Regression.

( ∂b )
∂J
bnew = bold − α

Here bnew is next iteration, bold is last iteration and α is

∂J
Learning Rate and is the partial derivative of the cost
∂b ∂J
function with respect to the intercept. would be
∂b
m

∑(
−2 yi − ^yi)
i=1

Here, ^yi = Xi ⋅ β + β 0 where Xi represents the feature

vector for the i-th data point, β is the vector of
coe cients, and β 0 is the intercept.

( ∂m )
∂J
m new = mold − α

Here m new is next iteration, mold is last iteration and α is

∂J
Learning Rate and is the partial derivative of the
∂m ∂J
cost function with respect to the intercept. would be
∂m
−2 m
(yi − yi) × X
^
m ∑i=1

Here, ^yi = Xi ⋅ β + β 0 where Xi represents the feature

vector for the i-th data point, β is the vector of
coe cients, and β 0 is the intercept.
ffi
ffi
2. Stochastic Gradient Discent

In SGD, instead of using the entire training dataset to

compute the gradient of the cost function, only one
randomly chosen training example is used at a time.

The parameters (weights and biases) are updated a er

each individual data point.

SGD introduces more randomness into the optimisation

process, which can sometimes help escape local minima
but may also introduce more noise.

Formula

It is same as Batch Gradient Descent but the small

changes is we would take one random row of independent
variable X instead of taking whole independent variable X
f
t
3. Mini-Batch Gradient Discent

Mini-Batch GD strikes a balance between the e ciency

of batch GD and the stochastic nature of SGD.

It updates the parameters using a small random subset

or "mini-batch" of the training data.

The mini-batch size is a hyper-parameter that is

typically chosen based on the available computational
resources.

Formula

As you know it a combination of Batch and Stochastic so

we will use both formula in combine. We will nd the
random value times the value of the epochs.
fi
ffi
 3. Polynomial Linear Regression
Instead of ing a straight line (as in linear regression),
polynomial regression uses a polynomial equation to
capture the non-linear pa erns in the data.

Equation

y = b 0 + b1 ⋅ x + b2 ⋅ x 2 + ⋅ ⋅ ⋅ + ε

y is the dependent variable (output). x is the independent

variable (input). b 0,b1,b2,...,bn are the coe cients of the
polynomial terms. n is the degree of the polynomial. ϵ
represents the error term.

The choice of the degree n determines the exibility of

the model. Higher degrees allow the model to capture
more complex pa erns but also increase the risk of
over ing, especially when the dataset is small.

In scikitlearn PolynomialFeatures() class. We use training

for t_transform() method and for test transform()
method.
fi
fi
t
t
fi
t
t
t
t
t
t
ffi
fl
 Bias Variance Trade-O
It refers to the balance between bias and variance in the
predictive performance of a model.

Bias

Bias refers to the error introduced by problem, which

may be extremely complex, by a much simpler model.
High bias can lead the model to under- t the data,
meaning it fails to capture the underlying pa erns in the
training data.

Variance

Variance is the model's sensitivity to small uctuations

or noise in the training data. A model with high variance
is very exible and can t the training data very closely,
but it might fail to generalise well to new, unseen data.

According to it we should keep low bias and low variance

for ba er result. Few techniques such as cross validation,
and proper model selection and nding a optimal trade-
o which can aim for right strike.
ff
t
t
fl
fi
ff
fi
fi
fl
t
t
4. Regularisation Linear Regression
It helps prevent over ing by discouraging overly
complex models, usually by penalising large coe cients.

Below, the types of Regularisation Techniques.

1. Ridge Regularisation (L2)

2. Lasso Regularisation (L1)

3. Elastic Net
fi
t
t
ffi
1. Ridge Regularisation (L2)

Ridge Regression is a regularisation technique used in

linear regression to prevent over ing and improve the
stability of the model.

Formula
1 n
( )
2
^yi + λ || w || 2
n∑
yi −
i=1

Ridge Regularisation with Simple Linear Regression.

y = mx + b

Now, we will see how we can get m and b.

b = ȳ − m x̄ (x̄ is mean of x-axis values, ȳ is mean of y-

axis values and m is below)
n
∑i=1 (x i − x̄ )(yi − ȳ)
m= n
∑i=1 (x i − x̄ )2+λ

λ is essential to strike a balance between ing the data

well and preventing over ing by adjusting it.

Ridge Regularisation with Multiple Linear Regression.

Formula

w = (X T X + α I )−1X T y

Here X is input matrix, w is the weight vector (including

the intercept) and y is observed target value. I is the
identity matrix. α is the regularisation parameter theta.
you will obtain the values for β which include the
fi
t
t
fi
t
t
fi
t
t
intercept β 0 and coe cients for each independent
variable.

Ridge Regularisation with Batch Gradient Descent.

Formula

w = w − λ ⋅ (X T X ⋅ w − X T y + α w)

Here X is input matrix, w is the weight vector (including

the intercept) and y is observed target value. λ is the
learning rate. αw is the alpha (regularisation parameter)
multiply with theta w. A er applying this formula we can
abstract b and m from it answer.

Also we can use regularisation parameter (Ridge) with

stochastic and mini-batch gradient descent.
ffi
f
t
 2. Lasso Regularisation (L1)

Lasso regression, or L1 regularisation, is a linear

regression technique that incorporates a regularisation
term in the objective function to prevent over ing and
promote feature selection.

The regularisation term is the absolute value of the

coe cients multiplied by a regularisation parameter (α).

The addition of the regularisation term encourages the

model to shrink some of the coe cients to exactly zero,
e ectively performing feature selection. This can be
bene cial in situations where only a subset of features is
truly relevant, leading to a simpler and more
interpretable model.

Formula
1 n
( )
2
^yi + λ || w ||
n∑
yi −
i=1
ff
ffi
fi
ffi
fi
t
t
3. Elastic Net

Elastic Net regression is a regularisation technique that

combines both L1 (Lasso) and L2 (Ridge) regularisation
penalties.

It is useful when dealing with datasets that have highly

correlated features, as it helps to select a group of
correlated features rather than selecting only one from
the group (as Lasso might do). When any input data has
more multicollinearity that time elastic net is useful.

Formula
1 n
( )
2
^yi + a || w || 2 + b || w ||
n∑
yi −
i=1