Scribd
Upload
119 views19 pages

Linear Classifiers in Python: Chapter3

The document discusses linear classifiers in Python, including logistic regression and regularization. It covers how regularization affects training and test accuracy, the differences between L1 and L2 regularization, and how to perform multi-class classification with logistic regression using one-vs-rest and multinomial approaches. Code examples are provided to fit logistic regression models and examine coefficients and predictions.

Uploaded by

Nishant
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
Download as pdf or txt
119 views19 pages

Linear Classifiers in Python: Chapter3

The document discusses linear classifiers in Python, including logistic regression and regularization. It covers how regularization affects training and test accuracy, the differences between L1 and L2 regularization, and how to perform multi-class classification with logistic regression using one-vs-rest and multinomial approaches. Code examples are provided to fit logistic regression models and examine coefficients and predictions.

Uploaded by

Nishant
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
Download as pdf or txt
Download as pdf or txt
You are on page 1/ 19

DataCamp Linear

Classifiers in Python

LINEAR CLASSIFIERS IN PYTHON

Logistic regression and


regularization

Michael (Mike) Gelbart


Instructor
The University of British Columbia
DataCamp Linear Classifiers in Python

Regularized logistic regression


DataCamp Linear Classifiers in Python

Regularized logistic regression


DataCamp Linear Classifiers in Python

How does regularization affect training accuracy?


In [1]: lr_weak_reg = LogisticRegression(C=100)

In [2]: lr_strong_reg = LogisticRegression(C=0.01)


In [3]: lr_weak_reg.fit(X_train, y_train)

In [4]: lr_strong_reg.fit(X_train, y_train)


In [3]: lr_weak_reg.score(X_train, y_train)
Out[3]: 1.0

In [4]: lr_strong_reg.score(X_train, y_train)


Out[4]: 0.92
regularized loss = original loss + large coefficient penalty
more regularization: lower training accuracy
DataCamp Linear Classifiers in Python

How does regularization affect test accuracy?


In [5]: lr_weak_reg.score(X_test, y_test)
Out[5]: 0.86

In [6]: lr_strong_reg.score(X_test, y_test)


Out[6]: 0.88
regularized loss = original loss + large coefficient penalty
more regularization: lower training accuracy
more regularization: (almost always) higher test accuracy
DataCamp Linear Classifiers in Python

L1 vs. L2 regularization
Lasso = linear regression with L1 regularization
Ridge = linear regression with L2 regularization
For other models like logistic regression we just say L1, L2, etc.
In [1]: lr_L1 = LogisticRegression(penalty='l1')

In [2]: lr_L2 = LogisticRegression() # penalty='l2' by default

In [3]: lr_L1.fit(X_train, y_train)

In [4]: lr_L2.fit(X_train, y_train)

In [5]: plt.plot(lr_L1.coef_.flatten())

In [6]: plt.plot(lr_L2.coef_.flatten())
DataCamp Linear Classifiers in Python

L2 vs. L1 regularization
DataCamp Linear Classifiers in Python

LINEAR CLASSIFIERS IN PYTHON

Let's practice!
DataCamp Linear Classifiers in Python

LINEAR CLASSIFIERS IN PYTHON

Logistic regression and


probabilities

Michael (Mike) Gelbart


Instructor
The University of British Columbia
DataCamp Linear Classifiers in Python

Logistic regression probabilities

Without regularization (C = 10 8 ):

model coefficients: [[1.55 1.57]]

model intercept: [-0.64]


DataCamp Linear Classifiers in Python

Logistic regression probabilities

Without regularization (C = 10 8 ):

model coefficients: [[1.55 1.57]]

model intercept: [-0.64]


DataCamp Linear Classifiers in Python

Logistic regression probabilities

Without regularization (C = 10 8 ): With regularization (C = 1):

model coefficients: [[1.55 1.57]] model coefficients: [[0.45 0.64]]

model intercept: [-0.64] model intercept: [-0.26]


DataCamp Linear Classifiers in Python

How are these probabilities computed?


logistic regression predictions: sign of raw model output
logistic regression probabilities: "squashed" raw model output
DataCamp Linear Classifiers in Python

LINEAR CLASSIFIERS IN PYTHON

Let's practice!
DataCamp Linear Classifiers in Python

LINEAR CLASSIFIERS IN PYTHON

Multi-class logistic
regression

Michael (Mike) Gelbart


Instructor
The University of British Columbia
DataCamp Linear Classifiers in Python

Combining binary classifiers with one-vs-rest


In [1]: lr0.fit(X, y==0)

In [2]: lr1.fit(X, y==1)

In [3]: lr2.fit(X, y==2)


In [4]: lr0.decision_function(X)[0] # get raw model output
Out[4]: 6.124

In [5]: lr1.decision_function(X)[0]
Out[5]: -5.429

In [6]: lr2.decision_function(X)[0]
Out[6]: -7.532
In [7]: lr.fit(X, y)
In [8]: lr.predict(X)[0]
Out[8]: 0
DataCamp Linear Classifiers in Python

One-vs-rest vs. multinomial/softmax

One-vs-rest: "Multinomial" or "softmax":

fit a binary classifier for each fit a single classifier for all
class classes
predict with all, take largest prediction directly outputs best
output class
pro: simple, modular con: more complicated, new
con: not directly optimizing code
accuracy pro: tackle the problem directly
common for SVMs as well possible for SVMs, but less
can produce probabilities common
DataCamp Linear Classifiers in Python

Model coefficients for multi-class


In [1]: lr_ovr = LogisticRegression() # one-vs-rest by default

In [2]: lr_ovr.fit(X,y)

In [3]: lr_ovr.coef_.shape
Out[3]: (3,13)

In [4]: lr_ovr.intercept_.shape
Out[4]: (3,)
In [5]: lr_mn = LogisticRegression(multi_class="multinomial",solver="lbfgs")

In [6]: lr_mn.fit(X,y)

In [7]: lr_mn.coef_.shape
Out[7]: (3,13)

In [8]: lr_mn.intercept_.shape
Out[8]: (3,)
DataCamp Linear Classifiers in Python

LINEAR CLASSIFIERS IN PYTHON

Let's practice!

You might also like