Data Science

A PROJECT REPORT
By: Soham B. Mistry
Enrollment no:216370307055

In partial fulfillment for the award of

Diploma ENGINEERING

in
computer engineering

(N. g. Patel polytechnic)

Gujarat Technological University, Ahmadabad

[Aug, 2023]

216370307055 1
Data Science

Certificate :

216370307055 2
Data Science

DECLARATION

I hereby certify that the work which is being presented in the report entitled
“Data Science” in fulfilment of the requirement for completion of one-month
industrial training in Department of Electronics and Communication
Engineering of “University Institute of Engineering and Technology,
Kurukshetra University” is an authentic record of my own work carried out
during industrial training.

Soham
216370305

CE-21 5th sem.

216370307055 3
Data Science

ACKNOWLEDGEMENT

The work in this report is an outcome of continuous work over a period and
drew intellectual support from Internshala and other sources. I would like to
articulate our profound gratitude and indebtedness to Internshala helped us
in completion of the training. I am thankful to Internshala Training Associates
for teaching and assisting me in making the training successful.

Soham
216370305

CE-21 5th sem.

216370307055 4
 Data Science

Introduction to Organization:
Company name:
STYPIX

Address:
A-903, Siddhivinayak Business Tower Nr, Vasna Telephone Exchange, Makarba, Ahmedabad, Gujarat 380051

About Us:
STYPIX delivers the best in class custom software solutions, elite software development teams and innovative
cloud software to enterprise businesses across numerous industries. We believe new technologies are the lifeline
of every business in the modern age and aim to connect businesses across all industries to innovative software,
technological development, solutions and services, in a manner that’s faster, easier and better than ever before.
Our mission is to create leading, innovative software that creates economic and social value on a global scale,
collaborating closely with our clients to help them achieve both their short and longer term goals.

Website URL:
https://www.stypix.co.in/

216370307055 5
Data Science

About Training:
The Data Science Training by Internshala is a 6-week online training program
in which Internshala aim to provide you with a comprehensive introduction
to data science. In this training program, you will learn the basics of python,
statistics, predictive modeling, and machine learning. This training program
has video tutorials and is packed with assignments, assessments tests,
quizzes, and practice exercises for you to get a hands-on learning experience.
At the end of this training program, you will have a solid understanding of
data science and will be able to build an end-to-end predictive model. For
doubt clearing, you can post your queries on the forum and get answers
within 24 hours.

216370307055 6
Data Science

Table of Content
Introduction to Organization

About Training

Module-1: Introduction to Data Science

1.1. Data Science Overview

Module-2: Python for Data Science

2.1. Introduction to Python

2.2. Understanding Operators
2.3. Variables and Data Types
2.4. Conditional Statements
2.5. Looping Constructs
2.6. Functions
2.7. Data Structure
2.8. Lists
2.9. Dictionaries
2.10. Understanding Standard Libraries in Python
2.11. Reading a CSV File in Python
2.12. Data Frames and basic operations with Data Frames
2.13. Indexing Data Frame
Module-3: Understanding the Statistics for Data Science

3.1. Introduction to Statistics

3.2. Measures of Central Tendency
3.3. Understanding the spread of data
3.4. Data Distribution
3.5. Introduction to Probability
3.6. Probabilities of Discreet and Continuous Variables
3.7. Central Limit Theorem and Normal Distribution
3.8. Introduction to Inferential Statistics
3.9. Understanding the Confidence Interval and margin of error

216370307055 7
Data Science

3.10. Hypothesis Testing

3.11. T tests
3.12. Chi Squared Tests
3.13. Understanding the concept of Correlation
Module-4: Predictive Modeling and Basics of Machine Learning

4.1. Introduction to Predictive Modeling

4.2. Understanding the types of Predictive Models
4.3. Stages of Predictive Models
4.4. Hypothesis Generation
4.5. Data Extraction
4.6. Data Exploration
4.7. Reading the data into Python
4.8. Variable Identification
4.9. Univariate Analysis for Continuous Variables
4.10. Univariate Analysis for Categorical Variables
4.11. Bivariate Analysis
4.12. Treating Missing Values
4.13. How to treat Outliers
4.14. Transforming the Variables
4.15. Basics of Model Building
4.16. Linear Regression
4.17. Logistic Regression
4.18. Decision Trees
4.19. Project

216370307055 8
Data Science

Module-1: Introduction to Data Science

1.1. Data Science Overview

Data science is the study of data. Like biological sciences is a study of biology,
physical sciences, it’s the study of physical reactions. Data is real, data has real
properties, and we need to study them if we’re going to work on them. Data
Science involves data and some signs.
It is a process, not an event. It is the process of using data to understand too many
different things, to understand the world. Let Suppose when you have a model or
proposed explanation of a problem, and you try to validate that proposed
explanation or model with your data.
It is the skill of unfolding the insights and trends that are hiding (or abstract) behind
data. It’s when you translate data into a story. So, use storytelling to generate
insight. And with these insights, you can make strategic choices for a company or
an institution.
We can also define data science as a field which is about processes and systems to
extract data of various forms and from various resources whether the data is
unstructured or structured.

Predictive modeling:
Predictive modeling is a form of artificial intelligence that uses data mining and
probability to forecast or estimate more granular, specific outcomes.
For example, predictive modeling could help identify customers who are likely to
purchase our new One AI software over the next 90 days.
Machine Learning:
Machine learning is a branch of artificial intelligence (ai) where computers learn to
act and adapt to new data without being programmed to do so. The computer is
able to act independently of human interaction.
Forecasting:
Forecasting is a process of predicting or estimating future events based on past
and present data and most commonly by analysis of trends. "Guessing" doesn't cut
it. A forecast, unlike a prediction, must have logic to it. It must be defendable. This
logic is what differentiates it from the magic 8 ball's lucky guess. After all, even a
broken watch is right two times a day.

216370307055 9
Data Science

Applications of Data Science:

Data science and big data are making an undeniable impact on businesses,
changing day-to-day operations, financial analytics, and especially interactions
with customers. It's clear that businesses can gain enormous value from the
insights data science can provide. But sometimes it's hard to see exactly how. So
let's look at some examples. In this era of big data, almost everyone generates
masses of data every day, often without being aware of it. This digital trace reveals
the patterns of our online lives. If you have ever searched for or bought a product
on a site like Amazon, you'll notice that it starts making recommendations related
to your search. This type of system known as a recommendation engine is a
common application of data science. Companies like Amazon, Netflix, and Spotify
use algorithms to make specific recommendations derived from customer
preferences and historical behavior. Personal assistants like Siri on Apple devices
use data science to devise answers to the infinite number of questions end users
may ask. Google watches your every move in the world, you're online shopping
habits, and your social media. Then it analyzes that data to create
recommendations for restaurants, bars, shops, and other attractions based on the
data collected from your device and your current location. Wearable devices like
Fitbits, Apple watches, and Android watches add information about your activity
levels, sleep patterns, and heart rate to the data you generate. Now that we know
how consumers generate data, let's take a look at how data science is impacting
business. In 2011, McKinsey & Company said that data science was going to
become the key basis of competition. Supporting new waves of productivity,
growth, and innovation. In 2013, UPS announced that it was using data from
customers, drivers, and vehicles, in a new route guidance system aimed to save
time, money, and fuel. Initiatives like this support the statement that data science
will fundamentally change the way businesses compete and operate. How does a
firm gain a competitive advantage? Let's take Netflix as an example. Netflix collects
and analyzes massive amounts of data from millions of users, including which
shows people are watching at what time a day when people pause, rewind, and
fast-forward, and which shows directors and actors they search for. Netflix can be
confident that a show will be a hit before filming even begins by analyzing users
preference for certain directors and acting talent, and discovering which
combinations people enjoy. Add this to the success of earlier versions of a show
and you have a hit. For example, Netflix knew many of its users had streamed to
the work of David Fincher. They also knew that films featuring Robin Wright had
always done well, and that the British version of House of Cards was very successful.

216370307055 10
Data Science

Netflix knew that significant numbers of people who liked Fincher also liked Wright.
All this information combined to suggest that buying the series would be a good
investment for the company.

216370307055 11
Data Science

Module-2: Python for Data Science

2.1. Introduction to Python

Python is a high-level, general-purpose and a very popular programming
language. Python programming language (latest Python 3) is being used in web
development, Machine Learning applications, along with all cutting edge
technology in Software Industry. Python Programming Language is very well suited
for Beginners, also for experienced programmers with other programming
languages like C++ and Java.

Below are some facts about Python Programming Language:

• Python is currently the most widely used multi-purpose, high-level

programming language.
• Python allows programming in Object-Oriented and Procedural paradigms.
• Python programs generally are smaller than other programming languages like
Java. Programmers have to type relatively less and indentation requirement of
the language, makes them readable all the time.
• Python language is being used by almost all tech-giant companies like –
Google, Amazon, Facebook, Instagram, Dropbox, Uber… etc.
• The biggest strength of Python is huge collection of standard library which can
be used for the following:
• Machine Learning
• GUI Applications (like Kivy, Tkinter, PyQt etc. )
• Web frameworks like Django (used by YouTube, Instagram, Dropbox)
• Image processing (like OpenCV, Pillow)
• Web scraping (like Scrapy, BeautifulSoup, Selenium)
• Test frameworks
• Multimedia
• Scientific computing
• Text processing and many more.

216370307055 12
Data Science

2.2. Understanding Operators

a. Arithmetic operators:
Arithmetic operators are used to perform mathematical operations like
addition, subtraction, multiplication and division.

OPERATOR DESCRIPTION SYNTAX

+ Addition: adds two operands x+y

- Subtraction: subtracts two operands x-y

* Multiplication: multiplies two operands x*y

/ Division (float): divides the first operand by the second x/y

// Division (floor): divides the first operand by the second x // y

Modulus: returns the remainder when first operand is

% divided by the second x%y

** Power : Returns first raised to power second x ** y

216370307055 13
Data Science

b. Relational Operators:
Relational operators compares the values. It either
returns True or False according to the condition.

OPERATOR DESCRIPTION SYNTAX

> Greater than: True if left operand is greater than the right x>y

< Less than: True if left operand is less than the right x<y

x ==

== Equal to: True if both operands are equal y

!= Not equal to - True if operands are not equal x != y

Greater than or equal to: True if left operand is greater than x >=

>= or equal to the right y

Less than or equal to: True if left operand is less than or equal x <=

<= to the right y

216370307055 14
Data Science

c. Logical operators:
Logical operators perform Logical AND, Logical OR and Logical
NOT operations.
OPERATOR DESCRIPTION SYNTAX

and Logical AND: True if both the operands are true x and y

or Logical OR: True if either of the operands is true x or y

not Logical NOT: True if operand is false not x

d. Bitwise operators:
Bitwise operators acts on bits and performs bit by bit operation.
OPERATOR DESCRIPTION SYNTAX

& Bitwise AND x&y

| Bitwise OR x|y

~ Bitwise NOT ~x

^ Bitwise XOR x^y

>> Bitwise right shift x>>

<< Bitwise left shift x<<

216370307055 15
Data Science

e. Assignment operators:
Assignment operators are used to assign values to the variables.

OPERATOR DESCRIPTION SYNTAX

Assign value of right side of expression to left

= side operand x=y+z

Add AND: Add right side operand with left

+= side operand and then assign to left operand a+=b a=a+b

Subtract AND: Subtract right operand from

-= left operand and then assign to left operand a-=b a=a-b

Multiply AND: Multiply right operand with left

*= operand and then assign to left operand a*=b a=a*b

Divide AND: Divide left operand with right

/= operand and then assign to left operand a/=b a=a/b

Modulus AND: Takes modulus using left and

right operands and assign result to left

%= operand a%=b a=a%b

216370307055 16
Data Science

Divide(floor) AND: Divide left operand with

right operand and then assign the

//= value(floor) to left operand a//=b a=a//b

Exponent AND: Calculate exponent(raise

power) value using operands and assign

**= value to left operand a**=b a=a**b

Performs Bitwise AND on operands and

&= assign value to left operand a&=b a=a&b

Performs Bitwise OR on operands and assign

|= value to left operand a|=b a=a|b

Performs Bitwise xOR on operands and assign

^= value to left operand a^=b a=a^b

Performs Bitwise right shift on operands and

>>= assign value to left operand a>>=b a=a>>b

a <<=

Performs Bitwise left shift on operands and b a= a <<

<<= assign value to left operand b

216370307055 17
Data Science

f. Special operators: There are some special type of operators like-

2.2.6.1 Identity operators:
is and is not are the identity operators both are used to check if two
values are located on the same part of the memory. Two variables that are
equal does not imply that they are identical.
is True if the operands are identical
is not True if the operands are not identical

2.2.6.2 Membership operators:

in and not in are the membership operators; used to test whether a value
or variable is in a sequence.
in True if value is found in the sequence
not in True if value is not found in the sequence

g. Precedence and Associativity of Operators:

Operator precedence and associativity as these determine the priorities of
the operator.
2.2.7.1 Operator Precedence:
This is used in an expression with more than one operator with different
precedence to determine which operation to perform first.
2.2.7.2 Operator Associativity:
If an expression contains two or more operators with the same precedence
then Operator Associativity is used to determine. It can either be Left to
Right or from Right to Left.

OPERATOR DESCRIPTION ASSOCIATIVITY

() Parentheses left-to-right

** Exponent right-to-left

* / % Multiplication/division/modulus left-to-right

216370307055 18
Data Science

OPERATOR DESCRIPTION ASSOCIATIVITY

+ - Addition/subtraction left-to-right

<< >> Bitwise shift left, Bitwise shift right left-to-right

Relational less than/less than or equal to

< <= Relational greater than/greater than or equal

> >= to left-to-right

== != Relational is equal to/is not equal to left-to-right

2.3. Variables and Data Types

Variables:
a. Python Variables Naming Rules:
There are certain rules to what you can name a variable(called an identifier).
• Python variables can only begin with a letter(A-Z/a-z) or an underscore(_).
• The rest of the identifier may contain letters(A-Z/a-z), underscores(_), and
numbers(0-9).
• Python is case-sensitive, and so are Python identifiers. Name and name are two
different identifiers.
b. Assigning and Reassigning Python Variables:
• To assign a value to Python variables, you don’t need to declare its type.
• You name it according to the rules stated in section 2a, and type the value after
the equal sign(=).
• You can’t put the identifier on the right-hand side of the equal sign.
• Neither can you assign Python variables to a keyword.

216370307055 19
Data Science

c. Multiple Assignment:
• You can assign values to multiple Python variables in one statement.
• You can assign the same value to multiple Python variables.
d. Deleting Variables:
• You can also delete Python variables using the keyword ‘del’.
Data Types:
A. Python Numbers:
There are four numeric Python data types.
a. int
int stands for integer. This Python Data Type holds signed integers. We can use the
type() function to find which class it belongs to.
b. float
This Python Data Type holds floating-point real values. An int can only store the
number 3, but float can store 3.25 if you want.
c. long
This Python Data type holds a long integer of unlimited length. But this construct
does not exist in Python 3.x.
d. complex
This Python Data type holds a complex number. A complex number looks like this:
a+bj Here, a and b are the real parts of the number, and j is imaginary.
B. Strings:
A string is a sequence of characters. Python does not have a char data type, unlike
C++ or Java. You can delimit a string using single quotes or double-quotes.
a. Spanning a String Across Lines:
To span a string across multiple lines, you can use triple quotes.
b. Displaying Part of a String:
You can display a character from a string using its index in the string. Remember,
indexing starts with 0.
c. String Formatters:
String formatters allow us to print characters and values at once. You can use the
% operator.
d. String Concatenation:
You can concatenate(join) strings using + operator. However, you cannot
concatenate values of different types.
C. Python Lists:
A list is a collection of values. Remember, it may contain different types of values.

216370307055 20
Data Science

To define a list, you must put values separated with commas in square brackets.
You don’t need to declare a type for a list either.
a. Slicing a List
You can slice a list the way you’d slice a string- with the slicing operator. Indexing
for a list begins with 0, like for a string. A Python doesn’t have arrays.
b. Length of a List
Python supports an inbuilt function to calculate the length of a list.
c. Reassigning Elements of a List
A list is mutable. This means that you can reassign elements later on.
d. Iterating on the List
To iterate over the list we can use the for loop. By iterating, we can access each
element one by one which is very helpful when we need to perform some
operations on each element of list.
e. Multidimensional Lists
A list may have more than one dimension. Have a detailed look on this in DataFlair’s
tutorial on Python Lists.
D. Python Tuples:
A tuple is like a list. You declare it using parentheses instead.
a. Accessing and Slicing a Tuple
You access a tuple the same way as you’d access a list. The same goes for slicing
it.
b. A tuple is Immutable
Python tuple is immutable. Once declared, you can’t change its size or elements.
E. Dictionaries:
A dictionary holds key-value pairs. Declare it in curly braces, with pairs separated
by commas. Separate keys and values by a colon(:).The type() function works with
dictionaries too.
a. Accessing a Value
To access a value, you mention the key in square brackets.
b. Reassigning Elements
You can reassign a value to a key.
c. List of Keys
Use the keys() function to get a list of keys in the dictionary.
F. Bool:
A Boolean value can be True or False.

216370307055 21
Data Science

G. Sets:
A set can have a list of values. Define it using curly braces. It returns only one
instance of any value present more than once. However, a set is unordered, so it
doesn’t support indexing. Also, it is mutable. You can change its elements or add
more. Use the add() and remove() methods to do so.
H. Type Conversion:
Since Python is dynamically-typed, you may want to convert a value into another
type. Python supports a list of functions for the same.
a. int()
b. float()
c. bool()
d. set()
e. list()
f. tuple()
g. str()
2.4. Conditional Statements
a. If statements
If statement is one of the most commonly used conditional statement in most of
the programming languages. It decides whether certain statements need to be
executed or not. If statement checks for a given condition, if the condition is true,
then the set of code present inside the if block will be executed.
The If condition evaluates a Boolean expression and executes the block of code
only when the Boolean expression becomes TRUE.
Syntax:
If (Boolean expression): Block of code #Set of statements to execute if the
condition is true

b. If-else statements
The statement itself tells that if a given condition is true then execute the
statements present inside if block and if the condition is false then execute the else
block.
Else block will execute only when the condition becomes false, this is the block
where you will perform some actions when the condition is not true.
If-else statement evaluates the Boolean expression and executes the block of code
present inside the if block if the condition becomes TRUE and executes a block of
code present in the else block if the condition becomes FALSE.

216370307055 22
Data Science

Syntax:
if(Boolean expression):
Block of code #Set of statements to execute if condition is true

else:
Block of code #Set of statements to execute if condition is false

c. elif statements
In python, we have one more conditional statement called elif statements. Elif
statement is used to check multiple conditions only if the given if condition false.
It’s similar to an if-else statement and the only difference is that in else we will not
check the condition but in elif we will do check the condition.
Elif statements are similar to if-else statements but elif statements evaluate
multiple conditions.
Syntax:
if (condition):
#Set of statement to execute if condition is true
elif (condition):
#Set of statements to be executed when if condition is false and elif
condition is true
else:
#Set of statement to be executed when both if and elif conditions are false

d. Nested if-else statements

Nested if-else statements mean that an if statement or if-else statement is present
inside another if or if-else block. Python provides this feature as well, this in turn
will help us to check multiple conditions in a given program.
An if statement present inside another if statement which is present inside another
if statements and so on.
Nested if Syntax:
if(condition):
#Statements to execute if condition is true
if(condition):
#Statements to execute if condition is true
#end of nested if
#end of if

216370307055 23
Data Science

Nested if-else Syntax:

if(condition):
#Statements to execute if condition is true
if(condition):
#Statements to execute if condition is true
else:
#Statements to execute if condition is false
else:

#Statements to execute if condition is false

e. elif Ladder
We have seen about the elif statements but what is this elif ladder. As the name
itself suggests a program which contains ladder of elif statements or elif statements
which are structured in the form of a ladder.
This statement is used to test multiple expressions.
Syntax:
if (condition):
#Set of statement to execute if condition is true
elif (condition):
#Set of statements to be executed when if condition is false and elif
condition is true
elif (condition):
#Set of statements to be executed when both if and first elif condition is
false and second elif condition is true
elif (condition):
#Set of statements to be executed when if, first elif and second elif
conditions are false and third elif statement is true
else:
#Set of statement to be executed when all if and elif conditions are false
2.5. Looping Constructs
Loops:
a. while loop:
Repeats a statement or group of statements while a given condition is TRUE. It
tests the condition before executing the loop body.

216370307055 24
Data Science

Syntax:
while expression:
statement(s)
b. for loop:
Executes a sequence of statements multiple times and abbreviates the code that
manages the loop variable.
Syntax:
for iterating_var in sequence:
statements(s)
c. nested loops:
You can use one or more loop inside any another while, for or do..while loop.
Syntax of nested for loop:
for iterating_var in sequence:
for iterating_var in sequence:
statements(s)
statements(s)
Syntax of nested while loop:
while expression:
while expression:
statement(s)
statement(s)
Loop Control Statements:
a. break statement:
Terminates the loop statement and transfers execution to the statement
immediately following the loop.
b. continue statement:
Causes the loop to skip the remainder of its body and immediately retest its
condition prior to reiterating.
c. pass statement:
The pass statement in Python is used when a statement is required syntactically
but you do not want any command or code to execute.
2.6. Functions
A. Built-in Functions or pre-defined functions:
These are the functions which are already defined by Python. For example: id (),
type(), print (), etc.

216370307055 25
Data Science

B. User-Defined Functions:
These are functions that are defined by the users for simplicity and to avoid
repetition of code. It is done by using def function.
2.7. Data Structure
Python has implicit support for Data Structures which enable you to store and
access data. These structures are called List, Dictionary, Tuple and Set.
2.8. Lists
Lists in Python are the most versatile data structure. They are used to store
heterogeneous data items, from integers to strings or even another list! They are
also mutable, which means that their elements can be changed even after the list
is created.
Creating Lists
Lists are created by enclosing elements within [square] brackets and each item is
separated by a comma.
Creating lists in Python
Since each element in a list has its own distinct position, having duplicate values in
a list is not a problem.
Accessing List elements
To access elements of a list, we use Indexing. Each element in a list has an index
related to it depending on its position in the list. The first element of the list has
the index 0, the next element has index 1, and so on. The last element of the list
has an index of one less than the length of the list.
Indexing in Python lists
While positive indexes return elements from the start of the list, negative indexes
return values from the end of the list. This saves us from the trivial calculation which
we would have to otherwise perform if we wanted to return the nth element from
the end of the list. So instead of trying to return List_name[len(List_name)-1]
element, we can simply write List_name[-1].

Using negative indexes, we can return the nth element from the end of the list
easily. If we wanted to return the first element from the end, or the last index, the
associated index is -1. Similarly, the index for the second last element will be -2,
and so on. Remember, the 0th index will still refer to the very first element in the
list.
Appending values in Lists
We can add new elements to an existing list using the append() or insert() methods:

216370307055 26
Data Science

append() – Adds an element to the end of the list

insert() – Adds an element to a specific position in the list which needs to be
specified along with the value
Removing elements from Lists
Removing elements from a list is as easy as adding them and can be done using
the remove() or pop() methods:
remove() – Removes the first occurrence from the list that matches the given value
pop() – This is used when we want to remove an element at a specified index from
the list. However, if we don’t provide an index value, the last element will be
removed from the list.
Sorting Lists
On comparing two strings, we just compare the integer values of each character
from the beginning. If we encounter the same characters in both the strings, we
just compare the next character until we find two differing characters.
Concatenating Lists
We can even concatenate two or more lists by simply using the + symbol. This will
return a new list containing elements from both the lists:
List comprehensions
A very interesting application of Lists is List comprehension which provides a neat
way of creating new lists. These new lists are created by applying an operation on
each element of an existing list. It will be easy to see their impact if we first check
out how it can be done using the good old for-loops.
Stacks & Queues using Lists
A list is an in-built data structure in Python. But we can use it to create user-defined
data structures. Two very popular user-defined data structures built using lists are
Stacks and Queues.
Stacks are a list of elements in which the addition or deletion of elements is done
from the end of the list. Think of it as a stack of books. Whenever you need to add
or remove a book from the stack, you do it from the top. It uses the simple concept
of Last-In-First-Out.
Queues, on the other hand, are a list of elements in which the addition of elements
takes place at the end of the list, but the deletion of elements takes place from the
front of the list. You can think of it as a queue in the real-world. The queue becomes
shorter when people from the front exit the queue. The queue becomes longer
when someone new adds to the queue from the end. It uses the concept of First-
In-First-Out.

216370307055 27
Data Science

2.9. Dictionaries
Dictionary is another Python data structure to store heterogeneous objects that
are immutable but unordered.
Generating Dictionary
Dictionaries are generated by writing keys and values within a { curly } bracket
separated by a semi-colon. And each key-value pair is separated by a comma:

Using the key of the item, we can easily extract the associated value of the item:

Dictionaries are very useful to access items quickly because, unlike lists and tuples,
a dictionary does not have to iterate over all the items finding a value. Dictionary
uses the item key to quickly find the item value. This concept is called hashing.

Accessing keys and values

You can access the keys from a dictionary using the keys() method and the values
using the values() method. These we can view using a for-loop or turn them into a
list using list():

216370307055 28
Data Science

We can even access these values simultaneously using the items() method which
returns the respective key and value pair for each element of the dictionary.

2.10. Understanding Standard Libraries in Python

Pandas
When it comes to data manipulation and analysis, nothing beats Pandas. It is the
most popular Python library, period. Pandas is written in the Python language
especially for manipulation and analysis tasks.
Pandas provides features like:
• Dataset joining and merging
• Data Structure column deletion and insertion
• Data filtration
• Reshaping datasets
• DataFrame objects to manipulate data, and much more!
NumPy
NumPy, like Pandas, is an incredibly popular Python library. NumPy brings in
functions to support large multi-dimensional arrays and matrices. It also brings in
high-level mathematical functions to work with these arrays and matrices. NumPy
is an open-source library and has multiple contributors.
Matplotlib
Matplotlib is the most popular data visualization library in Python. It allows us to
generate and build plots of all kinds. This is my go-to library for exploring data
visually along with Seaborn.
2.11. Reading a CSV File in Python
A CSV (Comma Separated Values) file is a form of plain text document which uses
a particular format to organize tabular information. CSV file format is a bounded
text document that uses a comma to distinguish the values. Every row in the
document is a data log. Each log is composed of one or more fields, divided by
commas. It is the most popular file format for importing and exporting
spreadsheets and databases.

216370307055 29
Data Science

• USing csv.reader(): At first, the CSV file is opened using the open() method
in ‘r’ mode(specifies read mode while opening a file) which returns the file
object then it is read by using the reader() method of CSV module that
returns the reader object that iterates throughout the lines in the specified
CSV document.
Note: The ‘with‘ keyword is used along with the open() method as it simplifies
exception handling and automatically closes the CSV file.

import csv

# opening the CSV file

with open('Giants.csv', mode ='r')as file:

# reading the CSV file

csvFile = csv.reader(file)

# displaying the contents of the CSV file

for lines in csvFile:
print(lines)
• Using pandas.read_csv() method: It is very easy and simple to read a CSV
file using pandas library functions. Here read_csv() method of pandas library
is used to read data from CSV files.

import pandas

# reading the CSV file

csvFile = pandas.read_csv('Giants.csv')

# displaying the contents of the CSV file

print(csvFile)

2.12. Data Frames and basic operations with Data Frames

Pandas DataFrame is two-dimensional size-mutable, potentially heterogeneous
tabular data structure with labeled axes (rows and columns). A Data frame is a two-
dimensional data structure, i.e., data is aligned in a tabular fashion in rows and
columns. Pandas DataFrame consists of three principal components, the data, rows,
and columns.

216370307055 30
Data Science

DataFrame Methods:

FUNCTION DESCRIPTION

index() Method returns index (row labels) of the DataFrame

insert() Method inserts a column into a DataFrame

add() Method returns addition of dataframe and other, element-wise (binary

operator add)

sub() Method returns subtraction of dataframe and other, element-wise

(binary operator sub)

216370307055 31
Data Science

mul() Method returns multiplication of dataframe and other, element-wise

(binary operator mul)

div() Method returns floating division of dataframe and other, element-wise

(binary operator truediv)

unique() Method extracts the unique values in the dataframe

nunique() Method returns count of the unique values in the dataframe

value_counts() Method counts the number of times each unique value occurs within

the Series

columns() Method returns the column labels of the DataFrame

axes() Method returns a list representing the axes of the DataFrame

isnull() Method creates a Boolean Series for extracting rows with null values

notnull() Method creates a Boolean Series for extracting rows with non-null

values

between() Method extracts rows where a column value falls in between a

predefined range

isin() Method extracts rows from a DataFrame where a column value exists

in a predefined collection

216370307055 32
Data Science

dtypes() Method returns a Series with the data type of each column. The

result’s index is the original DataFrame’s columns

astype() Method converts the data types in a Series

values() Method returns a Numpy representation of the DataFrame i.e. only the

values in the DataFrame will be returned, the axes labels will be

removed

sort_values()- Set1, Set2 Method sorts a data frame in Ascending or Descending order of

passed Column

sort_index() Method sorts the values in a DataFrame based on their index positions

or labels instead of their values but sometimes a data frame is made

out of two or more data frames and hence later index can be changed

using this method

loc[] Method retrieves rows based on index label

iloc[] Method retrieves rows based on index position

ix[] Method retrieves DataFrame rows based on either index label or index

position. This method combines the best features of the .loc[] and

.iloc[] methods

216370307055 33
Data Science

rename() Method is called on a DataFrame to change the names of the index

labels or column names

columns() Method is an alternative attribute to change the coloumn name

drop() Method is used to delete rows or columns from a DataFrame

pop() Method is used to delete rows or columns from a DataFrame

sample() Method pulls out a random sample of rows or columns from a

DataFrame

nsmallest() Method pulls out the rows with the smallest values in a column

nlargest() Method pulls out the rows with the largest values in a column

shape() Method returns a tuple representing the dimensionality of the

DataFrame

ndim() Method returns an ‘int’ representing the number of axes / array

dimensions.

Returns 1 if Series,

otherwise returns 2 if

DataFrame

216370307055 34
Data Science

dropna() Method allows the user to analyze and drop Rows/Columns with Null

values in different ways

fillna() Method manages and let the user replace NaN values with some value

of their own

rank() Values in a Series can be ranked in order with this method

query() Method is an alternate string-based syntax for extracting a subset

from a DataFrame

copy() Method creates an independent copy of a pandas object

duplicated() Method creates a Boolean Series and uses it to extract rows that have

duplicate values

drop_duplicates() Method is an alternative option to identifying duplicate rows and

removing them through filtering

2.13. Indexing Data Frame

This function allows us to retrieve rows and columns by position. In order to do
that, we’ll need to specify the positions of the rows that we want, and the positions
of the columns that we want as well. The df.iloc indexer is very similar to df.loc but
only uses integer locations to make its selections.

216370307055 35
Data Science

Module-3: Understanding the Statistics for Data

Science

3.1 Introduction to Statistics

Statistics simply means numerical data, and is field of math that generally deals with
collection of data, tabulation, and interpretation of numerical data. It is actually a
form of mathematical analysis that uses different quantitative models to produce a
set of experimental data or studies of real life. It is an area of applied mathematics
concern with data collection analysis, interpretation, and presentation. Statistics
deals with how data can be used to solve complex problems. Some people consider
statistics to be a distinct mathematical science rather than a branch of mathematics.
Statistics makes work easy and simple and provides a clear and clean picture of work
you do on a regular basis.
Basic terminology of Statistics:
• Population –
It is actually a collection of set of individuals or objects or events
whose properties are to be analyzed.
• Sample –
It is the subset of a population.
Types of Statistics :

216370307055 36
Data Science

3.2 Measures of Central Tendency

(i) Mean :
It is measure of average of all value in a sample set.
For example,

(ii) Median :
It is measure of central value of a sample set. In these, data set is ordered from
lowest to highest value and then finds exact middle.
For example,

216370307055 37
Data Science

(iii) Mode :
It is value most frequently arrived in sample set. The value repeated most of time
in central set is actually mode.
For example,

3.3 Understanding the spread of data

Measure of Variability is also known as measure of dispersion and used to
describe variability in a sample or population. In statistics, there are three
common measures of variability as shown below:
(i) Range :
It is given measure of how to spread apart values in sample set or data set.
Range = Maximum value - Minimum value
(ii) Variance :
It simply describes how much a random variable defers from expected value
and it is also computed as square of deviation.
S2= ∑ni=1 [(xi - x)2 ÷ n]
In these formula, n represent total data points, represent mean of data points
and xi represent individual data points.
(iii) Dispersion :

It is measure of dispersion of set of data from its mean.

σ= √ (1÷n) ∑ni=1 (xi - μ)2

3.4 Data Distribution

Terms related to Exploration of Data Distribution
-> Boxplot

-> Frequency Table

-> Histogram
-> Density Plot

216370307055 38
Data Science

Boxplot : It is based on the percentiles of the data as shown in the figure below.
The top and bottom of the boxplot are 75th and 25th percentile of the data. The
extended lines are known as whiskers that includes the range of rest of the data.
# BoxPlot Population In Millions
fig, ax1 = plt.subplots()
fig.set_size_inches(9, 15)

ax1 = sns.boxplot(x = data.PopulationInMillions, orient ="v")

ax1.set_ylabel("Population by State in Millions", fontsize = 15)
ax1.set_title("Population - BoxPlot", fontsize = 20)

Frequency Table : It is a tool to distribute the data into equally spaced ranges,
segments and tells us how many values fall in each segment.
Histogram: It is a way of visualizing data distribution through frequency table
with bins on the x-axis and data count on the y-axis.
Code – Histogram

# Histogram Population In Millions

fig, ax2 = plt.subplots()

fig.set_size_inches(9, 15)

ax2 = sns.distplot(data.PopulationInMillions, kde = False)

ax2.set_ylabel("Frequency", fontsize = 15)
ax2.set_xlabel("Population by State in Millions", fontsize = 15)
ax2.set_title("Population - Histogram", fontsize = 20)

216370307055 39
Data Science

Output :

Density Plot: It is related to histogram as it shows data-values being

distributed as continuous line. It is a smoothed histogram version. The output
below is the density plor superposed over histogram.
Code – Density Plot for the data

216370307055 40
Data Science

# Density Plot - Population

fig, ax3 = plt.subplots()

fig.set_size_inches(7, 9)

ax3 = sns.distplot(data.Population, kde = True)

ax3.set_ylabel("Density", fontsize = 15)
ax3.set_xlabel("Murder Rate per Million", fontsize = 15)
ax3.set_title("Desnsity Plot - Population", fontsize = 20)
Output :

3.5 Introduction to Probability

Probability refers to the extent of occurrence of events. When an event occurs like
throwing a ball, picking a card from deck, etc ., then the must be some probability
associated with that event.

216370307055 41
Data Science

In terms of mathematics, probability refers to the ratio of wanted outcomes to the

total number of possible outcomes. There are three approaches to the theory of
probability, namely:

1. Empirical Approach
2. Classical Approach
3. Axiomatic Approach
In this article, we are going to study about Axiomatic Approach.In this approach,
we represent the probability in terms of sample space(S) and other terms.
Basic Terminologies:
• Random Event :- If the repetition of an experiment occurs several
times under similar conditions, if it does not produce the same
outcome everytime but the outcome in a trial is one of the several
possible outcomes, then such an experiment is called random event
or a probabilistic event.
• Elementary Event – The elementary event refers to the outcome of
each random event performed. Whenever the random event is
performed, each associated outcome is known as elementary event.
• Sample Space – Sample Space refers to the set of all possible
outcomes of a random event.Example, when a coin is tossed, the
possible outcomes are head and tail.
• Event – An event refers to the subset of the sample space associated
with a random event.
• Occurrence of an Event – An event associated with a random event
is said to occur if any one of the elementary event belonging to it is
an outcome.
• Sure Event – An event associated with a random event is said to be
sure event if it always occurs whenever the random event is
performed.
• Impossible Event – An event associated with a random event is said
to be impossible event if it never occurs whenever the random event
is performed.
• Compound Event – An event associated with a random event is said
to be compound event if it is the disjoint union of two or more
elementary events.
• Mutually Exclusive Events – Two or more events associated with a
random event are said to be mutually exclusive events if any one of
the event occurrs, it prevents the occurrence of all other events.This
means that no two or more events can occur simultaneously at the
same time.

216370307055 42
Data Science

• Exhaustive Events – Two or more events associated with a random

event are said to be exhaustive events if their union is the sample
space.
Probability of an Event – If there are total p possible outcomes associated with a
random experiment and q of them are favourable outcomes to the event A, then
the probability of event A is denoted by P(A) and is given by
P(A) = q/p

3.6 Probabilities of Discreet and Continuous Variables

Random variable is basically a function which maps from the set of sample space
to set of real numbers. The purpose is to get an idea about result of a particular
situation where we are given probabilities of different outcomes.

Discrete Random Variable:

A random variable X is said to be discrete if it takes on finite number of values. The
probability function associated with it is said to be PMF = Probability mass function.
P(xi) = Probability that X = xi = PMF of X = pi.

1. 0 ≤ pi ≤ 1.
2. ∑pi = 1 where sum is taken over all possible values of x.
Continuous Random Variable:
A random variable X is said to be continuous if it takes on infinite number of
values. The probability function associated with it is said to be PDF = Probability
density function.
PDF: If X is continuous random variable.
P (x < X < x + dx) = f(x)*dx.
1. 0 ≤ f(x) ≤ 1; for all x
2. ∫ f(x) dx = 1 over all values of x
Then P (X) is said to be PDF of the distribution.

216370307055 43
Data Science

3.7 Central Limit Theorem and Normal Distribution

Whenever a random experiment is replicated, the Random Variable that equals
the average (or total) result over the replicates tends to have a normal
distribution as the number of replicates becomes large.It
is one of the cornerstones of probability theory and statistics, because of the
role it plays in the Central Limit Theorem, and because many real-world
phenomena involve random quantities that are approximately normal (e.g.,
errors in scientific measurement).
It is also known by other names such as- Gaussian Distribution, Bell shaped
Distribution.

It can be observed from the above graph that the distribution is symmetric about
its center, which is also the mean (0 in this case). This makes the probability of
events at equal deviations from the mean, equally probable. The density is highly
centered around the mean, which translates to lower probabilities for values
away from the mean.
Probability Density Function –
The probability density function of the general normal distribution is given as-

In the above formula, all the symbols have their usual meanings, is the Standard
Deviation and is the Mean.
It is easy to get overwhelmed by the above formula while trying to understand
everything in one glance, but we can try to break it down into smaller pieces so
as to get an intuition as to what is going on. The
z-score is a measure of how many standard deviations away a data point is

216370307055 44
Data Science

from the mean. Mathematically,

The exponent of in the above formula is the square of the z-score times . This
is actually in accordance to the observations that we made above. Values away
from the mean have a lower probability compared to the values near the mean.
Values away from the mean will have a higher z-score and consequentlya lower
probability since the exponent is negative. The opposite is true for valuescloser
to the mean.
This gives way for the 68-95-99.7 rule, which states that the percentage of
values that lie within a band around the mean in a normal distribution with a
width of two, four and six standard deviations, comprise 68%, 95% and 99.7% of
all the values. The figure given below shows this rule-

The effects of and on the distribution are shown below. Here is used to
reposition the center of the distribution and consequently move the graph left
or right, and is used to flatten or inflate the curve-

216370307055 45
Data Science

3.8 Introduction to Inferential Statistics

Inferential Statistics makes inference and prediction about population based on a
sample of data taken from population. It generalizes a large dataset and applies
probabilities to draw a conclusion. It is simply used for explaining meaning of
descriptive stats. It is simply used to analyze, interpret result, and draw conclusion.
Inferential Statistics is mainly related to and associated with hypothesis testing
whose main target is to reject null hypothesis.
Hypothesis testing is a type of inferential procedure that takes help of sample data
to evaluate and assess credibility of a hypothesis about a population. Inferential
statistics are generally used to determine how strong relationship is within sample.
But it is very difficult to obtain a population list and draw a random sample.
Types of inferential statistics –
Various types of inferential statistics are used widely nowadays and are very easy to
interpret. These are given below:
• One sample test of difference/One sample hypothesis test
• Confidence Interval
• Contingency Tables and Chi-Square Statistic
• T-test or Anova

216370307055 46
Data Science

3.9 Understanding the Confidence Interval and margin of

error
In simple terms, Confidence Interval is a range where we are certain that true value
exists. The selection of a confidence level for an interval determines the probability
that the confidence interval will contain the true parameter value. This range of
values is generally used to deal with population-based data, extracting specific,
valuable information with a certain amount of confidence, hence the term
‘Confidence Interval’.
Fig. Shows how a confidence interval generally looks like.

3.10 Hypothesis Testing

Hypothesis are statement about the given problem. Hypothesis testing is a
statistical method that is used in making a statistical decision using experimental
data. Hypothesis testing is basically an assumption that we make about a
population parameter. It evaluates two mutually exclusive statements about a
population to determine which statement is best supported by the sample data.
Parameters of hypothesis testing
• Null hypothesis(H0): In statistics, the null hypothesis is a general given
statement or default position that there is no relationship between two
measured cases or no relationship among groups.

216370307055 47
Data Science

In other words, it is a basic assumption or made based on the problem

knowledge.
Example: A company production is = 50 unit/per day etc.
• Alternative hypothesis(H1): The alternative hypothesis is the hypothesis
used in hypothesis testing that is contrary to the null hypothesis.
Example : A company production is not equal to 50 unit/per day etc.
• Level of significance
It refers to the degree of significance in which we accept or reject the null-
hypothesis. 100% accuracy is not possible for accepting a hypothesis, so we,
therefore, select a level of significance that is usually 5%. This is normally
denoted with and generally, it is 0.05 or 5%, which means your output
should be 95% confident to give similar kind of result in each sample.
• P-value
The P value, or calculated probability, is the probability of finding the
observed/extreme results when the null hypothesis(H0) of a study given
problem is true. If your P-value is less than the chosen significance level
then you reject the null hypothesis i.e. accept that your sample claims to
support the alternative hypothesis.
Error in Hypothesis Testing
• Type I error: When we reject the null hypothesis, although that hypothesis
was true. Type I error is denoted by alpha.
• Type II errors: When we accept the null hypothesis but it is false. Type II
errors are denoted by beta.
3.11 T tests
A t-test is a type of inferential statistic used to determine if there is a significant
difference between the means of two groups, which may be related in certain
features.

If t-value is large => the two groups belong to different groups.

If t-value is small => the two groups belong to same group.
There are three types of t-tests, and they are categorized as dependent and
independent t-tests.
1. Independent samples t-test: compares the means for two groups.
2. Paired sample t-test: compares means from the same group at different
times (say, one year apart).
3. One sample t-test test: the mean of a single group against a known mean.

216370307055 48
Data Science

3.12 Chi Squared Tests

Chi-square test is used for categorical features in a dataset. We calculate Chi-square
between each feature and the target and select the desired number of features with best
Chi-square scores. It determines if the association between two categorical variables of the
sample would reflect their real association in the population. Chi-
square score is given by :

3.13 Understanding the concept of Correlation

Correlation –
1. It show whether and how strongly pairs of variables are related to each
other.
2. Correlation takes values between -1 to +1, wherein values close to +1
represents strong positive correlation and values close to -1 represents
strong negative correlation.
3. In this variable are indirectly related to each other.
4. It gives the direction and strength of relationship between variables.
Formula –

Here,
x’ and y’ = mean of given sample set
n = total no of sample
xi and yi = individual sample of set
Example –

216370307055 49
Data Science

Module-4: Predictive Modeling and Basics of

Machine Learning

4.1. Introduction to Predictive Modeling

Predictive analytics involves certain manipulations on data from existing data sets
with the goal of identifying some new trends and patterns. These trends and patterns
are then used to predict future outcomes and trends. By performing predictive
analysis, we can predict future trends and performance. It is also defined as the
prognostic analysis, the word prognostic means prediction. Predictive analytics uses
the data, statistical algorithms and machine learning techniques to identify the
probability of future outcomes based on historical data.

4.2. Understanding the types of Predictive Models

Supervised learning
Supervised learning as the name indicates the presence of a supervisor as a teacher.
Basically supervised learning is a learning in which we teach or train the machine
using data which is well labeled that means some data is already tagged with the
correct answer. After that, the machine is provided with a new set of examples(data)
so that supervised learning algorithm analyses the training data(set of training
examples) and produces a correct outcome from labeled data.
Unsupervised learning
Unsupervised learning is the training of machine using information that is neither
classified nor labeled and allowing the algorithm to act on that information without
guidance. Here the task of machine is to group unsorted information according to
similarities, patterns and differences without any prior training of data.

4.3. Stages of Predictive Models

Steps To Perform Predictive Analysis:
Some basic steps should be performed in order to perform predictive analysis.
1. Define Problem Statement:
Define the project outcomes, the scope of the effort, objectives, identify the data
sets that are going to be used.
2. Data Collection:
Data collection involves gathering the necessary details required for the analysis.

216370307055 50
Data Science

It involves the historical or past data from an authorized source over which
predictive analysis is to be performed.
3. Data Cleaning:
Data Cleaning is the process in which we refine our data sets. In the process of
data cleaning, we remove un-necessary and erroneous data. It involves removing
the redundant data and duplicate data from our data sets.
4. Data Analysis:
It involves the exploration of data. We explore the data and analyze it thoroughly
in order to identify some patterns or new outcomes from the data set. In this
stage, we discover useful information and conclude by identifying some patterns
or trends.
5. Build Predictive Model:
In this stage of predictive analysis, we use various algorithms to build predictive
models based on the patterns observed. It requires knowledge of python, R,
Statistics and MATLAB and so on. We also test our hypothesis using standard
statistic models.
6. Validation:
It is a very important step in predictive analysis. In this step, we check the
efficiency of our model by performing various tests. Here we provide sample
input sets to check the validity of our model. The model needs to be evaluated
for its accuracy in this stage.
7. Deployment:
In deployment we make our model work in a real environment and it helps in
everyday discussion making and make it available to use.
8. Model Monitoring:
Regularly monitor your models to check performance and ensure that we have
proper results. It is seeing how model predictions are performing against actual
data sets.
4.4. Hypothesis Generation
A hypothesis is a function that best describes the target in supervised machine
learning. The hypothesis that an algorithm would come up depends upon the data
and also depends upon the restrictions and bias that we have imposed on the data.
To better understand the Hypothesis Space and Hypothesis consider the following
coordinate that shows the distribution of some data:

216370307055 51
Data Science

4.5. Data Extraction

In general terms, “Mining” is the process of extraction of some valuable material
from the earth e.g. coal mining, diamond mining etc. In the context of computer
science, “Data Mining” refers to the extraction of useful information from a bulk
of data or data warehouses. One can see that the term itself is a little bit confusing.
In case of coal or diamond mining, the result of extraction process is coal or
diamond. But in case of Data Mining, the result of extraction process is not data!!
Instead, the result of data mining is the patterns and knowledge that we gain at
the end of the extraction process. In that sense, Data Mining is also known as
Knowledge Discovery or Knowledge Extraction.
Data Mining as a whole process
The whole process of Data Mining comprises of three main phases:
1. Data Pre-processing – Data cleaning, integration, selection and transformation
takes place
2. Data Extraction – Occurrence of exact data mining
3. Data Evaluation and Presentation – Analyzing and presenting results

4.6. Data Exploration

216370307055 52
Data Science

Steps of Data Exploration and Preparation

Remember the quality of your inputs decide the quality of your output. So, once
you have got your business hypothesis ready, it makes sense to spend lot of time
and efforts here. With my personal estimate, data exploration, cleaning and
preparation can take up to 70% of your total project time.
Below are the steps involved to understand, clean and prepare your data for
building your predictive model:
• Variable Identification
• Univariate Analysis
• Bi-variate Analysis
• Missing values treatment
• Outlier treatment
• Variable transformation
• Variable creation
Finally, we will need to iterate over steps 4 – 7 multiple times before we come up
with our refined model.

4.7. Reading the data into Python

Python provides inbuilt functions for creating, writing and reading files. There are
two types of files that can be handled in python, normal text files and binary files
(written in binary language, 0s and 1s).
• Text files: In this type of file, Each line of text is terminated with a special character
called EOL (End of Line), which is the new line character (‘\n’) in python by default.
• Binary files: In this type of file, there is no terminator for a line and the data is
stored after converting it into machine-understandable binary language.

Access modes govern the type of operations possible in the opened file. It refers to
how the file will be used once it’s opened. These modes also define the location of
the File Handle in the file. File handle is like a cursor, which defines from where the
data has to be read or written in the file. Different access modes for reading a file
are –

1. Read Only (‘r’) : Open text file for reading. The handle is positioned at the
beginning of the file. If the file does not exists, raises I/O error. This is also the
default mode in which file is opened.
2. Read and Write (‘r+’) : Open the file for reading and writing. The handle is
positioned at the beginning of the file. Raises I/O error if the file does not exists.

216370307055 53
Data Science

3. Append and Read (‘a+’) : Open the file for reading and writing. The file is created
if it does not exist. The handle is positioned at the end of the file. The data being
written will be inserted at the end, after the existing data.
4.8. Variable Identification
First, identify Predictor (Input) and Target (output) variables. Next, identify the
data type and category of the variables.

Example:- Suppose, we want to predict, whether the students will play cricket or
not (refer below data set). Here you need to identify predictor variables, target
variable, data type of variables and category of variables.

Below, the variables have been defined in different category:

216370307055 54
Data Science

4.9. Univariate Analysis for Continuous Variables

Continuous Variables:- In case of continuous variables, we need to understand
the central tendency and spread of the variable. These are measured using various
statistical metrics visualization methods as shown below:

Note: Univariate analysis is also used to highlight missing and outlier values. In

the upcoming part of this series, we will look at methods to handle missing and
outlier values.

4.10. Univariate Analysis for Categorical Variables

For categorical variables, we’ll use frequency table to understand distribution
of each category. We can also read as percentage of values under each category.
It can be be measured using two metrics, Count and Count% against each
category. Bar chart can be used as visualization.

4.11. Bivariate Analysis

Bi-variate Analysis finds out the relationship between two variables. Here, we look
for association and disassociation between variables at a pre-defined significance
level. We can perform bi-variate analysis for any combination of categorical and
continuous variables. The combination can be: Categorical & Categorical,
Categorical & Continuous and Continuous & Continuous. Different methods are
used to tackle these combinations during analysis process.

Continuous & Continuous: While doing bi-variate analysis between two

continuous variables, we should look at scatter plot. It is a nifty way to find out the
relationship between two variables. The pattern of scatter plot indicates the
relationship between variables. The relationship can be linear or non-linear.

216370307055 55
Data Science

Scatter plot shows the relationship between two variable but does not indicates the
strength of relationship amongst them. To find the strength of the relationship, we
use Correlation. Correlation varies between -1 and +1.

• -1: perfect negative linear correlation

• +1:perfect positive linear correlation and

• 0: No correlation

Correlation can be derived using following formula:

Correlation = Covariance(X,Y) / SQRT( Var(X)* Var(Y))

Various tools have function or functionality to identify correlation between

variables. In Excel, function CORREL() is used to return the correlation between two
variables and SAS uses procedure PROC CORR to identify the correlation. These
function returns Pearson Correlation value to identify the relationship between two
variables:

216370307055 56
Data Science

In above example, we have good positive relationship(0.65) between two variables

X and Y.

Categorical & Categorical: To find the relationship between two categorical

variables, we can use following methods:

• Two-way table: We can start analyzing the relationship by creating a two-way

table of count and count%. The rows represents the category of one variable and
the columns represent the categories of the other variable. We show count or
count% of observations available in each combination of row and column
categories.
• Stacked Column Chart: This method is more of a visual form of Two-way table.

• Chi-Square Test: This test is used to derive the statistical significance of

relationship between the variables. Also, it tests whether the evidence in the sample
is strong enough to generalize that the relationship for a larger population as well.
Chi-square is based on the difference between the expected and observed
frequencies in one or more categories in the two-way table. It returns probability
for the computed chi-square distribution with the degree of freedom.

Probability of 0: It indicates that both categorical variable are dependent

Probability of 1: It shows that both variables are independent.

Probability less than 0.05: It indicates that the relationship between the variables is
significant at 95% confidence. The chi-square test statistic for a test of
independence of two categorical variables is found by:

where O represents the observed frequency. E is the expected

frequency under the null hypothesis and computed by:

From previous two-way table, the expected count for product category 1 to be of
small size is 0.22. It is derived by taking the row total for Size (9) times the column

216370307055 57
Data Science

total for Product category (2) then dividing by the sample size (81). This is
procedure is conducted for each cell. Statistical Measures used to analyze the
power of relationship are:

• Cramer’s V for Nominal Categorical Variable

• Mantel-Haenszed Chi-Square for ordinal categorical variable.

Different data science language and tools have specific methods to perform chi-
square test. In SAS, we can use Chisq as an option with Proc freq to perform this
test.

Categorical & Continuous: While exploring relation between categorical and

continuous variables, we can draw box plots for each level of categorical variables.
If levels are small in number, it will not show the statistical significance. To look at
the statistical significance we can perform Z-test, T-test or ANOVA.

• Z-Test/ T-Test:- Either test assess whether mean of two groups are statistically

different from each other or not. If the probability of Z is

small then the difference of two averages is more significant. The T-test is very
similar to Z-test but it is used when number of observation for both categories is
less than 30.

• ANOVA:- It assesses whether the average of more than two groups is statistically
different.

216370307055 58
Data Science

4.12. Treating Missing Values

Why missing values treatment is required?
Missing data in the training data set can reduce the power / fit of a model or can
lead to a biased model because we have not analysed the behavior and
relationship with other variables correctly. It can lead to wrong prediction or
classification.

Notice the missing values in the image shown above: In the left scenario, we have
not treated missing values. The inference from this data set is that the chances of
playing cricket by males is higher than females. On the other hand, if you look at
the second table, which shows data after treatment of missing values (based on
gender), we can see that females have higher chances of playing cricket compared
to males.

Why my data has missing values?

We looked at the importance of treatment of missing values in a dataset. Now, let’s

identify the reasons for occurrence of these missing values. They may occur at two
stages:

1. Data Extraction: It is possible that there are problems with extraction process. In
such cases, we should double-check for correct data with data guardians. Some
hashing procedures can also be used to make sure data extraction is correct. Errors
at data extraction stage are typically easy to find and can be corrected easily as
well.
2. Data collection: These errors occur at time of data collection and are harder to
correct. They can be categorized in four types:

216370307055 59
Data Science

o Missing completely at random: This is a case when the probability of missing

variable is same for all observations. For example: respondents of data collection
process decide that they will declare their earning after tossing a fair coin. If an
head occurs, respondent declares his / her earnings & vice versa. Here each
observation has equal chance of missing value.
o Missing at random: This is a case when variable is missing at random and missing
ratio varies for different values / level of other input variables. For example: We are
collecting data for age and female has higher missing value compare to male.
o Missing that depends on unobserved predictors: This is a case when themissing
values are not random and are related to the unobserved input variable. For
example: In a medical study, if a particular diagnostic causes discomfort, then there
is higher chance of drop out from the study. This missing value is not at random
unless we have included “discomfort” as an input variable for all patients.
o Missing that depends on the missing value itself: This is a case when the
probability of missing value is directly correlated with missing value itself. For
example: People with higher or lower income are likely to provide non-response
to their earning.

Which are the methods to treat missing values ?

1. Deletion: It is of two types: List Wise Deletion and Pair Wise Deletion.
o In list wise deletion, we delete observations where any of the variable is missing.
Simplicity is one of the major advantage of this method, but this method reduces
the power of model because it reduces the sample size.
o In pair wise deletion, we perform analysis with all cases in which the variables of
interest are present. Advantage of this method is, it keeps as many cases available
for analysis. One of the disadvantage of this method, it uses different sample size
for different variables.

216370307055 60
Data Science

o Deletion methods are used when the nature of missing data is “Missing
completely at random” else non random missing values can bias the model
output.
2. Mean/ Mode/ Median Imputation: Imputation is a method to fill in the missing
values with estimated ones. The objective is to employ known relationships that
can be identified in the valid values of the data set to assist in estimating the
missing values. Mean / Mode / Median imputation is one of the most frequently
used methods. It consists of replacing the missing data for a given attribute by the
mean or median (quantitative attribute) or mode (qualitative attribute) of all known
values of that variable. It can be of two types:-
o Generalized Imputation: In this case, we calculate the mean or median for all non
missing values of that variable then replace missing value with mean or median.
Like in above table, variable “Manpower” is missing so we take average of all non
missing values of “Manpower” (28.33) and then replace missing value with it.
o Similar case Imputation: In this case, we calculate average for gender “Male”
(29.75) and “Female” (25) individually of non missing values then replace the
missing value based on gender. For “Male“, we will replace missing values of
manpower with 29.75 and for “Female” with 25.
3. Prediction Model: Prediction model is one of the sophisticated method for
handling missing data. Here, we create a predictive model to estimate values that
will substitute the missing data. In this case, we divide our data set into two sets:
One set with no missing values for the variable and another one with missing
values. First data set become training data set of the model while second data set
with missing values is test data set and variable with missing values is treated as
target variable. Next, we create a model to predict target variable based on other
attributes of the training data set and populate missing values of test data set.We
can use regression, ANOVA, Logistic regression and various modeling technique
to perform this. There are 2 drawbacks for this approach:
o The model estimated values are usually more well-behaved than the true
values
o If there are no relationships with attributes in the data set and the attribute with
missing values, then the model will not be precise for estimating missing values.
4. KNN Imputation: In this method of imputation, the missing values of
an attribute are imputed using the given number of attributes that are most similar
to the attribute whose values are missing. The similarity of two attributes is
determined using a distance function. It is also known to have certain advantage
& disadvantages.
o Advantages:
▪ k-nearest neighbour can predict both qualitative & quantitative attributes
▪ Creation of predictive model for each attribute with missing data is not required

216370307055 61
Data Science

▪ Attributes with multiple missing values can be easily treated

▪ Correlation structure of the data is taken into consideration
o Disadvantage:
▪ KNN algorithm is very time-consuming in analyzing large database. It searches
through all the dataset looking for the most similar instances.
▪ Choice of k-value is very critical. Higher value of k would include attributes which
are significantly different from what we need whereas lower value of k implies
missing out of significant attributes.

4.13. How to treat Outliers

Outlier is a commonly used terminology by analysts and data scientists as it
needs close attention else it can result in wildly wrong estimations. Simply
speaking, Outlier is an observation that appears far away and diverges from an
overall pattern in a sample.
Most commonly used method to detect outliers is visualization. We use various
visualization methods, like Box-plot, Histogram, Scatter Plot (above, we have
used box plot and scatter plot for visualization). Some analysts also various thumb
rules to detect outliers. Some of them are:

• Any value, which is beyond the range of -1.5 x IQR to 1.5 x IQR
• Use capping methods. Any value which out of range of 5th and 95th percentile can
be considered as outlier
• Data points, three or more standard deviation away from mean are considered
outlier
• Outlier detection is merely a special case of the examination of data for influential
data points and it also depends on the business understanding
• Bivariate and multivariate outliers are typically measured using either an index of
influence or leverage, or distance. Popular indices such as Mahalanobis’ distance
and Cook’s D are frequently used to detect outliers.
• In SAS, we can use PROC Univariate, PROC SGPLOT. To identify outliers and
influential observation, we also look at statistical measure like STUDENT, COOKD,
RSTUDENT and others.

How to remove Outliers?

Most of the ways to deal with outliers are similar to the methods of missing values
like deleting observations, transforming them, binning them, treat them as a
separate group, imputing values and other statistical methods. Here, we will
discuss the common techniques used to deal with outliers:

216370307055 62
Data Science

Deleting observations: We delete outlier values if it is due to data entry error,

data processing error or outlier observations are very small in numbers. We can
also use trimming at both ends to remove outliers.

Transforming and binning values: Transforming variables can also eliminate

outliers. Natural log of a value reduces the variation caused by extreme values.
Binning is also a form of variable transformation. Decision Tree algorithm allows to
deal with outliers well due to binning of variable. We can also use the process of
assigning weights to different observations.

Imputing: Like imputation of missing values, we can also impute outliers. We can
use mean, median, mode imputation methods. Before imputing values, we should
analyse if it is natural outlier or artificial. If it is artificial, we can go with imputing
values. We can also use statistical model to predict values of outlier observation
and after that we can impute it with predicted values.

Treat separately: If there are significant number of outliers, we should treat them
separately in the statistical model. One of the approach is to treat both groups as
two different groups and build individual model for both groups and then combine
the output.

4.14. Transforming the Variables

• When we want to change the scale of a variable or standardize the values of a

variable for better understanding. While this transformation is a must if you have data
in different scales, this transformation does not change the shape of the variable
distribution

• When we can transform complex non-linear relationships into linear

relationships. Existence of a linear relationship between variables is easier to

216370307055 63
Data Science

comprehend compared to a non-linear or curved relation. Transformation helps us to

convert a non-linear relation into linear relation. Scatter plot can be used to find the
relationship between two continuous variables. These transformations also improve
the prediction. Log transformation is one of the commonly used transformation
technique used in these situations.

• Symmetric
distribution is preferred over skewed distribution as it is easier to interpret and
generate inferences. Some modeling techniques requires normal distribution of
variables. So, whenever we have a skewed distribution, we can use transformations
which reduce skewness. For right skewed distribution, we take square / cube root or
logarithm of variable and for left skewed, we take square / cube or exponential of
variables.

• Variable Transformation is also done from an implementation point of view

(Human involvement). Let’s understand it more clearly. In one of my project on
employee performance, I found that age has direct correlation with performance of
the employee i.e. higher the age, better the performance. From an implementation
stand point, launching age based progamme might present implementation
challenge. However, categorizing the sales agents in three age group buckets of <30
years, 30-45 years and >45 and then formulating three different strategies for each
group is a judicious approach. This categorization technique is known as Binning of
Variables.

216370307055 64
Data Science

4.15. Basics of Model Building

Lifecycle of Model Building –
• Select variables
• Balance data
• Build models
• Validate
• Deploy
• Maintain
• Define success
• Explore data
• Condition data
Data exploration is used to figure out gist of data and to develop first step
assessment of its quality, quantity, and characteristics. Visualization techniques can
be also applied. However, this can be difficult task in high dimensional spaces with
many input variables. In the conditioning of data, we group functional data which is
applied upon modeling techniques after then rescaling is done, in some cases
rescaling is an issue if variables are coupled. Variable section is very important to
develop quality model.
This process is implicity model-dependent since it is used to configure which
combination of variables should be used in ongoing model development. Data
balancing is to partition data into appropriate subsets for training, test, and
validation. Model building is to focus on desired algorithms. The most famous
technique is symbolic regression, other techniques can also be preferred.

4.16. Linear Regression

Linear Regression is a machine learning algorithm based on supervised
regression algorithm. Regression models a target prediction value based on
independent variables. It is mostly used for finding out the relationship between
variables and forecasting. Different regression models differ based on – the kind of
relationship between the dependent and independent variables, they are
considering and the number of independent variables being used.

4.17. Logistic Regression

Logistic regression is basically a supervised classification algorithm. In a
classification problem, the target variable(or output), y, can take only discrete values
for a given set of features(or inputs), X.

216370307055 65
Data Science

Any change in the coefficient leads to a change in both the direction and the
steepness of the logistic function. It means positive slopes result in an S-shaped
curve and negative slopes result in a Z-shaped curve.
4.18. Decision Trees
Decision Tree : Decision tree is the most powerful and popular tool for classification
and prediction. A Decision tree is a flowchart like tree structure, where each internal
node denotes a test on an attribute, each branch represents an outcome of the test,
and each leaf node (terminal node) holds a class label.
Decision Tree Representation :
Decision trees classify instances by sorting them down the tree from the root to
some leaf node, which provides the classification of the instance. An instance is
classified by starting at the root node of the tree,testing the attribute specified by
this node,then moving down the tree branch corresponding to the value of the
attribute as shown in the above figure.This process is then repeated for the subtree
rooted at the new node.
Strengths and Weakness of Decision Tree approach
The strengths of decision tree methods are:
• Decision trees are able to generate understandable rules.
• Decision trees perform classification without requiring much computation.
• Decision trees are able to handle both continuous and categorical variables.
• Decision trees provide a clear indication of which fields are most important for
prediction or classification.
The weaknesses of decision tree methods :
• Decision trees are less appropriate for estimation tasks where the goal is to predict
the value of a continuous attribute.
• Decision trees are prone to errors in classification problems with many class and
relatively small number of training examples.
• Decision tree can be computationally expensive to train. The process of growing a
decision tree is computationally expensive. At each node, each candidate splitting
field must be sorted before its best split can be found. In some algorithms,
combinations of fields are used and a search must be made for optimal combining
weights. Pruning algorithms can also be expensive since many candidate sub-trees
must be formed and compared.

216370307055 66
Data Science

4.19 Project:

Project Abtract:
Accurate prediction of flight prices plays a crucial role in the travel industry,
enablingtravelers to plan their journeys effectively and make informed decisions
regarding their air travel expenses. This project utilizes data science techniques to
predict flight prices based on a comprehensive dataset comprising various attributes
and historical flight data. By leveraging the power of data analysis and predictive
modeling, this study aims to provide valuable insights into the factors influencing flight
prices.

Through data exploration, feature engineering, and rigorous model training, this project
develops a robust framework for estimating flight prices. While the specific algorithms
and techniques employed in this project are not discussed in this abstract, the focus
remains on the predictive power of the model and the potential benefits it offers to
travelers and airlines alike.

Accurate flight price predictions empower travelers to plan their trips more effectively,
enabling them to budget and make informed choices regarding their flight bookings.
Moreover, airlines can leverage these predictions to optimize revenue management,
pricing strategies, and flight availability. By understanding the underlying patterns and
trends in flight prices, airlines can better anticipate demand fluctuations and adjust
fares accordingly.

The project's findings shed light on the key factors influencing flight prices, such as
departure location, destination, time of travel, seasonality, airline preferences, and
flight duration. This understanding aids travelers in finding the best deals and
optimizing their travel plans, while airlines can enhance their pricing strategies and
maximize profitability.

216370307055 67
Data Science

Columns Details
Airline: This column represents the name or code of the airline associated with the flight.
Date_of_Journey: This column denotes the date of the flight journey.
Source: This column indicates the source or departure city of the flight.
Destination: This column represents the destination city of the flight.
Route: This column provides the route or path of the flight, including any stopovers or layovers.
Dep_Time: This column denotes the departure time of the flight.
Arrival_Time: This column represents the arrival time of the flight.
Duration: This column indicates the duration of the flight journey.
Total_Stops: This column represents the total number of stops or layovers during the flight journey.
Additional_Info: This column may contain additional information or special remarks about the flight.
Price: This column denotes the price or fare of the flight ticket.

216370307055 68
 Data Science

Code:
1.

2.

3.

216370307055 69
 Data Science

4.

5.

216370307055 70
 Data Science

6.

7.

216370307055 71
 Data Science

8.

9.

216370307055 72
 Data Science

10.

11.

216370307055 73
 Data Science

12. visualization

13.

216370307055 74
 Data Science

14.

216370307055 75
 Data Science

15.

216370307055 76
 Data Science

16.

17.

216370307055 77
 Data Science

18.

19.

216370307055 78
 Data Science

20.

21.

22.

216370307055 79
 Data Science

23.

216370307055 80