Data Analysis With Python & Pandas

DATA ANALYSIS WITH
With Expert Python Instructor Chris Bruehl
*Copyright Maven Analytics, LLC

COURSE STRUCTURE
This is a project-based course for students looking for a practical, hands-on, and highly
engaging approach to learning data analysis with Python using the Pandas library
Additional resources include:
Downloadable PDF to serve as a helpful reference when you’re offline or on the go
Quizzes & Assignments to test and reinforce key concepts, with step-by-step solutions
Interactive demos to keep you engaged and apply your skills throughout the course

COURSE OUTLINE
Introduce Pandas & NumPy, two critical Python libraries that help structure
1 Intro to Pandas & NumPy data in arrays & DataFrames and contain built-in functions for data analysis
Introduce Pandas Series, the Python equivalent of a column of data, and cover
2 Pandas Series their basic properties, creation, manipulation, and useful functions for analysis
Work with Pandas DataFrames, the Python equivalent of an Excel or SQL

3 DataFrames table, and use them to store, manipulate, and analyze data efficiently
4 Aggregating & Reshaping Aggregate & reshape data in DataFrames by grouping columns, performing
aggregation calculations, and pivoting & unpivoting data
Learn the basics of data visualization in Pandas, and use the plot method to
5 Data Visualization create & customize line charts, bar charts, scatterplots, and histograms

COURSE OUTLINE
Put your skills to the test on a new dataset by analyzing key metrics for a new
6 Mid-Course Project retailer and share insights into its potential acquisition by Maven MegaMart
Learn how to work with the datetime data type in Pandas to extract date
7 Time Series components, group by dates, and perform calculations like moving averages
Read in data from flat files and apply processing steps during import, create
8 Importing & Exporting Data DataFrames by querying SQL tables, and write data back out to its source
Combine multiple DataFrames by joining data from related fields to add new
9 Combining DataFrames columns, and appending data with the same fields to add new rows
10 Final Project Put the finishing touches on your project by joining a new table, performing
time series analysis, and optimizing your workflow, and writing out your results

INTRODUCING THE COURSE PROJECT
You’ve just been hired as a Data Analyst for Maven MegaMart, a multinational
THE corporation that operates a chain of retail and grocery stores. They recently
SITUATION received a sample of data from a new retailer they’re looking to acquire, and
they need you to identify and deliver key insights about their sales history.
Your task is to analyze over 2 million transactions by product, household,

THE and store to get a better understanding of the retailer’s main strengths.
ASSIGNMENT From there, you need to review the company’s discount scheme to assess
whether they can expect to attract customers without losing margin.
Use Python to:

THE
• Read in multiple flat files efficiently
OBJECTIVES
• Join tables to provide a single source of information
• Shape & aggregate sales data to calculate KPIs
• Visualize the data to communicate your findings

SETTING EXPECTATIONS
This course covers the core Pandas functionality

• We’ll cover critical functions, methods, and best practices for manipulating and analyzing data using
Pandas DataFrames, but won’t dive into advanced statistical analysis or data prep for machine learning
We will only cover basic data visualization in Python

• We'll use Matplotlib to create & customize common charts like histograms, scatterplots, bar & line charts,
but we'll cover data visualization in depth in a separate course
We’ll use Jupyter Notebooks as our primary coding environment

• Jupyter Notebooks are free to use, and the industry standard for conducting data analysis with Python
(we’ll introduce Google Colab as an alternative, cloud-based environment as well)
You do NOT need to be a Python expert to take this course

• It is strongly recommended that you complete our Python Essentials for Analysts course, or have a solid
understanding of Base Python concepts like data types, variables, conditional logic, loops, and functions
INSTALLING ANACONDA (MAC)
1) Go to anaconda.com/products/distribution and click
4) Follow the installation steps

(default settings are OK)
2) Click X on the Anaconda Nucleus pop-up

(no need to launch)
3) Launch the downloaded Anaconda pkg file

INSTALLING ANACONDA (PC)
1) Go to anaconda.com/products/distribution and click
4) Follow the installation steps

(default settings are OK)
2) Click X on the Anaconda Nucleus pop-up

(no need to launch)
3) Launch the downloaded Anaconda exe file

LAUNCHING JUPYTER
1) Launch Anaconda Navigator 2) Find Jupyter Notebook and click

YOUR FIRST JUPYTER NOTEBOOK
1) Once inside the Jupyter interface, create a folder to store your notebooks for the course
NOTE: You can rename your folder by clicking “Rename” in the top left corner
2) Open your new coursework folder and launch your first Jupyter notebook!
NOTE: You can rename your notebook by clicking on the title at the top of the screen

THE NOTEBOOK SERVER
NOTE: When you launch a Jupyter notebook, a terminal window may pop up as
well; this is called a notebook server, and it powers the notebook interface
If you close the server window,

your notebooks will not run!
Depending on your OS, and method

of launching Jupyter, one may not
open. As long as you can run your
notebooks, don’t worry!

ALTERNATIVE: GOOGLE COLAB
Google Colab is Google’s cloud-based version of Jupyter Notebooks
To create a Colab notebook:

1. Log in to a Gmail account
2. Go to colab.research.google.com
3. Click “new notebook”
Colab is very similar to Jupyter Notebooks

(they even share the same file extension); the main
difference is that you are connecting to Google
Drive rather than your machine, so files will be
stored in Google’s cloud

INTRO TO PANDAS & NUMPY
In this section we’ll introduce Pandas & NumPy, two critical Python libraries that help
structure data in arrays & DataFrames and contain built-in functions for data analysis
TOPICS WE’LL COVER: GOALS FOR THIS SECTION:

• Convert Python lists to NumPy arrays, and create
new arrays from scratch using functions
• Apply array indexing, slicing, methods, and functions

to perform operations on NumPy arrays
• Understand the concepts of vectorization and

broadcasting, which are critical in making NumPy
and Pandas more efficient than base Python

MEET PANDAS
Pandas is Python’s most widely used library for data analysis, and contains
functions for accessing, aggregating, joining, and analyzing data
Its data structure, the DataFrame, is analogous to SQL tables or Excel worksheets
Custom indices & column

titles make working with
datasets more intuitive!

MEET NUMPY
NumPy is an open-source library that is the universal standard for working with
numerical data in Python, and forms the foundation of other libraries like Pandas
Pandas DataFrames are built on NumPy arrays and can leverage NumPy functions
The indices, column names,

and data columns are all
stored as NumPy arrays
Pandas just adds convenient
wrappers and functions!

NUMPY ARRAYS
NumPy arrays are fixed-size containers of items that are more efficient than
Python lists or tuples for data processing
• They only store a single data type (mixed data types are stored as a string)
• They can be one dimensional or multi-dimensional
• Array elements can be modified, but the array size cannot change
‘np’ is the standard alias for the NumPy library
NumPy’s array function

converts Python lists and
tuples into NumPy arrays

ARRAY PROPERTIES
NumPy arrays have these key properties:

• ndim – the number of dimensions (axes) in the array
• shape – the size of the array for each dimension
• size – the total number of elements in the array
• dtype – the data type of the elements in the array
NumPy arrays are a ndarray Python data type,

which stands for n-dimensional array
The sales_array has 1 dimension

The dimension has a size of 10
The array has 10 elements total
The elements are stored as 64-bit integers

ARRAY PROPERTIES
NumPy arrays have these key properties:

• ndim – the number of dimensions (axes) in the array
• shape – the size of the array for each dimension
• size – the total number of elements in the array
• dtype – the data type of the elements in the array
Converting a nested list creates a multi-dimensional

array, where each nested list is a dimension
NOTE: The nested lists must be of equal length
The sales_array has 2 dimensions

The first dimension has a size of 2 (rows) and the second a size of 5 (columns)
It has 10 elements total
The elements are stored as 64-bit integers

ASSIGNMENT: ARRAY BASICS
Results Preview
NEW MESSAGE
June 16, 2022
From: Ross Retail (Head of Analytics)

Subject: NumPy?
Hi there, welcome to the Maven MegaMart!
Your resume mentions you have basic Python experience.

Our finance team has been asking us to cut software costs –
can you help us dig into Python as an analysis tool?
I know NumPy is foundational, but not much beyond that.
Can you convert a Python list into a NumPy array and help me
get familiar with their properties?
Thanks!
numpy_assignments.ipynb

SOLUTION: ARRAY BASICS
Solution Code
NEW MESSAGE
June 16, 2022

Subject: NumPy?
Hi there, welcome to the Maven MegaMart!
Your resume mentions you have basic Python experience.

Our finance team has been asking us to cut software costs –
can you help us dig into Python as an analysis tool?
I know NumPy is foundational, but not much beyond that.
Can you convert a Python list into a NumPy array and help me
get familiar with their properties?
Thanks!

ARRAY CREATION
As an alternative to converting lists, you can create arrays using functions
Creates an array of ones of a given size, as float by default np.ones((rows, cols), dtype)
Creates an array of zeros of a given size, as float by default np.zeros((rows, cols), dtype)
Creates an array of integers with given start & stop values, and a step
size (only stop is required, and is not inclusive) np.arange(start, stop, step)
Creates an array of floats with given start & stop values with n
elements, separated by a consistent step size (stop is inclusive) np.linspace(start, stop, n)
Changes an array into the specified dimensions, if compatible np.array.reshape(rows, cols)

ARRAY CREATION
As an alternative to converting lists, you can create arrays using functions
np.ones((rows, cols), dtype) np.arange(start, stop, step) start is 0 and step is 1 by default
stop is not inclusive
np.zeros((rows, cols), dtype) np.linspace(start, stop, n)
stop is inclusive
np.array.reshape(rows, cols)

RANDOM NUMBER ARRAYS
You can create random number arrays from a variety of distributions using
NumPy functions and methods (great for sampling and simulation!)
Creates a random number generator (the seed is for reproducibility) np.default_rng(seed)
Returns n random numbers from a uniform distribution between 0 and 1 rng.random(n)
Returns n random numbers from a normal distribution with a given mean

and standard deviation rng.normal(mean, stdev, n)
'rng' is the standard variable name for

the default_rng number generator

RANDOM NUMBER ARRAYS
You can create random number arrays from a variety of distributions using
NumPy functions and methods (great for sampling and simulation!)
First, we're creating a random number generator with a

seed of 12345 and assigning it to 'rng' using default_rng
Then we're using the random method on 'rng' to return
an array with 10 random numbers
Here we're using the normal method on 'rng' to return an

array with 10 random numbers from a normal distribution
with a mean of 5 and a st. deviation of 1
PRO TIP: Even though it’s optional, make sure to set a seed when generating random numbers to
ensure you and others can recreate the work you’ve done (the value for the seed is less important)

ASSIGNMENT: ARRAY CREATION
Results Preview
NEW MESSAGE
June 17, 2022

Subject: Array Creation
Thanks for your help last time – I’m starting to understand

NumPy Arrays!
Are there any NumPy functions that can create arrays so we

don’t have to convert from a Python list? Recreate the array
from the first assignment but make it 5 rows by 2 columns.
Once you’ve done that, create an array of random numbers

between 0 and 1 in a 3x3 shape. One of our data scientists has
been asking about this so I want to make sure it’s possible.
Thanks!

SOLUTION: ARRAY CREATION
Solution Code
NEW MESSAGE
June 17, 2022

Subject: Array Creation
Thanks for your help last time – I’m starting to understand

NumPy Arrays!
Are there any NumPy functions that can create arrays so we

don’t have to convert from a Python list? Recreate the array
from the first assignment but make it 5 rows by 2 columns.
Once you’ve done that, create an array of random numbers

between 0 and 1 in a 3x3 shape. One of our data scientists has
been asking about this so I want to make sure it’s possible.
Thanks!

INDEXING & SLICING ARRAYS
Indexing & slicing one-dimensional arrays is the same as base Python

• array[index] – indexing to access a single element (0-indexed)
• array[start:stop:step size] – slicing to access a series of elements (stop is not inclusive)
This grabs the second and last

elements of product_array
This grabs the first five

elements of product_array
This starts at the sixth element

and grabs every other element
until the end of product_array

INDEXING & SLICING ARRAYS
Indexing & slicing two-dimensional arrays requires an extra index or slice

• array[row index, column index] – indexing to access a single element (0-indexed)
• array[start:stop:step size, start:stop:step size] – slicing to access a series of elements
This goes to the second row

and grabs the third element
This goes to all rows and grabs

all the elements starting from
the third in each row
This goes to the second row

and grabs all its elements

ASSIGNMENT: ARRAY ACCESS
Results Preview
NEW MESSAGE
May 1,
June 17,2017
2022

Subject: Indexing &
and
Slicing
Slicing
Arrays
Arrays
First two ‘rows’
Ok, last ‘theoretical’ exercise before we start working with

real data.
I am familiar with indexing and slicing in base Python but have

no idea how it works in multiple dimensions. First ‘column’
I’ve provided a few different ‘cuts’ of the data in the notebook
– can you slice and dice the random array we created in the
last exercise?
Thanks! Second number in third ‘row’:
section01_NumPy.ipynb

ASSIGNMENT: ARRAY ACCESS
Results Preview
NEW MESSAGE
May 1,
June 17,2017
2022

Subject: Indexing &
and
Slicing
Slicing
Arrays
Arrays
First two ‘rows’
Ok, last ‘theoretical’ exercise before we start working with

real data.
I am familiar with indexing and slicing in base Python but have

no idea how it works in multiple dimensions. First ‘column’
I’ve provided a few different ‘cuts’ of the data in the notebook
– can you slice and dice the random array we created in the
last exercise?
Thanks! Second number in third ‘row’:
section01_NumPy.ipynb

ARRAY OPERATIONS
Arithmetic operators can be used to perform array operations
Array operations are applied via

vectorization and broadcasting,
which eliminates the need to loop
through the array's elements
This adds 2 to every

element in the array
This assigns all the elements in the first row to 'quantity'

Then assigns all the elements in the second row to 'price'
Finally, it multiplies the corresponding elements in each array:
(0*0, 5*1827, 155*616, 0*317, and 518*325)

ASSIGNMENT: ARRAY OPERATIONS
Results Preview
NEW MESSAGE
May 1,
June 18,2017
2022

Subject: Random
IndexingDiscount
and Slicing Arrays
Ok, so now that we’ve gotten the basics down, we can start
using NumPy for our first tasks. As part of a promotion, we
want to apply a random discount to surprise our customers
and generate social media buzz.
First, add a flat shipping cost of 5 to our prices to get the

‘total’ amount owed.
The numbers in the random array represent ‘discount

percent’. To get the ‘percent owed’, subtract the first 6
numbers in the random array from 1, then multiply ‘percent The .round() array method rounds
owed’ by ‘total’ to get the final amount owed. the elements of the array to the
specified decimals

SOLUTION: ARRAY OPERATIONS
Solution Code
NEW MESSAGE
May 1,
June 18,2017
2022

Subject: Random
IndexingDiscount
and Slicing Arrays
Ok, so now that we’ve gotten the basics down, we can start
using NumPy for our first tasks. As part of a promotion, we
want to apply a random discount to surprise our customers
and generate social media buzz.
First, add a flat shipping cost of 5 to our prices to get the

‘total’ amount owed.
The numbers in the random array represent ‘discount

percent’. To get the ‘percent owed’, subtract the first 6
numbers in the random array from 1, then multiply ‘percent
owed’ by ‘total’ to get the final amount owed.
The .round() array method rounds
the elements of the array to the
specified decimals

FILTERING ARRAYS
You can filter arrays by indexing them with a logical test

• Only the array elements in positions where the logical test returns True are returned
Performing a logical test on a NumPy array

returns a Boolean array with the results of the
logical test on each array element
Indexing an array with a Boolean array returns an array with

the elements where the Boolean value is True

FILTERING ARRAYS
You can filter arrays with multiple logical tests

• Use | for or conditions and & for and conditions
This returns an array with elements

equal to 616 or less than 100
This returns an array with elements

greater than 100 and less than 500
PRO TIP: Store complex

filtering criteria in a variable
(known as a Boolean mask)

FILTERING ARRAYS
You can filter arrays based on values in other arrays

• Use the Boolean array returned from the other array to index the array you want to filter
This returns the elements from product_array

where values in sales_array are greater than 0

MODIFYING ARRAY VALUES
You can modify array values by assigning new ones
This assigns a single value via indexing
This filters the zero values in sales_array and

assigns them a new value of 5

THE WHERE FUNCTION
The where() NumPy function performs a logical test and returns a given value if
the test is True, or another if the test is False
A logical expression that

np.where(logical test, evaluates to True or False
value if True,
Value to return when
value if False) the expression is True
Calls the NumPy Value to return when

function library the expression is False

THE WHERE FUNCTION
The where() NumPy function performs a logical test and returns a given value if
the test is True, or another if the test is False
If inventory is zero or negative,

assign ‘Out of Stock’, otherwise
assign ‘In Stock’
If inventory is zero or negative,

assign ‘Out of Stock’, otherwise
assign the product_array value

ASSIGNMENT: FILTERING ARRAYS
Results Preview
NEW MESSAGE
May 1,
June 19,2017
2022

Subject: Subsetting
Indexing and
Our
Slicing
DataArrays
Hey there,
We’re working on some more promotions. Can you filter our

product list to only include prices greater than 25?
Once you’ve done that, modify your logic to force cola into
the list. Call this array ‘fancy_feast_special’.
Finally, we need to modify our shipping logic. Create a new

shipping cost array, but this time if price is greater than 20,
shipping cost is 0, otherwise shipping cost is 5.
Thanks!

SOLUTION: FILTERING ARRAYS
Solution Code
NEW MESSAGE
May 1,
June 19,2017
2022

Subject: Subsetting
Indexing and
Our
Slicing
DataArrays
Hey there,
We’re working on some more promotions. Can you filter our

product list to only include prices greater than 25?
Once you’ve done that, modify your logic to force cola into
the list. Call this array ‘fancy_feast_special’.
Finally, we need to modify our shipping logic. Create a new

shipping cost array, but this time if price is greater than 20,
shipping cost is 0, otherwise shipping cost is 5.
Thanks!

ARRAY AGGREGATION METHODS
Array aggregation methods let you calculate metrics like sum, mean, and max
array.sum() Returns the sum of all values in an array array.max() Returns the largest value in an array
array.mean() Returns the average of the values in an array array.min() Returns the smallest value in an array

ARRAY AGGREGATION METHODS
You can also aggregate across rows or columns
array.sum() Returns the sum of all values in an array array.sum(axis=0) Aggregates across rows
array.sum(axis=1) Aggregates across columns

ARRAY FUNCTIONS
Array functions let you perform other aggregations like median and percentiles
np.median(array) Returns the median value in an array
np.percentile(array, n) Returns a value in the nth percentile in an array
This uses linear interpolation by default

ARRAY FUNCTIONS
You can also return a unique list of values or the square root for each number
np.unique(array) Returns the unique values in an array
np.sqrt(array) Returns the square root of each value in an array

SORTING ARRAYS
The sort() method will sort arrays in place

• Use the axis argument to specify the dimension to sort by
axis=1 by default, which sorts a

two-dimensional array row by row
axis=0 will sort by columns

ASSIGNMENT: SORTING AND AGGREGATING
Results Preview
NEW MESSAGE
May 1,
June 19,2017
2022
Top 3 Price Stats
Subject: Top
Indexing
Tier Products
and Slicing Arrays
Hey there,
Thanks for all your hard work. I know we’re working with
small sample sizes, but we’re proving that analysis can be Unique Price Tiers
done in Python!
Can you calculate the mean, min, max, and median of our 3
most expensive product prices? Sorting the array first should
help!
Then, calculate the number of unique price tiers we have.
Thanks!

SOLUTION: SORTING AND AGGREGATING
Solution Code
NEW MESSAGE
May 1,
June 19,2017
2022
Top 3 Price Stats
Subject: Top
Indexing
Tier Products
and Slicing Arrays
Hey there,
Thanks for all your hard work. I know we’re working with
small sample sizes, but we’re proving that analysis can be Unique Price Tiers
done in Python!
Can you calculate the mean, min, max, and median of our 3
most expensive product prices? Sorting the array first should
help!
Then, calculate the number of unique price tiers we have.
Thanks!

VECTORIZATION
Vectorization is the process of pushing array operations into optimized C code,

which is easier and more efficient than writing for loops
Function that multiplies two Python lists
Function that multiplies two NumPy arrays
Generating and multiplying two lists
Converting and multiplying two arrays

~86 times faster!

VECTORIZATION
Vectorization is the process of pushing array operations into optimized C code,

which is easier and more efficient than writing for loops
PRO TIP: Use vectorized

operations whenever possible
when manipulating data, and
avoid writing loops
Converting and multiplying two arrays

~86 times faster!

BROADCASTING
Broadcasting lets you perform vectorized operations with arrays of different

sizes, where NumPy will expand the smaller array to ‘fit’ the larger one
• Single values (scalars) can be broadcast into arrays of any dimension
How does this code work?
1 2 3 1 2 3 4
1 2 3 + = 2 3 4
1 2 3 2 3 4

BROADCASTING

• Dimensions with a length greater than one must be the same size
1 2 3 3 2 1 4 4 4
1 2 3 + = 4 4 4
1 2 3 4 4 4

BROADCASTING

1 2 3 3 4 5 6
1 2 3 + 2 = 3 4 5
1 2 3 1 2 3 4

BROADCASTING

1 2 3 2 1
1 2 3 + =
1 2 3

BROADCASTING

1 2 3 2 3 4 5
+ 2 = 3 4 5
2 3 4 5

BROADCASTING

1 1 1 1 1 1
1 1 1 1 1 1 2 2 2
1 1 1 + = 1 1 1
1 1 1 2 2 2
1 1 1 1 1 1
1 1 1 2 2 2

BROADCASTING

Compatible shapes: Incompatible shapes:
(3, 3) (100, 3, 3) (100, 3, 3) (4, 3) (100, 3, 3)
(3, 1, 5, 1) (4, 1, 6) (3, 4, 5, 6) (3, 1, 5, 1) (4, 2, 6)

ASSIGNMENT: BRINGING IT ALL TOGETHER
Results Preview
NEW MESSAGE
May 1,
June 20,2017
2022

Subject: Final
Indexing
NumPy
and Challenge
Slicing Arrays
Alright, our new data scientist set up a little test case for us.
She provided code to read in data from a csv and convert two
columns to arrays.
Filter ‘sales_array’ to only include sales that had the product

family ‘PRODUCE’ in the ‘family_array’. Call this
produce_sales.
Then randomly sample half of the remaining sales and

calculate the mean of those sales.
Thanks!

SOLUTION: BRINGING IT ALL TOGETHER
Solution Code
NEW MESSAGE
May 1,
June 20,2017
2022

Subject: Final
Indexing
NumPy
and Challenge
Slicing Arrays
Alright, our new data scientist set up a little test case for us.
She provided code to read in data from a csv and convert two
columns to arrays.
Filter ‘sales_array’ to only include sales that had the product

family ‘PRODUCE’ in the ‘family_array’. Call this
produce_sales.
Then randomly sample half of the remaining sales and

calculate the mean of those sales.
Thanks!

KEY TAKEAWAYS
NumPy forms the foundation for Pandas

• As an analyst, it's important to be comfortable working with NumPy arrays & functions in order to use Pandas
data structures & functions properly
NumPy arrays are more efficient than base Python lists and tuples
• They are semi-mutable data structures capable of storing many data types
• Their values can be modified, but their size cannot
Array operations let you aggregate, filter, and sort data

• Broadcasting and vectorization make these operations convenient and efficient without the use of loops
• The syntax for NumPy array operations is very similar to Pandas

SERIES
In this section we’ll introduce Pandas Series, the Python equivalent of a column of data,
and cover their basic properties, creation, manipulation, and useful functions for analysis

• Understand the relationship between Pandas Series
and NumPy arrays
• Use the .loc() and .iloc() methods to access Series

data by their indices or values
• Learn to sort, filter, and aggregate Pandas Series

using methods and functions
• Apply custom functions using conditional logic to

Pandas Series

PANDAS SERIES
Series are Pandas data structures built on top of NumPy arrays

• Series also contain an index and an optional name, in addition to the array of data
• They can be created from other data types, but are usually imported from external sources
• Two or more Series grouped together form a Pandas DataFrame
‘pd’ is the standard alias for the Pandas library
Pandas’ Series function converts Python lists

and NumPy arrays into Pandas Series
The name argument lets you specify a name
The index is an array

of integers starting at
0 by default, but it can
be modified
The series name and data type are stored as well

SERIES PROPERTIES
Pandas Series have these key properties:

• values – the data array in the Series
• index – the index array in the Series
• name – the optional name for the Series (useful for accessing columns in a DataFrame)
• dtype – the data type of the elements in the values array
The index is a
range of integers
from 0 to 10

PANDAS DATA TYPES
Pandas data types mostly expand on their base Python and NumPy equivalents
Numeric: Object / Text:
bool Boolean True/False 8 object Any Python object

int64 (default) Whole numbers 8, 16, 32, 64 string Only contains strings or text
float64 (default) Decimal numbers 8, 16, 32, 64 category Maps categorical data to a numeric array for efficiency
boolean Nullable Boolean True/False 8
Int64 (default) Nullable whole numbers 8, 16, 32, 64 Time Series:
Float64 (default) Nullable decimal numbers 32, 64
*Gray = NumPy data type A single moment in time

datetime64
*Yellow = Pandas data type (January 4, 2015, 2:00:00 PM)
The duration between two dates or times
timedelta64
(10 days, 3 seconds, etc.)
We'll review the nuances of data types
A span of time
in depth when covering DataFrames period
(a day, a week, etc.)

TYPE CONVERSION
You can convert the data type in a Pandas Series by using the .astype() method
and specifying the desired data type (if compatible)
These are integers This converts them to floats
This converts them to Booleans This attempts to convert them to the

(0 is False, others are True) Datetime datatype, but isn’t compatible

ASSIGNMENT: SERIES BASICS
Results Preview
NEW MESSAGE
June 21, 2022
From: Rachel Revenue (Financial Analyst)

Subject: Oil Price Series
Hi there, glad to have you on the team!
I work in the finance department, and I’m working on an

analysis on the impact of oil prices on our sales. Our last
analyst read in oil data and created a NumPy array, can you
convert that to a Pandas Series and report back on properties
of the Series?
Make sure to include name, dtype, size, index, then take the
mean of the values array. Finally, convert the series to an
integer data type and recalculate the mean.
Thanks!
section02_Series.ipynb

SOLUTION: SERIES BASICS
Solution Code
NEW MESSAGE
June 21, 2022

Subject: Oil Price Series
Hi there, glad to have you on the team!
I work in the finance department, and I’m working on an

analysis on the impact of oil prices on our sales. Our last
analyst read in oil data and created a NumPy array, can you
convert that to a Pandas Series and report back on properties
of the Series?
Make sure to include name, dtype, size, index, then take the
mean of the values array. Finally, convert the series to an
integer data type and recalculate the mean.
Thanks!

THE INDEX
The index lets you easily access “rows” in a Pandas Series or DataFrame
Here we’re using the

default integer index,
which is preferred
You can index and slice Series like

other sequence data types, but we’ll
learn a better method

CUSTOM INDICES
There are cases where it’s applicable to use a custom index for accessing rows
Custom indices can be assigned when

creating the series or by assignment
This will become more relevant

when working with datetimes
(covered later in the course!)

CUSTOM INDICES
There are cases where it’s applicable to use a custom index for accessing rows
You can still index and slice

to retrieve Series values using
the custom indices
Note that slicing custom indices

makes the stop point inclusive

THE ILOC METHOD
The .iloc[] method is the preferred way to access values by their positional index
• This method works even when Series have a custom, non-integer index
• It is more efficient than slicing and is recommended by Pandas’ creators
df.iloc[row position, column position]
Series or DataFrame The row position(s) for the The column position(s) for the
to access values from value(s) you want to access value(s) you want to access
Examples:
• 0 (single row)
• [5, 9] (multiple rows) We’ll use the column position
• [0:11] (range of rows) argument once we start working
with Pandas DataFrames

THE ILOC METHOD
The .iloc[] method is the preferred way to access values by their positional index
• This method works even on Series with a custom, non-integer index
• It is more efficient than slicing and is recommended by Pandas’ creators
Note that this Series

has a custom index
This returns the value in the 3rd position (0-indexed), even

though the custom index for that value is “tea”
This returns the values from the 3rd to the 4th position (stop
is non-inclusive)

THE LOC METHOD
The .loc[] method is the preferred way to access values by their custom labels
df.loc[row label, column label]
Series or DataFrame The custom row index for the The custom column index for
to access values from value(s) you want to access the value(s) you want to access
Examples:
• "pizza" (single row)
• ["mike", "ike"] (multiple rows)
• ["jan":"dec"] (range of rows)

THE LOC METHOD
The .loc[] method is the preferred way to access values by their custom labels
The .loc[] method works even

The custom indices when the indices are integers,
are the labels but if they are custom integers
not ordered from 0 to n-1, the
rows will be returned based on
the labels themselves and NOT
their numeric position
Note that slices are inclusive

when using custom labels

DUPLICATE INDEX VALUES
It is possible to have duplicate index values in a Pandas Series or DataFrame

• Accessing these indices by their label using .loc[] returns all corresponding rows
Note that ‘coffee’ is used as an index value twice Warning! Duplicate index values are
generally not advised, but there are
some edge cases where they are useful
This returns both rows with the same label

RESETTING THE INDEX
You can reset the index in a Pandas Series or DataFrame back to the default
range of integers by using the .reset_index() method
• By default, the existing index will become a new column in a DataFrame
Use drop=True when resetting the

index if you don’t want the
This returns a DataFrame by previous index values stored
default, with the previous index
values stored as a new column

ASSIGNMENT: ACCESSING SERIES DATA
Results Preview
NEW MESSAGE
June 21, 2022

Subject: Oil Price Series w/Dates
Thanks for picking up this work, but this data isn’t really useful
without dates since I need to understand trends over time to
improve my forecasts.
Can you set the date series to be the index?
Then, take the mean of the first 10 and last 10 prices. After
that, can you grab all oil prices from January 1st, 2017 to
January 7th, 2017 and revert the index of this slice back to
integers?
Thanks!

SOLUTION: ACCESSING SERIES DATA
Solution Code
NEW MESSAGE
June 21, 2022

Subject: Oil Price Series w/Dates
Thanks for picking up this work, but this data isn’t really useful
without dates since I need to understand trends over time to
improve my forecasts.
Can you set the date series to be the index?
Then, take the mean of the first 10 and last 10 prices. After
that, can you grab all oil prices from January 1st, 2017 to
January 7th, 2017 and revert the index of this slice back to
integers?
Thanks!

FILTERING SERIES
You can filter a Series by passing a logical test into the .loc[] accessor (like arrays!)
This returns all rows from sales_series

with a value greater than 0
This uses a mask to store complex

logic and returns all rows from
sales_series with a greater than 0
and an index equal to “coffee”

LOGICAL OPERATORS & METHODS
You can use these operators & methods to create Boolean filters for logical tests
Equal == .eq()
Not Equal != .ne()
Less Than or Equal <= .le()
Less Than < .lt() Python Operator: Pandas Method:

Greater Than or Equal >= .ge()
Greater Than > .gt()
Membership Test in .isin()
Inverse Membership Test not in ~.isin()

LOGICAL OPERATORS & METHODS
You can use these operators & methods to create Boolean filters for logical tests
Equal == .eq()
Not Equal != .ne()
Less Than or Equal <= .le()
Less Than < .lt()
Greater Than or Equal >= .ge()
Greater Than > .gt()
Membership Test in .isin()
Inverse Membership Test not in ~.isin()

The tilde ‘~’ inverts Boolean values!
The Python operators ‘in’ and ‘not in’ won’t work for many
operations, so the Pandas method must be used

SORTING SERIES
You can sort Series by their values or their index

1. The .sort_values() method sorts a Series by its values in ascending order
Specify ascending=False
to sort in descending order
2. The .sort_index() method sorts a Series by its index in ascending order

ASSIGNMENT: SORTING & FILTERING SERIES
Results Preview
NEW MESSAGE
June 22, 2022

Subject: Oil Price Anomalies
Hi again, your work has been super helpful already!
I need to look at this data from a few more angles.
First, can you get me the 10 lowest prices from the data,
sorted by date, starting with the most recent and ending with
the oldest?
After that, return to the original data. I’ve provided a list of

dates I want to narrow down to, and I also want to look only at
prices less than or equal to 50 dollars per barrel.
Thanks!

SOLUTION: SORTING & FILTERING SERIES
Solution Code
NEW MESSAGE
June 22, 2022

Subject: Oil Price Anomalies
Hi again, your work has been super helpful already!
I need to look at this data from a few more angles.
First, can you get me the 10 lowest prices from the data,
sorted by date, starting with the most recent and ending with
the oldest?
After that, return to the original data. I’ve provided a list of

dates I want to narrow down to, and I also want to look only at
prices less than or equal to 50 dollars per barrel.
Thanks!

ARITHMETIC OPERATORS & METHODS
You can use these operators & methods to perform numeric operations on Series
Addition + .add()
.sub(),
Subtraction -
.subtract()
.mul(),
Multiplication *
.multiply()
These both add two to every row
.div(),
Division / .truediv(),
.divide()
Floor Division // .floordiv()
Modulo % .mod() This uses string arithmetic to add a dollar sign,

converts to float to add decimals (cents), then
Exponentiation ** .pow() converts back to a string

STRING METHODS
The Pandas str accessor lets you access many string methods
• These methods all return a Series (split returns multiple series)
.strip(), .lstrip(), Removes all leading and/or trailing characters

.rstrip() (spaces by default)
.upper(), .lower() Converts all characters to upper or lower case
.slice(start:stop:step) Applies a slice to the strings in a Series
.count("string") Counts all instances of a given string The str accessor lets you
access the string methods
.contains("string") Returns True if a given string is found; False if not
.replace("a", "b") Replaces instances of string "a" with string "b"
.split("delimiter", Splits strings based on a given delimiter string, and

expand=True) returns a DataFrame with a Series for each split
.len() Returns the length of each string in a Series This is removing the dollar sign,
then converting to float
.startswith("string"), Returns True if a string starts or ends with given
.endswith("string") string; False if not

ASSIGNMENT: SERIES OPERATIONS
Results Preview
NEW MESSAGE
June 22, 2022

Subject: Sensitivity Analysis
Hey there,
I’m doing some ‘stress testing’ on my models. I want to look at

the financial impact if oil prices were 10% higher and add an
additional two dollars per barrel on top of that.
Once you’ve done that, create a series that represents the

percent difference between each price and the max price.
Finally, extract the month from the string dates in the index,
and store them as an integer.
Thanks!

SOLUTION: SERIES OPERATIONS
Solution Code
NEW MESSAGE
June 22, 2022

Subject: Sensitivity Analysis
Hey there,
I’m doing some ‘stress testing’ on my models. I want to look at

the financial impact if oil prices were 10% higher and add an
additional two dollars per barrel on top of that.
Once you’ve done that, create a series that represents the

percent difference between each price and the max price.
Finally, extract the month from the string dates in the index,
and store them as an integer.
Thanks!

NUMERIC SERIES AGGREGATION
You can use these methods to aggregate numerical Series
.count() Returns the number of items
.first(), .last() Returns the first or last item
.mean(), .median() Calculates the mean or median
.min(), .max() Returns the smallest or largest value
.argmax(), .argmin() Returns the index for the smallest or largest values
.std(), .var() Calculates the standard deviation or variance
.mad() Calculates the mean absolute deviation
.prod() Calculates the product of all the items
.sum() Calculates the sum of all the items
.quantile() Returns a specified percentile, or list of percentiles

CATEGORICAL SERIES AGGREGATION
You can use these methods to aggregate categorical Series
.unique() Returns an array of unique items in a Series
.nunique() Returns the number of unique items
.value_counts() Returns a Series of unique items and their frequency
Specify normalize=True to
return the percentage of total
for each category

ASSIGNMENT: SERIES AGGREGATIONS
Results Preview
NEW MESSAGE
June 23, 2022

Subject: Additional Metrics
Hi again!
I need a few more metrics. Can you calculate the sum and
mean of prices in the month of march? Next, how many prices
did we have in Jan and Feb?
Then, calculate the 10th and 90th percentiles across all data.
Finally, how often did integer dollar values (e.g. 51, 52) occur
in the dataNormalize the results to a percentage.
Thanks!
SOLUTION: SERIES AGGREGATIONS
Solution Code
NEW MESSAGE
June 23, 2022

Hi again!
I need a few more metrics. Can you calculate the sum and
mean of prices in the month of march? Next, how many prices
did we have in Jan and Feb?
Then, calculate the 10th and 90th percentiles across all data.
Finally, how often did integer dollar values (e.g. 51, 52) occur
in the dataNormalize the results to a percentage.
Thanks!
MISSING DATA
Missing data in Pandas is often represented by NumPy “NaN” values

• This is more efficient than Python’s “None” data type
• Pandas treats NaN values as a float, which allows them to be used in vectorized operations
np.nan creates a NaN value

These are rarely created by hand,
and typically appear when reading
in data from external sources
If NaN was not present here,

the data type would be int64
Most operation methods include a

Arithmetic operations performed
‘fill_value’ argument that lets you
on NaN values will return NaN
pass a value instead of NaN

MISSING DATA
Pandas released its own missing data type, NA, in December 2020
• This allows missing values to be stored as integers, instead of needing to convert to float
• This is still a new feature, but most bugs end up converting the data to NumPy’s NaN
pd.NA creates an NA value

Note that if dtype=“Int16”
wasn’t specified, the values
would be stored as objects
At this time, neither np.NaN nor pd.NA are perfect,

but pd.NA functionality should continue to improve,
and having a nullable integer is usually worth it
(more on that in the next section!)

IDENTIFYING MISSING DATA
The .isna() and .value_counts() methods let you identify missing data in a Series
• The .isna() method returns True if a value is missing, and False otherwise
.isna().sum() returns the

count of NaN values
You can use this as a

Boolean mask!
• The .value_counts() method returns unique values and their frequency
Most methods ignore NaN

values, so you need to specify
dropna=False to return the
count of NaN values

HANDLING MISSING DATA
The .dropna() and .fillna() methods let you handle missing data in a Series
• The .dropna() method removes NaN values from your Series or DataFrame
Note that the index has gaps,

so you can use .reset_index() to
restore the range of integers
• The .fillna(value) method replaces NaN values with a specified value

It’s important to be thoughtful and deliberate in how you handle missing data
EXAMPLE Handling missing values from product sales
Do you keep them? Do you remove them? Do you replace them with zeros?
Do you impute them with the mean?

PRO TIP: These operations can dramatically
impact the results of an analysis, so make sure
you understand these impacts and talk to a data
SME to understand why data is missing

ASSIGNMENT: MISSING DATA
Results Preview
NEW MESSAGE
June 27, 2022

Subject: Erroneous Data
Hey,
I just got a promotion thanks to the analysis you helped me

with. I owe you lunch!
I noticed that two prices (51.44, 47.83), were incorrect, so I

had them filled in with missing values. I’m not sure if I did this
correctly. Can you confirm the number of missing values in
the price column? Once you’ve done that, fill the prices in with
the median of the oil price series.
Thanks!

SOLUTION: MISSING DATA
Solution Code
NEW MESSAGE
June 27, 2022

Subject: Erroneous Data
Hey,
I just got a promotion thanks to the analysis you helped me

with. I owe you lunch!
I noticed that two prices (51.44, 47.83), were incorrect, so I

had them filled in with missing values. I’m not sure if I did this
correctly. Can you confirm the number of missing values in
the price column? Once you’ve done that, fill the prices in with
the median of the oil price series.
Thanks!

THE APPLY METHOD
The .apply() method lets you apply custom functions to Pandas Series
• This function will not be vectorized, so it’s not as efficient as native functions
This function applies a 10% discount to prices over 20 Discount applied!
You can also use Lambda

functions for one-off tasks!

THE WHERE METHOD
Pandas’ .where() method lets you manipulate data based on a logical condition
A logical expression that

df.where(logical test, evaluates to True or False
value if False,
Value to return when
inplace=False) the expression is False
Series or DataFrame Whether to perform

to evaluate data from the operation in place
(default is False) Heads up! This is different
from NumPy’s where function

THE WHERE METHOD
This expression returns False if the price is greater than 20,

and the value if false statement for the discount is applied

THE WHERE METHOD
You can use a tilde ‘~’ to invert the Boolean

values and turn this into a “value if True”

CHAINING WHERE
You can chain .where() methods to combine logical expressions
The first where method applies a 90% discount if a price is greater than 20
The second applies a value of 0 when a price is NOT greater than 10

NUMPY VS. PANDAS WHERE
NumPy’s where function is often more convenient & useful than Pandas’ method
Note that this returns a NumPy array that

you’d need to convert into a Pandas Series

ASSIGNMENT: APPLY & WHERE
Results Preview
NEW MESSAGE
June 29, 2022

Hey, our ‘well’ of oil analysis is almost dried up!
Write a function that outputs ‘buy’ if price is less than the

90th percentile and ‘wait’ if it’s not. Apply it to the oil series.
Then, I need to fix two final prices. Create a series that

multiplies price by .9 if the date is ‘2016-12-23’ or ‘2017-05-
10’, and 1.1 for all other dates.
Thanks!

SOLUTION: APPLY & WHERE
Solution Code
NEW MESSAGE
June 29, 2022

Hey, our ‘well’ of oil analysis is almost dried up!
Write a function that outputs ‘buy’ if price is less than the

90th percentile and ‘wait’ if it’s not. Apply it to the oil series.
Then, I need to fix two final prices. Create a series that

multiplies price by .9 if the date is ‘2016-12-23’ or ‘2017-05-
10’, and 1.1 for all other dates.
Thanks!

KEY TAKEAWAYS
Pandas Series add an index & title to NumPy arrays

• Pandas Series form the columns for DataFrames, which we will cover in the next section
The .loc() & .iloc() methods are key in working with Pandas data structures
• These methods allow you to access rows in Series (and later columns in DataFrames), either by their
positional index or by their labels
Pandas & NumPy have similar operations for filtering, sorting & aggregating
• Use built-in Pandas and NumPy functions and methods to take advantage of vectorization, which is
much more efficient than writing for loops in base Python
Pandas lets you easily handle missing data

• It’s important to understand the impact dropping or imputing might have on your analysis, so make
sure you consult an expert about the root cause of missing data

DATAFRAMES
In this section we’ll introduce Pandas DataFrames, the Python equivalent of an Excel or
SQL table which we’ll use to store and analyze data

• Learn to read in DataFrames, explore their contents,
and access data within them
• Manipulate DataFrames by filtering & sorting rows,

handling missing data, and applying Pandas functions
• Understand the different data types in Pandas

DataFrames, as well as how to optimize memory
consumption by converting & downcasting them

THE PANDAS DATAFRAME
DataFrames are Pandas “tables” made up from columns and rows

• Each column of data in a DataFrame is a Pandas Series that shares the same row index
• The column headers work as a column index that contains the Series names
The column index

points to each
individual Series
The row index points
to the corresponding (axis = 1)
row in each Series
(axis = 0)
Each column is a Pandas Series

DATAFRAME PROPERTIES
DataFrames have these key properties:

• shape – the number of rows and columns in a DataFrame (the index is not considered a column)
• index – the row index in a DataFrame, represented as a range of integers (axis=0)
• columns – the column index in a DataFrame, represented by the Series names (axis=1)
• axes – the row and column indices in a DataFrame
• dtypes – the data type for each Series in a DataFrame (they can be different!)
Pandas will try to guess the data

types when creating a DataFrame
(we’ll modify them later!)

CREATING A DATAFRAME
You can create a DataFrame from a Python dictionary or NumPy array by using
the Pandas DataFrame() function
This creates a DataFrame from a Python dictionary

Note that the keys are used as column names

CREATING A DATAFRAME
You’ll more likely create a DataFrame by reading in a flat file (csv, txt, or tsv) with
Pandas read_csv() function
While the read_csv() function has more arguments

for manipulating the data during the import process,
all you need is the file path and name to get started!
PRO TIP: Pandas is a great

alternative to Excel when
dealing with large datasets!

ASSIGNMENT: DATAFRAME BASICS
Results Preview
NEW MESSAGE
July 5, 2022
From: Chandler Capital (Accountant)

Subject: Transactions Data
Hi there,
I heard you did some great work for our finance department. I
need some help analyzing our transactions to make sure there
aren’t any irregularities in the numbers. The data is stored in
transactions.csv.
Can you read this data in and remind me how many rows are
in the data, which columns are in the data, and their
datatypes? More to come soon!
Thanks!
section03_DataFrames.ipynb

SOLUTION: DATAFRAME BASICS
Solution Code
NEW MESSAGE
July 5, 2022

Subject: Transactions Data
Hi there,
I heard you did some great work for our finance department. I
need some help analyzing our transactions to make sure there
aren’t any irregularities in the numbers. The data is stored in
transactions.csv.
Can you read this data in and remind me how many rows are
in the data, which columns are in the data, and their
datatypes? More to come soon!
Thanks!

EXPLORING A DATAFRAME
You can explore a DataFrame with these Pandas methods:
Returns the first n rows of the DataFrame (5 by default) df.head(nrows)
Returns the last n rows of the DataFrame (5 by default) df.tail(nrows)
Returns n rows from a random sample (1 by default) df.sample(n)rows
Returns key details on a DataFrame’s size , columns, and memory usage df.info()
Returns descriptive statistics for the columns in a DataFrame (only numeric

columns by default; use the ‘include’ argument to specify more columns) df.describe(include)

HEAD & TAIL
The .head() and .tail() methods return the top or bottom rows in a DataFrame
• This is a great way to QA data upon import!
This returns the top 5 rows by default
You can specify the number of rows

to return, in this case the bottom 3

SAMPLE
The .sample() method returns a random sample of rows from a DataFrame
This returns 1 row by default
You can specify the number of rows

to return, in this case 5, and set a
random_state to ensure your sample
can be reproduced later in needed

INFO
The .info() method returns details on a DataFrame’s properties and memory usage
Rows & columns

in the DataFrame
Position, name,
and data type for
each column
Memory usage

INFO
The .info() method returns details on a DataFrame’s properties and memory usage
The .info() method will show non-null counts

on a DataFrame with less than ~1.7m rows,
but you can specify show_counts=True to
ensure they are always displayed
This is a great way to quickly identify missing
values – if the non-null count is less than the
total number of rows, then the difference is
the number of NaN values in that column!
(In this case there are none)

DESCRIBE
The .describe() method returns key statistics on a DataFrame’s columns
Only numeric columns by default
Non-null values
Mean & standard

deviation values
Quartile values

DESCRIBE
The .describe() method returns key statistics on a DataFrame’s columns
Use include=“all” to return statistics

for all columns, or choose a specific
data type to include
Note that the .round() method
suppresses scientific notation and
makes the output more readable
Unique values,
most common
value (top), and
its frequency

ASSIGNMENT: EXPLORING DATAFRAMES
Results Preview
NEW MESSAGE
July 7, 2022

Subject: Quick Statistics
Hi there,
Can you dive a bit more deeply into the retail data and check if
there are any missing values?
What about the number of unique dates? I want to make sure

we didn’t leave any out.
Finally, can you report the mean, median, min and max of
“transactions”? I want to check for any anomalies in our data.
Thanks!

SOLUTION: EXPLORING DATAFRAMES
Solution Code
NEW MESSAGE
July 7, 2022

Subject: Quick Statistics
Hi there,
Can you dive a bit more deeply into the retail data and check if
there are any missing values?
What about the number of unique dates? I want to make sure

we didn’t leave any out.
Finally, can you report the mean, median, min and max of
“transactions”? I want to check for any anomalies in our data.
Thanks!

ACCESING DATAFRAME COLUMNS
You can access a DataFrame column by using bracket or dot notation

• Dot notation only works for valid Python variable names (no spaces, special characters, etc.),
and if the column name is not the same as an existing variable or method
PRO TIP: Even though you’ll see many examples of dot notation in use, stick
to bracket notation for single columns of data as it is less likely to cause issues

ACCESSING DATAFRAME COLUMNS
You can use Series operations on DataFrame columns (each column is a Series!)
Number of unique values in a column Mean of values in a column
First 5 unique values in a column with their frequencies Rounded sum of values in a column

ACCESSING DATAFRAME COLUMNS
You can select multiple columns with a list of column names between brackets
• This is ideal for selecting non-consecutive columns in a DataFrame
PRO TIP: Use .loc() to access more than one column

of data – column bracket notation should primarily be
used for creating new columns and quick exploration

ACCESING DATA WITH ILOC
The .iloc() accessor filters DataFrames by their row and column indices
• The first parameter accesses rows, and the second accesses columns
First 5 rows,
all columns

All rows,
2-4 columns

First 5 rows,
2-4 columns

ACCESING DATA WITH LOC
The .loc() accessor filters DataFrames by their row and column labels
All rows,
“date” column
Wrap single columns

This is a Series in brackets to return
a DataFrame

ACCESING DATA WITH LOC
The .loc() accessor filters DataFrames by their row and column labels
All rows, All rows,

“date” & “sales” “date” through
columns “sales” columns
(list of columns) (slice of columns)

ASSIGNMENT: ACCESSING DATA
Results Preview
NEW MESSAGE
July 9, 2022

Subject: A Few More Stats….
Hi there,
I noticed that the first row is the only one from 2013-01-01.
Can you get me a copy of the DataFrame that excludes that

row, and only includes “store_nbr” and “transactions”?
Also, can you report the number of unique store numbers?
Finally, report the total number of transactions in millions.
Thanks!

SOLUTION: ACCESSING DATA
Solution Code
NEW MESSAGE
July 9, 2022

Subject: A Few More Stats….
Hi there,
I noticed that the first row is the only one from 2013-01-01.
Can you get me a copy of the DataFrame that excludes that

row, and only includes “store_nbr” and “transactions”?
Also, can you report the number of unique store numbers?
Finally, report the total number of transactions in millions.
Thanks!

DROPPING ROWS & COLUMNS
The .drop() method drops rows and columns from a DataFrame

• Specify axis=0 to drop rows by label, and axis=1 to drop columns
This returns the first 5 rows of the retail_df

DataFrame without the “id” column
You can specify inplace=True to permanently

remove rows or columns from a DataFrame
PRO TIP: Drop unnecessary columns early in your workflow

to save memory and make DataFrames more manageable
(ideally, they shouldn’t be imported – more on that later!)

DROPPING ROWS & COLUMNS
The .drop() method drops rows and columns from a DataFrame

• Specify axis=0 to drop rows by label, and axis=1 to drop columns
This returns the first 5 rows of the retail_df

DataFrame after removing the first row
Note that the row label is passed as a list
You can pass a range to remove rows with

consecutive labels, in this case 0-4
You’ll typically drop rows via slicing or filtering, but

it’s worth being aware that .drop() can be used as well

IDENTIFYING DUPLICATE ROWS
The .duplicated() method identifies duplicate rows of data

• Specify subset=column(s) to look for duplicates across a subset of columns
If the number of unique values for a column

is less than the total number of rows, then
that column contains duplicate values


The .duplicated() method returns True for the second

row here because it is a duplicate of the first row


Specifying subset=‘product’ will only

look for duplicates in that column
In this case rows 2 and 3 are
duplicates of the first row (“Dairy”)

DROPPING DUPLICATE ROWS
The .drop_duplicates() method drops duplicate rows from a DataFrame

This removed the second row from the product_df

DataFrame, as it is a duplicate of the first row
Note that the row index now has a gap between 0 & 2

DROPPING DUPLICATE ROWS
The .drop_duplicates() method drops duplicate rows from a DataFrame


• subset=“product” will look for
duplicates in the product column
(index 0, 1, and 2 for “Dairy”)
• keep=“last” will keep the final
duplicate row, and drop the rest
• ignore_index=True will reset the
index so there are no gaps

ASSIGNMENT: DROPPING DATA
Results Preview
NEW MESSAGE
July 9, 2022

Subject: Reducing The Data
Hi there,
Can you drop the first row of data this time? A slice is fine for
viewing but we want this permanently removed. Also, drop
the date column, but not in place.
Then, can you get me a DataFrame that includes only the last
row for each of our stores? I want to take a look at the most
recent data for each.
Thanks!

SOLUTION: DROPPING DATA
Solution Code
NEW MESSAGE
July 9, 2022

Subject: Reducing The Data
Hi there,
Can you drop the first row of data this time? A slice is fine for
viewing but we want this permanently removed. Also, drop
the date column, but not in place.
Then, can you get me a DataFrame that includes only the last
row for each of our stores? I want to take a look at the most
recent data for each.
Thanks!

IDENTIFYING MISSING DATA
You can identify missing data by column using the .isna() and .sum() methods
• The .info() method can also help identify null values
The .isna() method returns a

DataFrame with Boolean values
(True for NAs, False for others)
This is a Series The .sum() method adds these for
each column (True=1, False=0)
and returns the summarized results
The difference between the total

entries and non-null values for
each column gives you the missing
values in each

Like with Series, the .dropna() and .fillna() methods let you handle missing data
in a DataFrame by either removing them or replacing them with other values
Use a dictionary
to specify a value
for each column
This drops any row Use subset to drop rows

with missing values with missing values in
specified columns

ASSIGNMENT: MISSING DATA
Results Preview
NEW MESSAGE
July 12, 2022

Subject: Missing Price Data
Hi again,
I was reviewing some of Rachel Revenue’s work on the oil

price data as I was closing the books and I noticed quite a few
missing dates and values, so now I’m concerned…
Can you tell if any dates or prices are missing?
I’d like to see the mean oil price if we fill in the missing values
with 0 and compare it to the mean oil price if we fill them in
with the mean.
Thanks!

SOLUTION: MISSING DATA
Solution Code
NEW MESSAGE
July 12, 2022

Subject: Missing Price Data
Hi again,
I was reviewing some of Rachel Revenue’s work on the oil

price data as I was closing the books and I noticed quite a few
missing dates and values, so now I’m concerned…
Can you tell if any dates or prices are missing?
I’d like to see the mean oil price if we fill in the missing values
with 0 and compare it to the mean oil price if we fill them in
with the mean.
Thanks!

FILTERING DATAFRAMES
You can filter the rows in a DataFrame by passing a logical test into the .loc[]
accessor, just like filtering a Series or a NumPy array
This filters the retail_df DataFrame

and only returns rows where the
date is equal to “2016-10-28”

You can filter the columns in a DataFrame by passing them into the .loc[]
accessor as a list or a slice
Row filter Column filter

This filters the retail_df DataFrame
to the columns selected, and only
returns rows where the date is
equal to “2016-10-28”

You can apply multiple filters by joining the logical tests with an “&” operator
• Try creating a Boolean mask for creating filters with complex logic
The Boolean mask here is filtering

the DataFrame for rows where the
family is “CLEANING” or “DAIRY”,
and the sales are greater than 0

PRO TIP: QUERY
The .query() method lets you use SQL-like syntax to filter DataFrames
• You can specify any number of filtering conditions by using the “and” & “or” keywords
This query filters rows where the

and the sales are greater than 0
Note that you don’t need to call the
DataFrame name repeatedly, saving
keystrokes and making the filter
easier to interpret

PRO TIP: QUERY
The .query() method lets you use SQL-like syntax to filter DataFrames
• You can specify any number of filtering conditions by using the “and” & “or” keywords
• You can reference variables by using the “@” symbol
This query filters rows where the

and the sales are greater than the
avg_sales value (from the variable!)
… output truncated

ASSIGNMENT: FILTERING DATAFRAMES
Results Preview
NEW MESSAGE
July 12, 2022

Subject: Store 25 Deep Dive
I need some quick research on store 25:
• First, calculate the percentage of times ALL stores had

more than 2000 transactions
• Then, calculate the percentage of times store 25 had
more than 2000 transactions, and calculate the sum of
transactions on these days
• Finally, sum the transactions for stores 25 and 3, that
occurred in May or June, and had less than 2000
transactions

SOLUTION: FILTERING DATAFRAMES
Solution Code
NEW MESSAGE
July 12, 2022

Subject: Store 25 Deep Dive
I need some quick research on store 25:
• First, calculate the percentage of times ALL stores had

more than 2000 transactions
• Then, calculate the percentage of times store 25 had
more than 2000 transactions, and calculate the sum of
transactions on these days
• Finally, sum the transactions for stores 25 and 3, that
occurred in May or June, and had less than 2000
transactions

SORTING DATAFRAMES BY INDICES
You can sort a DataFrame by its indices using the .sort_index() method
• This sorts rows (axis=0) by default, but you can specify axis=1 to sort the columns
This creates a sample DataFrame by

filtering rows for the 3 specified product
families, and grabbing 5 random rows
This sorts the sample DataFrame in

descending order by its row index
+ (it sorts in ascending order by default)

SORTING DATAFRAMES BY INDICES
You can sort a DataFrame by its indices using the .sort_index() method
• This sorts rows (axis=0) by default, but you can specify axis=1 to sort the columns
Remember that DataFrame methods

don’t sort in place by default, allowing
you to chain multiple methods together
This sorts the sample DataFrame in

ascending order by its column index,
- + and modifies the underlying values

SORTING DATAFRAMES BY VALUES
You can sort a DataFrame by its values using the .sort_values() method
• You can sort by a single column or by multiple columns
This sorts the sample DataFrame by

the values in the store_nbr column in
- ascending order by default
This sorts the sample DataFrame by the values

in the family column in ascending order, then
- + by the values in the sales column in descending
order within each family
+ -

ASSIGNMENT: SORTING DATAFRAMES
Results Preview
NEW MESSAGE
July 13, 2022

Subject: Sorting help!
Hi there,
Can you get me a dataset that includes the 5 days with the
highest transactions counts? Any similarities between them?
Then, can you get me a dataset sorted by date from earliest to

most recent, but with the highest transactions first and the
lowest transactions last for each day?
Finally, sort the columns in reverse alphabetical order.
Thanks!

SOLUTION: SORTING DATAFRAMES
Solution Code
NEW MESSAGE
July 13, 2022

Subject: Sorting help!
Hi there,
Can you get me a dataset that includes the 5 days with the
highest transactions counts? Any similarities between them?
Then, can you get me a dataset sorted by date from earliest to

most recent, but with the highest transactions first and the
lowest transactions last for each day?
Finally, sort the columns in reverse alphabetical order.
Thanks!

RENAMING COLUMNS
Rename columns in place via assignment using the “columns” property
Simply assign a list with the new column

names using the columns property
Use a list comprehension to clean

or standardize column titles using
methods like .upper()

RENAMING COLUMNS
You can also rename columns with the .rename() method
Use a dictionary to map the new

column names to the old names
Note that the .rename() method

doesn’t rename in place by default,
so you can chain methods together
Use lambda functions to clean

or standardize column titles
using methods like .upper()

REORDERING COLUMS
Reorder columns with the .reindex() method when sorting won’t suffice
Pass a list of the existing columns in

their desired order, and specify axis=1

ASSIGNMENT: MODIFYING COLUMNS
Results Preview
NEW MESSAGE
July 15, 2022

Subject: Cosmetic Changes
Hi again,
Just some quick work, but can you send me the transaction
data with the columns renamed?
I want them looking a bit prettier for my presentation – you

can find more details in the notebook.
Also, could you reorder the columns so date is first, then

transaction count, then store number?
Thanks!

SOLUTION: MODIFYING COLUMNS
Solution Code
NEW MESSAGE
July 15, 2022

Subject: Cosmetic Changes
Hi again,
Just some quick work, but can you send me the transaction
data with the columns renamed?
I want them looking a bit prettier for my presentation – you

can find more details in the notebook.
Also, could you reorder the columns so date is first, then

transaction count, then store number?
Thanks!

ARITHMETIC COLUMN CREATION
You can create columns with arithmetic by assigning them Series operations
• Simply specify the new column name and assign the operation of interest
This creates a new tax_amount column equal to sales * 0.05

This creates a new total column equal to sales + tax_amount
The new columns are added to the

end of the DataFrame by default

BOOLEAN COLUMN CREATION
You can create Boolean columns by assigning them a logical test
This creates a new taxable_category column

with Boolean values – True if the family is not
“BABY CARE”, and False if it is
This creates a new tax_amount column by

leveraging both Boolean logic & arithmetic:
If the family is not “BABY CARE”, then
calculate the sales tax (sales * 0.05 * 1),
otherwise return zero (sales * 0.05 * 0)

ASSIGNMENT: COLUMN CREATION
Results Preview
NEW MESSAGE
July 18, 2022

Subject: Bonus Calculations
Doing some work on bonus estimates and I need help!!!
Create a ‘pct_to_target’ column that divides transactions by

2500 – our transaction target for all stores. Then create a
‘met_target’ column that is True if ‘pct_to_target’ is greater
than or equal to 1, and False if not. Finally, create a
‘bonus_payable’ column that equals 100 if ‘met_target’ is True,
and 0 if not, then sum the total ‘bonus_payable’ amount.
Finally, create columns for month and day of week as integers.

I put some code for the date parts in the notebook.
Thanks!

SOLUTION: COLUMN CREATION
Solution Code
NEW MESSAGE
July 18, 2022

Subject: Bonus Calculations
Doing some work on bonus estimates and I need help!!!
Create a ‘pct_to_target’ column that divides transactions by

2500 – our transaction target for all stores. Then create a
‘met_target’ column that is True if ‘pct_to_target’ is greater
than or equal to 1, and False if not. Finally, create a
‘bonus_payable’ column that equals 100 if ‘met_target’ is True,
and 0 if not, then sum the total ‘bonus_payable’ amount.
Finally, create columns for month and day of week as integers.

I put some code for the date parts in the notebook.
Thanks!

PRO TIP: NUMPY SELECT
NumPy’s select() function lets you create columns based on multiple conditions
• This is more flexible than NumPy’s where() function or Pandas’ .where() method
Specify a set of conditions

and outcomes (choices) for
each condition
Then use np.select and
pass in the conditions, the
choices, and an optional
default outcome if none of
the conditions are met to
the new Sale_Name column
The first condition is met, so

the first choice is returned

ASSIGNMENT: NUMPY SELECT
Results Preview
NEW MESSAGE
July 20, 2022

Subject: Seasonal Bonuses
Create a ‘seasonal_bonus’ column that applies to these dates:

• All days in December
• Sundays in May
• Mondays in August
Call the December bonus ‘Holiday Bonus’, the May bonus

‘Corporate Month’, and the August bonus ‘Summer Special’. If
no bonus applies, the column should display ‘None’.
Finally, calculate the total bonus owed at $100 per day.
Thanks!

SOLUTION: NUMPY SELECT
Solution Code
NEW MESSAGE
July 20, 2022

Subject: Seasonal Bonuses
Create a ‘seasonal_bonus’ column that applies to these dates:

• All days in December
• Sundays in May
• Mondays in August
Call the December bonus ‘Holiday Bonus’, the May bonus

‘Corporate Month’, and the August bonus ‘Summer Special’. If
no bonus applies, the column should display ‘None’.
Finally, calculate the total bonus owed at $100 per day.
Thanks!

MAPPING VALUES TO COLUMNS
The .map() method maps values to a column or an entire DataFrame

• You can pass a dictionary with existing values as the keys, and new values as the values
This creates a new Vegan?

column by mapping the
dictionary keys to the values
in the product column and
returning the corresponding
dictionary values in each row
The dictionary keys will be mapped
to the values in the column selected
Keys Values

MAPPING VALUES TO COLUMNS
The .map() method maps values to a column or an entire DataFrame

• You can pass a dictionary with existing values as the keys, and new values as the values
• You can apply lambda functions (and others!)
This overwrites the price column

by adding a dollar sign to each of
the previous values

PRO TIP: COLUMN CREATION WITH ASSIGN
The .assign() method creates multiple columns at once and returns a DataFrame
• This can be chained together with other data processing methods
To create a column using .assign(),

simply specify the column name and
assign its values as you normally would
(arithmetic, Boolean logic, mapping, etc.)
To create multiple columns using .assign(),

simply separate them using commas
Note that this is chained with .query() at
the end to filter the DataFrame once the
new columns are created

Let’s create some columns for the sample_df DataFrame!

First, a column called tax_amount:

• Equal to sales * 0.05
• Rounded to 2 decimals
• Starting with “$”

Next, a column called on_promotion_flag:

• If onpromotion > 0 return True
• Otherwise, return False

Next, a last column called year:

• Equal to the first 4 characters
on the date column
• Converted to an integer

Finally, return the DataFrame and filter it:

• For rows where family is “DAIRY”

ASSIGNMENT: ASSIGN
Results Preview
NEW MESSAGE
July 23, 2022

Subject: Cleaning Up The Code
Hi there,
Time to clean up our workflow!
Drop the columns that have been created so far (keep only
date, store_number, and transaction count), and recreate
them using the assign method.
Then sum the seasonal bonus owed once again to make sure
the numbers are correct.
Thanks!

SOLUTION: ASSIGN
Solution Code
NEW MESSAGE
July 23, 2022

Subject: Cleaning Up The Code
Hi there,
Time to clean up our workflow!
Drop the columns that have been created so far (keep only
date, store_number, and transaction count), and recreate
them using the assign method.
Then sum the seasonal bonus owed once again to make sure
the numbers are correct.
Thanks!

REVIEW: PANDAS DATA TYPES
Pandas data types mostly expand on their base Python and NumPy equivalents
Numeric: Object / Text:
bool Boolean True/False 8 object Any Python object

int64 (default) Whole numbers 8, 16, 32, 64 string Only contains strings or text
float64 (default) Decimal numbers 8, 16, 32, 64 category Maps categorical data to a numeric array for efficiency
boolean Nullable Boolean True/False 8
Int64 (default) Nullable whole numbers 8, 16, 32, 64 Time Series:
Float64 (default) Nullable decimal numbers 32, 64
*Gray = NumPy data type A single moment in time

datetime
*Yellow = Pandas data type (January 4, 2015, 2:00:00 PM)
The duration between two dates or times
timedelta
(10 days, 3 seconds, etc.)
A span of time
period
(a day, a week, etc.)

THE CATEGORICAL DATA TYPE
The Pandas categorical data type stores text data with repeated values efficiently
• Python maps each unique category to an integer to save space
• As a rule of thumb, only consider this data type when unique categories < number of rows / 2
These are now stored as

0
integers in the backend
1
The categorical data type has some quirks during some data manipulation operations that will
force it back into an object data type , but it’s not something we’ll cover in depth in this course

TYPE CONVERSION
Like Series, you can convert data types in a DataFrame by using the .astype()
method and specifying the desired data type (if compatible)
This creates a new ‘sales_int’ column

by converting ‘sales’ to integers
You can use the .astype() method on

the entire DataFrame and pass a
dictionary with the columns as keys
and the desired data type as values

TYPE CONVERSION
Like Series, you can convert data types in a DataFrame by using the .astype()
method and specifying the desired data type (if compatible)
.astype() will return a ValueError

if the data type is incompatible
But applying cleaning steps

to the data can make it work

PRO TIP: MEMORY OPTIMIZATION
DataFrames are stored entirely in memory, so memory optimization is key in

working with large datasets in Pandas
Memory Optimization Best Practices (in order):

1. Drop unnecessary columns (when possible, avoid reading them in at all)
2. Convert object types to numeric or datetime datatypes where possible
3. Downcast numeric data to the smallest appropriate bit size
4. Use the categorical datatype for columns where the number of unique values < rows / 2
The .memory_usage() method returns the memory

used by each column in a DataFrame (in bytes),
and deep=True provides more accurate results
PRO TIP: A good rule of thumb is to have around

5-10 times the RAM as the size of your DataFrame
The total memory usage is 1,472 bytes

STEP 1: DROP COLUMNS
Dropping unnecessary columns is an easy way to free up significant space

• You may not know which columns are important when reading in a dataset for the first time
• If you do, you can limit the columns you read in to begin with (more on that later!)
Note that the id column

is identical to the index
By dropping the ‘id’ column, around 40

bytes were freed (~4% of memory use)
Note that the memory usage

went down from 1,470 bytes

STEP 2: CONVERTING OBJECT DATA TYPES
Try to convert object data types to numeric or datetime whenever possible
Note that the missing values in ‘price’ and

‘students_enrolled’ are not NaN values
Use memory_usage=“deep”
with the .info() method to get
total memory usage along
with the column data types Text data, usually including dates,
is read in as an object by default
Numeric data is usually read in as 64-bit

(like ‘class_level’) but errors in the data
can cause it to be read in as an object

Note that by converting ‘start_date’ to a

datetime data type, around 295 bytes
were freed (~20% of memory use)
On the other hand, nothing changed by
converting ‘title’ to a string data type

Note that additional methods were chained to

convert ‘price’ and ‘students_enrolled’ to floats:
• “-” was replaced by NaN values on both
• “$” was stripped on ‘price’
This is now only 619 bytes!

STEP 3: DOWNCAST NUMERIC DATA
Integers and floats are cast as 64-bit by default to handle any possible value, but
you can downcast numeric data to a smaller bit size to save space if possible
• 8-bits = -128 to 127
• 16-bits = -32,768 to 32,767
• 32-bits = -2,147,483,648 to 2,147,483,647
• 64-bits = -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
Based on the ranges above, these

columns can be downcast as follows:
• ‘class_level’ to Int8
• ‘price’ to Int16
• ‘students_enrolled’ to Int16

STEP 3: DOWNCAST NUMERIC DATA
Integers and floats are cast as 64-bit by default to handle any possible value, but
you can downcast numeric data to a smaller bit size to save space if possible
Note that, since these are Pandas’

Nullable data types, the NumPy NaN
values are now Pandas NA values
This is now only 539 bytes!

STEP 4: USING CATEGORICAL DATA TYPES
Use the categorical data type if you have string columns where the number of
unique values is less than half of the total number of rows
In this case, there are five unique categories

in five rows, so this just added overhead
(the demo will show the memory reduction)
This went up to 716 bytes!

RECAP: MEMORY OPTIMIZATION
BEFORE AFTER
63% less memory!

A NOTE ON EFFICIENCY
“Premature optimization is the root of all evil” – Don Knuth
• Data analysis is an iterative process

• It’s normal to not know the best or most efficient version of a dataset from the beginning
• Once you understand the data and have an analysis path, then you can optimize
• Efficiency’s importance depends on the use case

• If all you need is a quick analysis, fully optimizing your code will only waste time
• If you’re building a pipeline that will run frequently, efficiency & optimization are critical
• Build efficient habits & workflows

• As you work more with Python and Pandas, take time to review your code and note areas
for improvement – you’ll be able to incorporate better practices next time!

ASSIGNMENT: MEMORY OPTIMIZATION
Results Preview
NEW MESSAGE
July 25, 2022

Subject: Saving Disc Space
Hi,
I’m going to take my work home this weekend, and I’ll be

staying at my grandparents’ house.
They have a very, very old computer, so do you think you can
reduce the memory usage to below 5MB, without losing any
of the information?
Thanks!

SOLUTION: MEMORY OPTIMIZATION
Solution Code
NEW MESSAGE
July 25, 2022

Subject: Saving Disc Space
Hi,
I’m going to take my work home this weekend, and I’ll be

staying at my grandparents’ house.
They have a very, very old computer, so do you think you can
reduce the memory usage to below 5MB, without losing any
of the information?
Thanks!

KEY TAKEAWAYS
Pandas DataFrames are data tables with rows & columns

• They are technically collections of Pandas Series that share an index, and are the primary data structure that
data analysts work with in Python
Use exploration methods to quickly understand the data in a DataFrame

• The head, tail, describe, and info methods let you get a glimpse of the data and its characteristics to identify
the cleaning steps needed
You can easily filter, sort, and modify DataFrames with methods & functions
• DataFrames rows & columns can be sorted by index or values, and filtered using multiple conditions
• Columns can be created with arithmetic or complex logic, and multiple columns can be created with .assign()
Memory optimization is critical in working with large datasets in Pandas

• Once you understand the data, dropping unnecessary columns, converting object data types, downcasting
numerical data types, and using the categorical data types when possible will help save significant memory

AGGREGATING & RESHAPING
In this section we’ll cover aggregating & reshaping DataFrames, including grouping
columns, performing aggregation calculations, and pivoting & unpivoting data

• Group DataFrames by one or more columns and
calculate aggregate statistics by group
• Learn to access multi-index DataFrames and reset

them to return to a single index
• Create Excel-style PivotTables to summarize data
• Melt “wide” tables of data into a “long” tabular form

AGGREGATING DATAFRAMES
You can aggregate a DataFrame column by using aggregation methods (like Series!)
But what if you want multiple aggregate statistics, or summarized statistics by groups?

GROUPING DATAFRAMES
Grouping a DataFrame allows you to aggregate the data at a different level

• For example, transform daily data into monthly, roll up transaction level data by store, etc.
Original DataFrame Grouped by Family
Raw sales data by transaction Total sales by family

GROUPING DATAFRAMES
To group data, use the .groupby() method and specify a column to group by
• The grouped column becomes the index by default
Just specify the columns to group by
This returns a “groupby” object
To return the groups created, you need to

calculate aggregate statistics:
Using single brackets 1. Specify the column for the calculations

returns a Series 2. Apply an aggregation method

GROUPING DATAFRAMES
To group data, use the .groupby() method and specify a column to group by
• The grouped column becomes the index by default
Just specify the columns to group by
This returns a “groupby” object
To return the groups created, you need to

calculate aggregate statistics:
Using double brackets 1. Specify the column for the calculations

returns a DataFrame 2. Apply an aggregation method

ASSIGNMENT: GROUPBY
Results Preview
NEW MESSAGE
August 3, 2022
From: Phoebe Product (Merchandising)

Subject: Top Stores by Transactions
Hi there, it’s Phoebe!
I want to create some custom displays for our busiest stores.
Can you return a table containing the top 10 stores by total

transactions in the data?
Make sure they’re sorted from highest to lowest.
Thanks!
P.S. Let me know if you want to hear my music!
section04_aggregations.ipynb

SOLUTION: GROUPBY
Solution Code
NEW MESSAGE
August 3, 2022

Subject: Top Stores by Transactions
Hi there, it’s Phoebe!
I want to create some custom displays for our busiest stores.
Can you return a table containing the top 10 stores by total

transactions in the data?
Make sure they’re sorted from highest to lowest.
Thanks!
P.S. Let me know if you want to hear my music!

GROUPING BY MULTIPLE COLUMNS
You can group by multiple columns by passing the list of columns into .groupby()
• This creates a multi-index object with an index for each column the data was grouped by
This returns the sum of sales for each

combination of ‘family’ and ‘store_nbr’
This is a multi-index DataFrame

GROUPING BY MULTIPLE COLUMNS
You can group by multiple columns by passing the list of columns into .groupby()
• This creates a multi-index object with an index for each column the data was grouped by
• Specify as_index=False to prevent the grouped columns from becoming indices
This still returns the sum of sales for

each combination of ‘family’ and
‘store_nbr’, but keeps a numeric index

ASSIGNMENT: GROUPBY MULTIPLE COLUMNS
Results Preview
NEW MESSAGE
August 4, 2022

Subject: Transactions by Store and Month
Hi there, it’s Phoebe again!
Taking this analysis a layer deeper...
Can you get me the total transactions by store and month?
Sort the table from first month to last, then by highest

transactions to lowest within each month.
I’ll likely analyze this data further in a dashboard software, but

this will help me set up special seasonal displays.
Thanks!

SOLUTION: GROUPBY MULTIPLE COLUMNS
Solution Code
NEW MESSAGE
August 4, 2022

Subject: Transactions by Store and Month
Hi there, it’s Phoebe again!
Taking this analysis a layer deeper...
Can you get me the total transactions by store and month?
Sort the table from first month to last, then by highest

transactions to lowest within each month.
I’ll likely analyze this data further in a dashboard software, but

this will help me set up special seasonal displays.
Thanks!

MULTI-INDEX DATAFRAMES
Multi-index DataFrames are generally created through aggregation operations

• They are stored as a list of tuples, with an item for each layer of the index

ACCESSING MULTI-INDEX DATAFRAMES
The .loc[] accessor lets you access multi-index DataFrames in different ways:
1. Access rows via the outer index only
All rows from

“AUTOMOTIVE”
to “BEAUTY”
(inclusive)
All rows for “AUTOMOTIVE”
(note that ‘family’ is dropped)

ACCESSING MULTI-INDEX DATAFRAMES
The .loc[] accessor lets you access multi-index DataFrames in different ways:
2. Access rows via the outer & inner indices
All rows for “AUTOMOTIVE” and “12”

(note that ‘family’ and ‘store_nbr’ are dropped)
All rows from “AUTOMOTIVE” and “11”

to “BEAUTY” and “11” (inclusive)
….All rows in Automotive and Baby Care

MODIFYING MULTI-INDEX DATAFRAMES
There are several ways to modify multi-index DataFrames:
Reset the index Swap the index level Drop an index level
Moves the index levels back Changes the hierarchy for Drops an index level from
to DataFrame columns the index levels the DataFrame entirely
PRO TIP: In most cases it’s best to reset the index and
avoid multi-index DataFrames – they’re not very intuitive! Be careful! You may lose
important information

ASSIGNMENT: MULTI-INDEX DATAFRAMES
Results Preview
NEW MESSAGE
August 5, 2022

Subject: Multi-Index Help!
Hey, I’ve looked at your work – folks are really impressed.
Can you help me access rows and columns with multiple

indices? I’ve been struggling with multi-index DataFrames.
Also, I might prefer to just return to an integer-based index

and drop the second column index that got created when I
performed multiple aggregations.
Do you know how to do this?
I attached the notebook and added a few questions in there.

SOLUTION: MULTI-INDEX DATAFRAMES
Solution Code
NEW MESSAGE
August 5, 2022

Subject: Multi-Index Help!
Hey, I’ve looked at your work – folks are really impressed.
Can you help me access rows and columns with multiple

indices? I’ve been struggling with multi-index DataFrames.
Also, I might prefer to just return to an integer-based index

and drop the second column index that got created when I
performed multiple aggregations.
Do you know how to do this?
I attached the notebook and added a few questions in there.

THE AGG METHOD
The .agg() method lets you perform multiple aggregations on a “groupby” object
The .agg() method will

perform the aggregation
on all compatible
columns, in this case
‘sales’ and ‘onpromotion’,
which are numeric

MULTIPLE AGGREGATIONS
You can perform multiple aggregations by passing a list of aggregation functions
This creates two levels in the column index,

one for the original column names, and
another for the aggregations performed

MULTIPLE AGGREGATIONS
You can perform specific aggregations by column by passing a dictionary with

column names as keys, and lists of aggregation functions as values
The calculates the sum and mean for ‘sales’

and the min and max for ‘onpromotion’

NAMED AGGREGATIONS
You can name aggregated columns upon creation to avoid multi-index columns
Specify the new column name

and assign it a tuple with the
column you want to aggregate
and the aggregation to perform
A single column index!

ASSIGNMENT: THE AGG METHOD
Results Preview
NEW MESSAGE
August 6, 2022
By Store:
From: Chandler Capital (Accounting)
Subject: Bonus Rate and Bonus Payable
Hey again,
I’m performing some further analysis on our bonuses. By Month:

Can you create a table that has the average number of days
each store hit the target?
Calculate the total bonuses payable to each store and sort the
DataFrame from highest bonus owed to lowest.
By Weekday:
Then do the same for day of week and month.
Thanks!
NOTE: Only the top 5 rows for each DataFrame are included here

SOLUTION: THE AGG METHOD
Solution Code
NEW MESSAGE
August 6, 2022
By Store:
Subject: Bonus Rate and Bonus Payable
Hey again,
By Month:
I’m performing some further analysis on our bonuses.
Can you create a table that has the average number of days
each store hit the target?
Calculate the total bonuses payable to each store and sort the
DataFrame from highest bonus owed to lowest. By Weekday:
Then do the same for day of week and month.
Thanks!

PRO TIP: TRANSFORM
The .transform() method can be used to perform aggregations without reshaping

• This is useful for calculating group-level statistics to perform row-level analysis
This uses .assign() to create a new DataFrame column,

and .transform() calculates the sum of ‘sales’ by ‘store_nbr’
and applies the corresponding value to each row
The value for rows with store 48 is the same!

ASSIGNMENT: TRANSFORM
Results Preview
NEW MESSAGE
August 6, 2022

Subject: Store Transactions Vs. Average
Hey again,
I need some more data on store performance.
This time I want a column added to the transactions data that

has the average transactions for each store… but I don’t want
to lose the rows for each store/day.
Once you’ve created that column, can you create a new

column that contains the difference between the store’s
average and its transactions on that day?
Thanks!

SOLUTION: TRANSFORM
Solution Code
NEW MESSAGE
August 6, 2022

Hey again,
I need some more data on store performance.
This time I want a column added to the transactions data that

has the average transactions for each store… but I don’t want
to lose the rows for each store/day.
Once you’ve created that column, can you create a new

column that contains the difference between the store’s
average and its transactions on that day?
Thanks!

PIVOT TABLES
The .pivot_table() method let’s you create Excel-style pivot tables
Unlike Excel, Pandas pivot tables don’t have a “filter”

argument”, but you can filter your DataFrame before
pivoting to return a filtered pivot table

PIVOT TABLE ARGUMENTS
The .pivot_table() method has these arguments:

• index: returns a row index with distinct values from the specified column
• columns: returns a column index with distinct values from the specified column
• values: the column, or columns, to perform the aggregations on
• aggfunc: defines the aggregation function, or functions, to perform on the “values”
• margins: returns row and column totals when True (False by default)
This returns distinct ‘family’ values as rows,

distinct ‘store_nbr’ values as columns, and
sums the ‘sales’ for each combination of
‘family’ and ‘str_nbr’ as the values

PIVOT TABLE ARGUMENTS
The .pivot_table() method has these arguments:

• index: returns a row index with distinct values from the specified column
• columns: returns a column index with distinct values from the specified column
• values: the column, or columns, to perform the aggregations on
• aggfunc: defines the aggregation function, or functions, to perform on the “values”
• margins: returns row and column totals when True (False by default)
Specifying margins=True adds row and

column totals based on the aggregation
(the corner represents the grand total)

MULTIPLE AGGREGATION FUNCTIONS
Multiple aggregation functions can be passed to the “aggfunc” argument

• The new values are added as additional columns
The functions are passed as a tuple
There is a column for each store_nbr min and max

(this can create a very wide dataset very quickly)

MULTIPLE AGGREGATION FUNCTIONS
Multiple aggregation functions can be passed to the “aggfunc” argument

• The new values are added as additional columns
Use a dictionary to
apply specific functions
to specific columns

PIVOT TABLES VS. GROUPBY
If the column argument isn’t specified in a pivot table, it will return a table that’s
identical to one grouped by the index columns

PIVOT TABLES VS. GROUPBY
If the column argument isn’t specified in a pivot table, it will return a table that’s
identical to one grouped by the index columns
PRO TIP: Use groupby if you don’t need

columns in the pivot, as you can use named
aggregations to flatten the column index

PRO TIP: HEATMAPS
You can style a DataFrame based on its values to create a heatmap

• Simply chain .style.background_gradient() to your DataFrame and add a “cmap” argument
axis=None adds a red-yellow-green heatmap to the whole table axis=1 adds a red-yellow-green heatmap to each row

MELT
The .melt() method will unpivot a DataFrame, or convert columns into rows

• The original column names
(country, 2000, etc.) are turned
into a single “variable” column
• The values for each original
column are placed on a single
“value” column next to its
corresponding column name
Note that the resulting table isn’t perfect, as .melt()

unpivots a DataFrame around its index, while ideally
you’d want to pivot this around the country values

MELT
Use the “id_vars” argument to specify the column to unpivot the DataFrame by

• The “id_vars” column (country) is
kept in the DataFrame
• The rest of the DataFrame columns
are “melted” around the countries
in matching variable/value pairs

MELT
You can also select the columns to melt and name the “variable” & “value” columns

• id_vars melts the DataFrame
around the “country” column
• value_vars selects 2001, 2002,
and 2003 as the columns to melt
(omitting 2000)
• var_name & value_name set
“year” and “GDP” as column names
instead of “variable” and “value”

ASSIGNMENT: PIVOT & MELT
Results Preview
NEW MESSAGE
August 7, 2022

Hey again,
I need to summarize store numbers 1 through 11 by total

bonus payable for each day of the week.
Can you create a pivot table that has the sum of bonus
payable by day of week? Make sure to filter out any rows that
had 0 bonus payable first and add a heatmap across the rows.
Then unpivot the table so we have one row for each store and
day of the week with the corresponding total owed.
Thanks

SOLUTION: PIVOT & MELT
Solution Code
NEW MESSAGE
August 7, 2022

Hey again,
I need to summarize store numbers 1 through 11 by total

bonus payable for each day of the week.
Can you create a pivot table that has the sum of bonus
payable by day of week? Make sure to filter out any rows that
had 0 bonus payable first and add a heatmap across the rows.
Then unpivot the table so we have one row for each store and
day of the week with the corresponding total owed.
Thanks

KEY TAKEAWAYS
Use the .groupby() method to aggregate a DataFrame by specific columns

• You also need to specify a column of values to aggregate and an aggregate function
Avoid working with multi-index DataFrames whenever possible

• Multi-index DataFrames are created by default when grouping by more than one column
• It’s worth knowing how to access multi-index DataFrames, but it’s advised to avoid them by resetting the index
Use the .agg() method to specify multiple aggregation functions when grouping
• Named aggregations allow you to set intuitive column names and prevent multi-index columns
The .pivot_table() and .melt() methods let you pivot and unpivot DataFrames
• Pandas pivot tables work just like Excel, and make data “wide” by converting unique row values into columns
• With .melt(), you can make “wide” tables “long” in order to analyze the data traditionally

DATA VISUALIZATION
In this section we’ll cover basic data visualization in Pandas, and use the plot method to
create & customize line charts, bar charts, pie charts, scatterplots, and histograms

• Use the Matplotlib API and the .plot() method to
create visualizations from Pandas DataFrames
• Create different chart types for individual and

multiple series of data
• Modify the chart elements and formatting to tell a

clear story with data

THE MATPLOTLIB API
Pandas uses the Matplotlib API to create charts and visualize data
• This is an integration with the main Matplotlib library
This course only covers Pandas’ limited data visualization capabilities, but other full Python
libraries like Matplotlib, Seaborn, and Plotly offer more chart types, options and capabilities

THE PLOT METHOD
You can visualize a DataFrame by using the .plot() method

• This creates a line chart by default, using the row index as the x-axis and plotting each
numerical column as a separate series on the y-axis
Note that ‘date’

was plotted as a
numeric series
The index is used as

the x-axis by default

THE PLOT METHOD
You can visualize a DataFrame by using the .plot() method

• This creates a line chart by default, using the row index as the x-axis and plotting each
numerical column as a separate series on the y-axis
PRO TIP: Add “;” at the end of the

code to remove “<AxesSubplot:>”
when using Jupyter Notebooks

THE PLOT METHOD
You can change the x-axis by setting a different index or using the “x” argument
This sets ‘date’ as the index before plotting the chart This sets ‘date’ as the x-axis in the .plot() parameters
It automatically splits
the date components!

THE PLOT METHOD
You can select series to plot with the .loc[] accessor or using the “y” argument
This filters the ‘grocery’ column before plotting the chart This plots ‘grocery’ values along the y-axis

ASSIGNMENT: THE PLOT METHOD
Results Preview
NEW MESSAGE
August 14, 2022
From: Rachel Revenue (Sr. Financial Analyst)

Subject: Oil Price Trends
Hi there,
I don’t have a great grasp on the trends in oil prices, there are
way too many values to understand from table data.
Can you plot the oil prices with a simple line chart?
Can you Google ‘oil price decline 2014’ to help understand

the big drop in prices?
Thanks!
section05_data_viz.ipynb

SOLUTION: THE PLOT METHOD
Solution Code
NEW MESSAGE
August 14, 2022

Subject: Oil Price Trends
Hi there,
I don’t have a great grasp on the trends in oil prices, there are
way too many values to understand from table data.
Can you plot the oil prices with a simple line chart?
Can you Google ‘oil price decline 2014’ to help understand

the big drop in prices?
Thanks!

CHART FORMATTING
You can modify the chart formatting by using .plot() method arguments
• title = “title” – title to use for the chart
• xlabel = “title” – name to use for the x-axis
• ylabel = “title” – name to use for the y-axis
• color = “color” or “#hexacode” – color(s) to use for the data series
• cmap = “color palette” – preset color palette to apply to the chart
• style = “symbol” – style for the line (dashed, dotted, etc.)
• legend = True/False – adds or removes the legend from the chart
• rot = degrees – degrees to rotate the x-axis labels for readability
• figsize = (width, height) – size for the chart in inches
• grid = True/False – adds gridlines to the chart when True
• subplots = True/False – creates a separate chart for each series when True
• sharex = True/False – each series in the subplot shares the x-axis when True
• sharey = True/False – each series in the subplot shares the y-axis when True
• layout = (rows, columns) – the rows and columns to break the subplots into
For a full list of arguments, visit: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.plot.html *Copyright Maven Analytics, LLC
CHART TITLES
You can add a custom chart title and axis labels to increase understanding
Break each .plot() argument into separate

rows to make the code easier to read

SERIES COLORS
You can modify the series colors by using common color names or hex codes
Note that Python understands basic

colors like “Black”, “Red”, “Green”, etc.

SERIES COLORS
You can modify the series colors by using common color names or hex codes
Hexadecimal codes work too! If you have more than one series,
you can pass the colors in a list
PRO TIP: Sites like Google have

helpful hexadecimal color pickers

COLOR PALETTES
You can also modify the entire color palette for the series in the chart
Default Color Palette:
The “Pastel12” color

palette is applied here
Series colors are applied

in this sequential order
For more on color palettes, visit: https://matplotlib.org/3.5.0/tutorials/colors/colormaps.html *Copyright Maven Analytics, LLC
LINE STYLE
Line charts are solid by default, but you can change the line style if needed
This sets the first series as a dashed

line and the second as dot-dashed
- Solid (default)
-- Dashed
-. Dash-dot
.. Dotted

CHART LEGEND
The “legend” .plot() argument lets you add or remove the legend
• This can be useful in some scenarios, but in others you’ll want to reposition the legend
Pandas tries to place

it in the “best” spot

CHART LEGEND
You can reposition the legend by chaining the .legend() method to .plot() and
specifying its location with the “loc” or “bbox_to_anchor” arguments
best (default)
upper right
upper left
upper center
lower right
lower left
lower center
center right
center left
center

CHART LEGEND
You can reposition the legend by chaining the .legend() method to .plot() and
specifying its location with the “loc” or “bbox_to_anchor” arguments
best (default)
upper right
upper left
upper center
lower right
“bbox_to_anchor” lets you specify
lower left the coordinates for the legend
lower center (it takes some trial and error!)
center right
center left
center

GRIDLINES
You can add gridlines to charts for visual reference
This adds gridlines on the y-axis for line charts,

but other chart types get x and y gridlines

PRO TIP: CHART STYLES
Matplotlib & Seaborn have premade style templates that can be applied to charts
• Once a style is set, it will automatically be applied to all charts
The style is set in advance
You can still customize each chart

This is Seaborn’s
darkgrid style, This is Matplotlib’s
which has better fivethirtyeight style,
font sizing, and a but with dashed lines
background color instead of solid

ASSIGNMENT: FORMATTED CHART
Results Preview
NEW MESSAGE
August 15, 2022

Subject: Oil Price Presentation
Hi there, thanks for helping me visualize the price trends.
Can you format it so it’s more suitable for management?
Our company tends to use “darkgrid” for the style, but I’d also
like a chart title, axis labels, the line to be in black, and to
rename the price column to a more intuitive name.
Finally, consider converting the ‘date’ column to datetime64 –

I’ve heard this will create better labels on the x-axis.
Thanks!

SOLUTION: FORMATTED CHART
Solution Code
NEW MESSAGE
August 15, 2022

Subject: Oil Price Presentation
Hi there, thanks for helping me visualize the price trends.
Can you format it so it’s more suitable for management?
Our company tends to use “darkgrid” for the style, but I’d also
like a chart title, axis labels, the line to be in black, and to
rename the price column to a more intuitive name.
Finally, consider converting the ‘date’ column to datetime64 –

I’ve heard this will create better labels on the x-axis.
Thanks!

SUBPLOTS
You can leverage subplots to create a separate chart for each series
The subplots are stacked in

a single column by default

SUBPLOTS
You can leverage subplots to create a separate chart for each series
There are several subplot options you can use:

• layout lets you specify the rows and columns for the subplots
• sharey & sharex let you set consistent axes across the subplots
• title can map a list of specified titles to each subplot

CHART SIZE
You can adjust the size of the plot or subplots (in inches) using “figsize”
The default size is 6.4 x 4.8 inches This has been resized to 8 x 8 inches

ASSIGNMENT: SUBPLOTS
Results Preview
NEW MESSAGE
August 16, 2022
From: Joey Justin Time (Logistics Clerk)

Subject: Transactions by Store
Hi there, nice to meet you!
I need some help assessing the volumes moving through each

store, so I’ve provided a pivot table with store transactions.
Can you create a 2 x 2 plot the transactions for each of our

stores in a line chart? Make sure they share an axis and
consider making the figure a bit larger.
Which stores have the most transactions? Do they spike at

similar times?
Thanks again!

SOLUTION: SUBPLOTS
Solution Code
NEW MESSAGE
August 16, 2022

Hi there, nice to meet you!
I need some help assessing the volumes moving through each

store, so I’ve provided a pivot table with store transactions.
Can you create a 2 x 2 plot the transactions for each of our

stores in a line chart? Make sure they share an axis and
consider making the figure a bit larger.
Which stores have the most transactions? Do they spike at

similar times?
Thanks again!

CHANGING CHART TYPES
You can change chart types with the “kind” argument or the attribute for each chart
This selects the chart type within the .plot() parameters This chains the chart type as an attribute to .plot()
Note that the arguments live

in the chart attribute now
PREFERRED
COMMON CHART TYPES
LINE CHART (default) BAR CHART (bar or barh)
PIE CHART (pie) HISTOGRAM (hist) SCATTERPLOT (scatter)
For more chart types, visit: https://pandas.pydata.org/docs/user_guide/visualization.html *Copyright Maven Analytics, LLC
LINE CHARTS
Line charts are used for showing trends and changes over time
They are the default chart

type when using .plot()
The x-axis should be continuous!

BAR CHARTS
Bar charts are used for making comparisons with categorical data
You’ll typically aggregate

the data by category first
“bar” will return a vertical

bar chart (or column chart),
and “rot” rotates the labels

BAR CHARTS
Bar charts are used for making comparisons with categorical data
You’ll typically aggregate

the data by category first
“barh” will return a

horizontal bar chart
Note that horizontal bar charts

are sorted in reverse order

GROUPED BAR CHARTS
Plotting multiple series with a bar plot will create a grouped bar chart
The bars are grouped by

the DataFrame index

STACKED BAR CHARTS
Specify “stacked=True” when plotting multiple series to create a stacked bar chart
• This still lets you compare the categories, but also shows the composition of each category
You can set custom labels

using “set_xticklabels”

100% STACKED BAR CHARTS
Calculate percent of total values by index to create a 100% stacked bar chart
• This emphasizes the difference in composition between categories (instead of absolute values)
The values in each

row add up to 100

ASSIGNMENT: BAR CHARTS
Results Preview
NEW MESSAGE
August 17, 2022

Hi there, thanks for your help!
Can we summarize total transactions by sales in a bar chart?
Then, create a horizontal bar chart with the y axis as month,

and the x axis as sales, sorted from January to February.
Some code has been provided in the notebook attached to

create the table for the stacked bar plot.
Thanks!

SOLUTION: BAR CHARTS
Solution Code
NEW MESSAGE
August 17, 2022

Hi there, thanks for your help!
Can we summarize total transactions by sales in a bar chart?
Then, create a horizontal bar chart with the y axis as month,

and the x axis as sales, sorted from January to February.
Some code has been provided in the notebook attached to

create the table for the stacked bar plot.
Thanks!

PIE CHARTS
Pie charts are used for showing composition with categorical data
PRO TIP: Pie charts get a bad rep,

but they can be effective as long as:
• # of categories <= 5
• Slices are sorted
• First slice starts at 0°

SCATTERPLOTS
Scatterplots are used for showing the relationship between numerical series
Just specify the x and y values!
Each dot represents

one row of data

SCATTERPLOTS
You can classify the dots by category by using color
This sets a different color

based on “dayofweek”

ASSIGNMENT: SCATTERPLOTS
Results Preview
NEW MESSAGE
August 18, 2022

Subject: Relationship Between Store Sales
Hi there,
I was curious to see whether stores tend to have high sales at

the same time.
Can you make a scatterplot of the relationship between store

2 and 3 sales?
Then, color the dots by month to check if the relationship is

different across time.
Thanks!

SOLUTION: SCATTERPLOTS
Solution Code
NEW MESSAGE
August 18, 2022

Subject: Relationship Between Store Sales
Hi there,
I was curious to see whether stores tend to have high sales at

the same time.
Can you make a scatterplot of the relationship between store

2 and 3 sales?
Then, color the dots by month to check if the relationship is

different across time.
Thanks!

HISTOGRAMS
Histograms are used for showing the distribution of numerical series

• It divides the data in “bins” and plots the frequency of values that fall into each bin as bars
This is great for seeing the

“shape” of the data, in this
case centered around 70
And identifying outliers!

HISTOGRAMS
Histograms are used for showing the distribution of numerical series

• It divides the data in “bins” and plots the frequency of values that fall into each bin as bars
“alpha” sets the transparency

(0 is invisible, 1 is solid)
Plotting multiple series lets you compare their

distributions and understand drivers of statistics
like mean, min, max and standard deviation

ASSIGNMENT: HISTOGRAMS
Results Preview
NEW MESSAGE
August 20, 2022

Subject: Transaction Distributions
Hi there,
Can you plot the distributions of sales for stores 2 and 3?
I want to see how often store 2 has sales similar to store 3, as

well as if there are any outlier type days.
Thanks!

SOLUTION: HISTOGRAMS
Solution Code
NEW MESSAGE
August 20, 2022

Subject: Transaction Distributions
Hi there,
Can you plot the distributions of sales for stores 2 and 3?
I want to see how often store 2 has sales similar to store 3, as

well as if there are any outlier type days.
Thanks!

SAVING PLOTS
You can save plots as images with the .figure.savefig() method
PRO TIP: You can also take a

screenshot of the plot and save it
Just specify the file name & extension

(bbox_inches=“tight” pads the image
to include the legend)

MORE DATA VISUALIZATION
There is much more to data visualization in Python, and most libraries are built
to be compatible with Pandas Series and DataFrames
For more chart types, customization, and interactivity, consider these libraries:
Matplotlib is what we’ve used to visualize data so far, but there are more chart
types and customization options available (which can make it challenging to work with)
Seaborn is an extension of Matplotlib that is easier to work with and produces

better looking visuals – it also introduces new chart types!
Plotly is used for creating beautiful, interactive visuals, and has a sister app (Dash)
that allows you to easily create dashboards as web applications
Folium is great for creating geospatial maps, especially since it contains features
that are not available in the rest of the libraries

KEY TAKEAWAYS
The .plot() method lets you visualize the data in a DataFrame

• This creates a line chart by default, using the row index as the x-axis and plotting each numerical column as a
separate series on the y-axis, but can easily be modified using .plot() arguments
Use .plot() arguments to modify the chart formatting options

• You can customize the chart titles, axis labels, colors, styles, legend, and more
Consider the chart’s purpose before plotting to communicate effectively

• The charts we covered have a specific purpose – line charts show trends over time, bar charts show comparison
across categories, scatterplots show relationships between series, and histograms show the data’s distribution
There is more to data visualization with Python and Pandas

• Once you’re comfortable with the basics covered in this course, take a look at the other visualization libraries if
you need new chart types, want to add interactivity, or wish to create online dashboards

PROJECT DATA: OVERVIEW
Transactions

Products

ASSIGNMENT: MID-COURSE PROJECT
Key Objectives
NEW MESSAGE
August 23, 2022 1. Read in data from multiple csv files
From: Ross Retail (Head of Analytics) 2. Explore the data (millions of rows!)
Subject: Maven Acquisition Target Data
3. Create new columns to aid in analysis
Hi there, 4. Filter, sort, and aggregate the data to pinpoint
This is going to be a deep dive analysis presented to senior and summarize important information
management at our company. Maven MegaMart is planning
to acquire another retailer to expand our market share. As 5. Build plots to communicate key insights
part of the due diligence process, they’ve sent us over several
tables relating to their customers and sales.
I’ve taken a quick look, but given your great work so far I want
you to lead on this. There are some more detailed questions in
the attached notebook.
Thanks!
section06_midcourse_project.ipynb

TIME SERIES
In this section we’ll cover time series in Pandas, which leverages the datetime data type to
extract date components, group by dates, and perform calculations like moving averages

• Understand the nuances of the datetime data types
in base Python and Pandas
• Apply custom date formats and extract date and time
components from datetime data
• Access portions of time series data and offset time
series for period-by-period comparison
• Create custom time periods and reshape your data
to fit time periods of interest

THE DATETIME DATA TYPE
Base Python’s datetime data type lets you work with time series data
• Datetimes include both a date and time portion by default
The datetime.now() function returns the current

date and time, which is great for comparing time
deltas from the dates in the data to the present
Date components Time components
Datetime data type!

DATETIME64
NumPy’s datetime64 data type lets you work with datetimes in DataFrames
• Pandas will treat dates as “object” datatypes until converted to “datetime64”
Pandas correctly converted the strings to dates!

CONVERSION LIMITATIONS
The .astype() method can convert strings to datetimes, but has some limitations:
1. Any values that Pandas can’t correctly identify as dates will return an error
The missing value here is formatted as a string
2. Ignoring the errors will return the Series as an “object” data type, not datetime64
You can specify errors=“ignore”

to return the DataFrame, but the
conversion to datetime won’t work

TO_DATETIME
The .to_datetime() method is a more flexible option for converting datetimes
To convert using .to_datetime():

• Pass the column(s) to convert
• Specify errors=“coerce” to convert errors to NaT
• Use infer_datetime_format=True to let Pandas
intelligently guess the date format of the text strings
• Or use format= to specify the datetime format
yourself, especially if Pandas can’t guess correctly
This stands for “not a time”
This was converted correctly!

DATETIME CODES
You can use these datetime codes to format dates or extract date components:
Date: Time:
%D 07/14/2022 Zero-padded date (based on OS settings) %T 16:36:30 Zero-padded time (24-hour format)
%Y 2022 Four-digit year %H 16 Hour (24-hour format)
%m 07 Month of the year %I 04 Hour (12-hour format)
%B July Full month name (based on OS settings) %p PM AM or PM (based on OS settings)
%d 14 Day of the month %M 36 Minute
% 4 Weekday number (0-6, Sunday-Saturday) %S 30 Second
%A Thursday Full weekday name (based on OS settings)
%U 28 Week of the year (starting on the first Sunday)
%j 195 Day of the year
For a full list of codes, visit: https://strftime.org/ *Copyright Maven Analytics, LLC
FORMATTING DATETIMES
Use datetime codes with the .strftime() method to customize the datetime format
This is a string now
The .dt accessor lets you use .strftime() on datetimes in Pandas DataFrames
You can specify multiple datetime codes

and punctuation to create custom formats
Note that .strftime() returns a string, so assigning

the formatted datetime values to a DataFrame
column reverts the data type to an object
For a full list of codes, visit: https://strftime.org/ *Copyright Maven Analytics, LLC
EXTRACTING DATETIME COMPONENTS
While you can also use the .strftime() method to extract datetime components,
it’s preferred to use the dedicated .dt accessors
For a full list of datetime accessors, visit the official documentation: https://pandas.pydata.org/docs/reference/api/pandas.Series.dt.date.html *Copyright Maven Analytics, LLC
EXTRACTING DATETIME COMPONENTS
While you can also use the .strftime() method to extract datetime components,
it’s preferred to use the dedicated .dt accessors

ASSIGNMENT: DATEPARTS & TO_DATETIME
Results Preview
NEW MESSAGE
August 29, 2022

Subject: Time to Last Date
Hi,
I need some additional transactions data.
Can you create columns for year, month, and day of week?
Then create a column, days_to_acquisition, which represents

the difference between the planned acquisition date of new
stores, ‘2022-10-01’, and the date in the row.
Finally, create a column representing weeks to acquisition.
Thanks
section07_time_series.ipynb

SOLUTION: DATEPARTS & TO_DATETIME
Solution Code
NEW MESSAGE
August 29, 2022

Hi,
I need some additional transactions data.
Can you create columns for year, month, and day of week?
Then create a column, days_to_acquisition, which represents

the difference between the planned acquisition date of new
stores, ‘2022-10-01’, and the date in the row.
Finally, create a column representing weeks to acquisition.
Thanks

TIME DELTAS
Time deltas represent the amount or time, or difference, between two datetimes
• A time delta is returned when subtracting two datetime values
This returns a Series with the timedelta64 data type!
You can use .dt accessors on time

deltas to convert them to integers

TIME DELTAS
Time deltas represent the amount or time, or difference, between two datetimes
• A time delta is returned when subtracting two datetime values

TO_TIMEDELTA & FREQUENCIES
The .to_timedelta() method lets you use frequencies to return timedelta values
Simply specify a number

and the frequency as unit
“D” Day
“W” Week
“H” Hour
“T” Minute
“S” Second
“M” Month end

Because months, quarters, and years have differing lengths, they are sometimes known
“Q” Quarter end as “anchored” frequencies, since they are anchored to the end of calendar periods
They can’t be used in time delta arithmetic, but are useful for things like resampling
“Y” Year end

TIME DELTA ARITHMENTIC
You can use time delta arithmetic to offset dates by a specified period of time

ASSIGNMENT: TIME DELTA ARITHMETIC
Results Preview
NEW MESSAGE
August 29, 2022

Hi again,
I just got word the acquisition date is going to be pushed back

by three weeks.
Can you add three weeks to the days to acquisition column?

Just add 3 to the weeks to acquisition column.
Thanks

SOLUTION: TIME DELTA ARITHMETIC
Solution Code
NEW MESSAGE
August 29, 2022

Hi again,
I just got word the acquisition date is going to be pushed back

by three weeks.
Can you add three weeks to the days to acquisition column?

Just add 3 to the weeks to acquisition column.
Thanks

TIME SERIES INDICES
Use datetimes as the index to allow for intuitive slicing of your DataFrame
• Like with integers, make sure your time index is sorted or you will get very odd results!

TIME SERIES INDICES
Use datetimes as the index to allow for intuitive slicing of your DataFrame
• Like with integers, make sure your time index is sorted or you will get very odd results!

MISSING TIME SERIES DATA
Having a time series index provides methods for fixing missing data beyond .fillna()
Forward-filling and back-filling data

replaces missing values with either the
value from the previous available date,
or the next available date
Interpolating data replaces missing

You need to convert values with a linear approximation
to a float first based on the values from the previous
and next available dates
(15 + 20) / 2 = 17.5

ASSIGNMENT: MISSING TIME SERIES DATA
Results Preview
NEW MESSAGE
August 30, 2022 Mean with Missing Values: 67.71
From: Rachel Revenue (Sr. Financial Analyst) Mean with Forward Fill: 67.67
Mean with Backfill: 67.67
Subject: Missing Oil Prices
Mean with Interpolation: 67.66
Hey rockstar,
Management mostly enjoyed the presentation on oil prices,

but they were concerned about the gaps in the chart. I
explained these were holidays, but it didn’t matter. Can you
fill in the missing values?
Take a look at the mean value for the oil price using forward
fill, backfill, and interpolation. Are they very different?
Then, plot the series with forward fill for the year 2014, the
month of December 2014, and the days from December 1st to
December 15th, 2014.

SOLUTION: MISSING TIME SERIES DATA
Solution Code
NEW MESSAGE
August 30, 2022

Subject: Missing Oil Prices
Hey rockstar,
Management mostly enjoyed the presentation on oil prices,

but they were concerned about the gaps in the chart. I
explained these were holidays, but it didn’t matter. Can you
fill in the missing values?
Take a look at the mean value for the oil price using forward
fill, backfill, and interpolation. Are they very different?
Then, plot the series with forward fill for the year 2014, the
month of December 2014, and the days from December 1st to
December 15th, 2014.

SHIFTING SERIES
You can shift a Series by a specified number of rows using the .shift() method
• This is helpful when working with time series to compare values against previous periods
Series shift forward one row by default,

but you can specify any positive or
negative number to shift the Series by
This is calculating period-over-period

growth by leveraging a date shift:
Current Period
% Growth = −1
Previous Period

PRO TIP: DIFF
The .diff() method calculates the difference between the values in a Series and
those same values shifted a specified number of periods
• This is useful in measuring absolute changes over time
Both methods calculate

the difference between
the current row and the
previous one
.diff() calculates the difference from the

previous row by default, but you can
specify the number of rows as well

ASSIGNMENT: SHIFT & DIFF
Results Preview
NEW MESSAGE
August 31, 2022

Subject: Monthly and Annual Charts
Hello,
I’m looking into a few different year over year trends related
to year over year changes for store number 47.
Can you plot the monthly average transactions in year 2015

vs the monthly average in the year prior?
Then, can you plot year over year growth percent in average
monthly transactions?
Thanks

SOLUTION: SHIFT & DIFF
Solution Code
NEW MESSAGE
August 31, 2022

Hello,
I’m looking into a few different year over year trends related
to year over year changes for store number 47.
Can you plot the monthly average transactions in year 2015

vs the monthly average in the year prior?
Then, can you plot year over year growth percent in average
monthly transactions?
Thanks

AGGREGATING TIME SERIES
You can aggregate time series using the .groupby() method
Calculates the total sales by

month, regardless of the year
Calculates the total

sales by year

RESAMPLING TIME SERIES
Resampling is a special form of time series aggregation that modifies date indices
and fills any gaps to create continuous date values
• Frequencies are used to define the granularity (you can use years, quarters, and months now!)
This modified the index and created

consecutive month values from Jan
2022 to Jan 2023, even though only
4 months have data, and summed the
values for each month using .sum()
Use .transform() to keep the

original index and apply the
group-level aggregations!

ASSIGNMENT: RESAMPLING TIME SERIES
Results Preview
NEW MESSAGE
August 31, 2022

Hey again,
Really quick, can you also get me charts of the average oil
price by month and year!
It will help complete my new presentation to leadership.
Thank you!!!!

SOLUTION: RESAMPLING TIME SERIES
Solution Code
NEW MESSAGE
August 31, 2022

Hey again,
Really quick, can you also get me charts of the average oil
price by month and year!
It will help complete my new presentation to leadership.
Thank you!!!!

ROLLING AGGREGATIONS
Rolling aggregations let you perform calculations on shifting periods of time

• This can be used to calculate things like moving averages, which are helpful in reducing noise
when plotting trends

ASSIGNMENT: ROLLING AGGREGATIONS
Results Preview
NEW MESSAGE
August 31, 2022

Hey,
Thanks for your help on those year over year figures.
Can you plot the 90-day moving average for transactions for
store 47?
I want to present them with less noise than the daily figures.
Thanks!

SOLUTION: ROLLING AGGREGATIONS
Solution Code
NEW MESSAGE
August 31, 2022

Hey,
Thanks for your help on those year over year figures.
Can you plot the 90-day moving average for transactions for
store 47?
I want to present them with less noise than the daily figures.
Thanks!

KEY TAKEAWAYS
The datetime64 data type lets you work with time series in Pandas
• Conversion can be deceptively complicated, so use the .to_datetime() method to manage errors, or explicitly
state the datetime format for Pandas to interpret it correctly
Use datetime codes & accessors to format dates and extract date components
• There are dozens of options, but you only need to memorize the common parts and formats for most business
analysis scenarios – you can always reference the documentation if needed!
Leverage datetime indices to slice time series data intuitively & efficiently
• This also provides alternative methods for fixing missing data, like forward-filling, back-filling, and interpolation
Shifting & aggregation methods let you perform time intelligence calculations
• Shifting and differencing let you calculate period-over-period changes, grouping and resampling let you perform
unique time series summaries, while rolling aggregations let you calculate things like moving averages

IMPORTING & EXPORTING DATA
In this section we’ll cover importing & exporting data in Pandas, including reading in data
from flat files and SQL tables, applying processing steps on import, and writing back out

• Apply data processing steps like converting data
types, setting an index, and handling missing data
during the import process
• Read & write data from different flat files and

multiple Excel worksheets
• Connect to SQL Databases and create DataFrames

from custom SQL queries

READ_CSV REVISITED
The read_csv() function only needs a file path to read in a file, but it also has many
capabilities for preprocessing the data using other arguments:
• file = “path/name.csv” – file path & name to read in (can also be a URL)
• sep = “/” – character used to separate each column of values (default is comma)
• header = 0 – row number to use as the column names (default is “infer”)
• names = [“date”, “sales”] – list of column names to override the existing ones
• index_col = “date” – column name to be used as the DataFrame index
• usecols = [“date”, “sales”] – list of columns to keep in the DataFrame
• dtype = {“date”: “datetime64”, “sales” : “Int32”} – dictionary with column names and data types
• parse_dates = True – converts date strings into datetimes when True
• infer_datetime_format = True – makes parsing dates more efficient
• na_values = [“-”, “#N/A!”] – strings to recognize as NaN values
• nrows = 100 – number of rows to read in
• skiprows = [0, 2] – line numbers to skip (accepts lambda functions)
• converters = {“sales”: lambda x: f"${x}") – dictionary with column names and functions to apply
For a full list of arguments, visit: https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html *Copyright Maven Analytics, LLC
COLUMN NAMES
Pandas will try to infer the column names in your file by default, but there are
several options to override this behavior:
• Specify header=None to keep all the rows in the file as data, and use integers as column names
• Specify header=0 and use the names argument to pass a list of desired column names
The first row is kept

in the DataFrame
The first row is used for the

column headers by default,
and a suffix of “.1” was
added to avoid duplicates The new column
names are used

SETTING AN INDEX
You can set the index for the DataFrame with the “index_col” argument
• Pass a list of column names to create a multi-index DataFrame (not recommended!)
• Specify parse_dates=True to convert index date column to a datetime data type
This is a synonym for datetime64

SELECTING COLUMNS
You can select the columns to read in with the “use_cols” argument
• This can save a lot of processing time and memory

SELECTING ROWS
You can select the rows to read in from the top with the “nrows” argument, and
specify any rows to skip with “skiprows”
This skips these four

odd-numbered rows
Reading in the first few rows

is great for peeking at big
This lambda function skips
files before reading them in
all odd-numbered rows
completely

MISSING VALUES
You can specify strings (or other values) to treat as missing values with the
“na_values” argument
• They are replaced with NumPy NaN values

PARSING DATES
Dates are read in as object data types by default, but you can parse dates with the
“parse_dates” argument to convert them to datetime64
• Specifying infer_datetime_format=True will speed up the date parsing

DATA TYPES
You can set the data type for each column with the “dtype” argument by passing in
a dictionary with column names as keys and desired data types as values
• Get your data into its most efficient format from the start!

PRO TIP: CONVERTERS
You can apply functions to columns of data by using the “converters” argument
• Pass a dictionary with the column names as keys and the functions as values
Assign the functions to variables

to make the code easier to read

ASSIGNMENT: IMPORTING DATA
Results Preview
NEW MESSAGE
September 2, 2022
From: Monica Market (Marketing Analytics)

Subject: Streamlined Transaction Workflow
Hey there!
I saw some of the work you did with Chandler on the

transactions dataset.
I need to use the same data for some marketing analysis.
Can you streamline the code and send it over for analysis?
Thanks!
section08_import_export.ipynb

SOLUTION: IMPORTING DATA
Solution Code
NEW MESSAGE
September 2, 2022
From: Monica Market (Marketing Analytics)

Subject: Streamlined Transaction Workflow
Hey there!
I saw some of the work you did with Chandler on the

transactions dataset.
I need to use the same data for some marketing analysis.
Can you streamline the code and send it over for analysis?
Thanks!

READING TXT FILES
The read_csv() function can also read in .txt files, and other types of flat files
• Simply use the “sep” argument to specify the delimiter
• You can also read in .tsv (tab separated values) files and URLs pointing to text files
This represents a tab
Pandas is looking for

comma separators

READING EXCEL FILES
The read_excel() function is used to read in Excel files in Pandas

• You can specify worksheets by passing the sheet name or position to the “sheet_name” argument
0-indexed!

PRO TIP: APPENDING SHEETS
You can use Pandas’ concat() function to append data from multiple sheets
• We’ll cover combining DataFrames in the next section, but this is a sneak peek!
sheet_name=None makes Pandas read

every sheet and store them as a dictionary,
which is appended using .concat()
ignore_index=True makes sure each row
has a unique index
The DataFrame has the

data from both sheets!

EXPORTING TO FLAT FILES
The to_csv() and to_excel() functions let you export DataFrames to flat files
PRO TIP:
Multiple tabs!

ASSIGNMENT: EXPORTING DATA
Results Preview
NEW MESSAGE
September 2, 2022

Subject: Export Transactions To Excel
Hey again!
As you know, my department works primarily with Excel.
I’d like a copy of the modified transactions data stored in an

Excel workbook, with one sheet for each year in the data.
If you don’t have Excel, then a csv file for each year is ok too.
Thanks!

SOLUTION: EXPORTING DATA
Solution Code
NEW MESSAGE
September 2, 2022

Subject: Export Transactions To Excel
Hey again!
As you know, my department works primarily with Excel.
I’d like a copy of the modified transactions data stored in an

Excel workbook, with one sheet for each year in the data.
If you don’t have Excel, then a csv file for each year is ok too.
Thanks!

CONNECTING TO SQL DATABASES
The SQLAlchemy library lets you connect to SQL databases

• All major SQL implementations can be accessed (MySQL, Oracle, MS-SQL, etc.)
The create_engine() function creates the

database connection, in this case to a local
database, and inspect().get_table_names()
is used to view the database contents
This example shows a connection to an

Oracle database by providing a username
and password, as well as the server location

QUERYING SQL DATABASES
The read_sql() function lets you create a DataFrame from a SQL query
This query is being performed on the SQL

database connection set up previously, and
assigned to a DataFrame named league_df
PRO TIP: In most cases, it’s more

efficient to perform tasks like joins and
filters in your database – databases are
designed for these tasks and tend to have
more resources than your local machine

WRITING TO SQL DATABASES
The to_sql() function lets you create a SQL table from your DataFrame
• This lets you clean data using Pandas before storing it in a SQL Database
• You will likely need permission from your database administrator (DBA) to do this
This creates a new table called “pl_games”

on the existing database connection
• if_exists=“append” will append the
data to the table if it already exists
• index=False is leaving the default
index in the SQL table, but you can
select a column from the DataFrame
to use instead
Now you can query

the table you created!

ADDITIONAL FORMATS
Pandas has functions to read & write these additional formats

• These functions work similarly to read_csv() and to_csv()
Short for Java Script Open Notation, a common format returned by APIs
JSON read_json to_json
(similar to a nested dictionary in Python)
Feather is a relatively new file format - designed to read, write, and store
Feather read_feather to_feather
DataFrames as efficiently as possible
Can be used to read and write data in webpage formats, which makes it
HTML read_html to_html
nice for scraping tables on sites like Wikipedia
A serialized storage format that allows for quick reproduction of

Pickle read_pickle to_pickle
DataFrames. Mostly used in Machine Learning workflows
You can convert many Python data types to DataFrames, and can convert
Python Dictionary pd.DataFrame to_dict
them to Python dictionaries with to_dict()

KEY TAKEAWAYS
The read_csv() function is capable of significant data preprocessing

• Once you’ve cleaned your data, take the time to incorporate those steps into the import phase to save time and
memory the next time you work with the it
Pandas lets you easily read in & write to flat files like CSV and Excel
• Just make sure to specify the correct column delimiter for .tsv and .txt files, and the desired Excel worksheet
You can create a DataFrame from any SQL query

• SQLAlchemy lets you create a connection to any SQL database, and the read_sql() function lets you pass a
query into the database to create a DataFrame – this is what real-world Pandas workflows are built on!
Many additional formats can be read in using Pandas

• Check the documentation or Google some examples when encountering a format you haven’t worked with

COMBINING DATAFRAMES
In this section we’ll cover combining multiple DataFrames, including joining data from
related fields to add new columns, and appending data with the same fields to add new rows

• Append the rows for similar DataFrames
• Learn the different join types and their behavior
• Join DataFrames with single & multiple related fields

WHY MULTIPLE DATAFRAMES?
There are many scenarios where working with multiple DataFrames is necessary:
• Relational Databases save a lot of space by not repeating redundant data (normalization)
• Data may come from multiple sources (like complementing company numbers with external data)
• Multiple files can be used to store the same data split by different time periods or groups
These two tables come from the same database, so they

share the “date” and “store_nbr” columns
If you want to calculate transactions and sales by store,
you need to join the tables by their related columns

APPENDING & JOINING
There are two ways to combine DataFrames: appending & joining

• Appending stacks the rows from multiple DataFrames with the same column structure
• Joining adds related columns from one DataFrame to another, based on common values
Appending these two tables Joining these two tables added

with the same columns added the region column based on the
the rows from one to the other matching store values

APPENDING DATAFRAMES
You can append DataFrames with the concat() function

• The columns for the DataFrames must be identical
• pd.concat([df_1, df_2]) will stack the rows from df_2 at the bottom of df_1
pd.concat([df1, df2, df3, …] )

ASSIGNMENT: APPENDING DATAFRAMES
Results Preview
NEW MESSAGE
September 5, 2022

Subject: Concatenating DataFrames?
Hey again,
I’ve been looking at the data you sent to Chandler.
Looks great, but I am working on an analysis for 2014-2015.
Can you send me a code snippet to combine these tables?
Thanks!
section09_joining_dfs.ipynb

SOLUTION: APPENDING DATAFRAMES
Solution Code
NEW MESSAGE
September 5, 2022

Subject: Concatenating DataFrames?
Hey again,
I’ve been looking at the data you sent to Chandler.
Looks great, but I am working on an analysis for 2014-2015.
Can you send me a code snippet to combine these tables?
Thanks!
section09_joining_dfs.ipynb

JOINING DATAFRAMES
You can join DataFrames with the merge() function

• The DataFrames must share at least one column to match the values between them
“Right” DataFrame
left_df.merge(right_df, to join with “Left”
how, Type of join
left_on, Column(s) on “Left”

right_on) DataFrame to join by
“Left” DataFrame Column(s) on “Right”

to join with “Right” DataFrame to join by
For a full list of arguments, visit: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html *Copyright Maven Analytics, LLC
JOINING DATAFRAMES
You can join DataFrames with the merge() function

• The DataFrames must share at least one column to match the values between them
EXAMPLE Adding the “transactions” by date and store number to the family-level sales table
What would be the merge() arguments to join these two DataFrames?

• The left DataFrame is “item_sales”
• The right DataFrame is “store_transactions”
• The left columns to join by are “date” and “store_nbr”
• The right columns to join by are “date” and “store_nbr”
• What about the join type?
Joining these tables only on “date” or “store_nbr”

will lead to incorrect matches if either “date” or
“store_nbr” has more than 1 unique value

TYPES OF JOINS
There are 4 primary types of joins: Inner, Left, Right, and Outer
• Inner and Left joins are the most common, while Right is rarely used
Returns records that exist in BOTH tables, and

excludes unmatched records from either table
how=“inner”
Returns ALL records from the LEFT table, and

how=“left”
any matching records from the RIGHT table
Returns ALL records from the RIGHT table, and

how=“right”
any matching records from the LEFT table
Returns ALL records from BOTH tables,

including non-matching records how=“outer”

TYPES OF JOINS
Left Table Right Table
n=5
n=10
Left Join Inner Join Outer Join
n=6
n=10
n=11
EXAMPLE: INNER JOIN
Left Table Inner Join
Right Table
An inner join between these two DataFrames discards all rows

where “store_nbr” and “date” don’t match
Because ‘2016-01-01’ was NOT in transactions, it is NOT
included in the joined table, which leads to a reduced row count

EXAMPLE: LEFT JOIN
Left Table Left Join
Right Table
A left join will preserve all rows in the left table, and simply
includes NaN values in “transactions” when no match is found
PRO TIP: If you’re performing a join between two tables for the
first time, start with a left join to grasp the potential data loss from
an inner join and decide which works best for your analysis

ASSIGNMENT: JOINING DATAFRAMES
Results Preview
NEW MESSAGE
September 10, 2022
From: Joey Justin Time (Logistics)

Subject: Sales Patterns
Hey there!
I’m trying to get a better handle on sales patterns at our

stores, as we’re looking at opening more regional distribution
centers for faster delivery.
Can you join retail.csv with stores.csv?
Once you have that, plot total average sales by city, the sum
of sales by “type” over time, and a stacked bar chart with sales
by month, with “type” as the “stacks”.
Thanks!

SOLUTION: JOINING DATAFRAMES
Solution Code
NEW MESSAGE
September 10, 2022
From: Joey Justin Time (Logistics)

Subject: Sales Patterns
Hey there!
I’m trying to get a better handle on sales patterns at our

stores, as we’re looking at opening more regional distribution
centers for faster delivery.
Can you join retail.csv with stores.csv?
Once you have that, plot total average sales by city, the sum
of sales by “type” over time, and a stacked bar chart with sales
by month, with “type” as the “stacks”.
Thanks!

THE JOIN METHOD
The .join() method can join two DataFrames based on their index
The “transactions” column was joined based on the index

here, not on matches for “date” and “store_nbr”
(rsuffix=“2” lets Pandas know not to include the first two
columns from the right DataFrame – the are duplicates)

THE JOIN METHOD
You can also join based on indices with .merge()

• Specify left_index=True and right_index=True
PRO TIP: You will see examples of joining tables with the join
method, but it’s rare to have perfect alignment of DataFrame
indices in practice; since merge() can join on indices as well as
columns, it is the only method you need to know

KEY TAKEAWAYS
Combining DataFrames is a key step in Pandas’ data analysis workflow

• Being able to connect multiple database tables and/or external data sources together is necessary to answer
many questions and drive your analysis further
Use concat() to stack DataFrames with the same column structure

• This is equivalent to a SQL union, and is used to add new rows to a DataFrame
Use merge() to join DataFrames by their related fields

• This is equivalent to a SQL join, and is used to add new columns to a DataFrame
• You can perform left, inner, right, and outer joins depending on the records you’d like to return
Keep an eye on the shape of your data as you join tables

• Is your joined table smaller than one or both of your starting tables? Is it bigger? Is this behavior expected?

Transactions

Products

Households

ASSIGNMENT: FINAL PROJECT
Key Objectives
NEW MESSAGE
September 20, 2022 1. Read in data from multiple csv files
From: Ross Retail (Head of Analytics) 2. Explore the data (millions of rows!)
Subject: Maven Acquisition Target Data
3. Join multiple DataFrames
Hey again, 4. Create new columns to aid in analysis
We’re getting closer to a final decision on the acquisition. 5. Filter, sort, and aggregate the data to pinpoint
Management was impressed by your findings, and we need to and summarize important information
discuss a promotion for you soon.
6. Work with datetime fields to analyze time series
They did have a few more questions we need to answer
before they feel comfortable moving forward. 7. Build plots to communicate key insights
Those questions are in the attached notebook. 8. Optimize the import workflow
Thanks! 9. Write out summary tables for stakeholders
section10_final_project.ipynb

Data Analysis With Python & Pandas

Uploaded by

Data Analysis With Python & Pandas

Uploaded by

DATA ANALYSIS WITH

With Expert Python Instructor Chris Bruehl

*Copyright Maven Analytics, LLC

Additional resources include:

Downloadable PDF to serve as a helpful reference when you’re offline or on the go

*Copyright Maven Analytics, LLC

Work with Pandas DataFrames, the Python equivalent of an Excel or SQL

*Copyright Maven Analytics, LLC

*Copyright Maven Analytics, LLC

Your task is to analyze over 2 million transactions by product, household,

Use Python to:

*Copyright Maven Analytics, LLC

This course covers the core Pandas functionality

We will only cover basic data visualization in Python

We’ll use Jupyter Notebooks as our primary coding environment

You do NOT need to be a Python expert to take this course

1) Go to anaconda.com/products/distribution and click

4) Follow the installation steps

2) Click X on the Anaconda Nucleus pop-up

3) Launch the downloaded Anaconda pkg file

*Copyright Maven Analytics, LLC

1) Go to anaconda.com/products/distribution and click

4) Follow the installation steps

2) Click X on the Anaconda Nucleus pop-up

3) Launch the downloaded Anaconda exe file

*Copyright Maven Analytics, LLC

1) Launch Anaconda Navigator 2) Find Jupyter Notebook and click

*Copyright Maven Analytics, LLC

*Copyright Maven Analytics, LLC

If you close the server window,

Depending on your OS, and method

*Copyright Maven Analytics, LLC

Google Colab is Google’s cloud-based version of Jupyter Notebooks

To create a Colab notebook:

Colab is very similar to Jupyter Notebooks

*Copyright Maven Analytics, LLC

TOPICS WE’LL COVER: GOALS FOR THIS SECTION:

• Apply array indexing, slicing, methods, and functions

• Understand the concepts of vectorization and

*Copyright Maven Analytics, LLC

Custom indices & column

*Copyright Maven Analytics, LLC

The indices, column names,

*Copyright Maven Analytics, LLC

‘np’ is the standard alias for the NumPy library

NumPy’s array function

*Copyright Maven Analytics, LLC

NumPy arrays have these key properties:

NumPy arrays are a ndarray Python data type,

The sales_array has 1 dimension

*Copyright Maven Analytics, LLC

NumPy arrays have these key properties:

Converting a nested list creates a multi-dimensional

The sales_array has 2 dimensions

*Copyright Maven Analytics, LLC

From: Ross Retail (Head of Analytics)

Hi there, welcome to the Maven MegaMart!

Your resume mentions you have basic Python experience.

I know NumPy is foundational, but not much beyond that.

*Copyright Maven Analytics, LLC

From: Ross Retail (Head of Analytics)

Hi there, welcome to the Maven MegaMart!

Your resume mentions you have basic Python experience.

I know NumPy is foundational, but not much beyond that.

*Copyright Maven Analytics, LLC

As an alternative to converting lists, you can create arrays using functions

Changes an array into the specified dimensions, if compatible np.array.reshape(rows, cols)

*Copyright Maven Analytics, LLC

As an alternative to converting lists, you can create arrays using functions

stop is not inclusive

np.zeros((rows, cols), dtype) np.linspace(start, stop, n)

*Copyright Maven Analytics, LLC

Creates a random number generator (the seed is for reproducibility) np.default_rng(seed)

Returns n random numbers from a uniform distribution between 0 and 1 rng.random(n)

Returns n random numbers from a normal distribution with a given mean

'rng' is the standard variable name for