m y

d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
 m y
This course is about

d e
c a
the Python language and built-in types
te A
the standard library
B y
a th
M
becoming an expert Python developer

t
idiomatic Python ©
i g h
p yr
C o
The Zen of Python Tim Peters import this PEP 20

m y
e
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
c a d
A
Complex is better than complicated.

te
Flat is better than nested.

y
Sparse is better than dense.
Readability counts.

th B
Special cases aren't special enough to break the rules.
Although practicality beats purity.

M
Errors should never pass silently. a
t ©
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.

i g h
There should be one-- and preferably only one --obvious way to do it.

yr
Although that way may not be obvious at first unless you're Dutch.

p
Now is better than never.

C o
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
 Quick Refresher
m y
d e
Python's type hierarchy
c a
variables
te A
B y
conditionals

a th
functions

© M
loops
h t
yr i g
break, continue and try

o p
C
classes
 Variables and Memory
m y
d e
what are variables? à symbols for memory addresses
c a
te A
memory

B y
Python memory management
a th
à reference counting, garbage collection

© M
mutability
h t
à function arguments, shared references

yr i g
p
what is equality of two objects?

C o
Python memory optimizations à interning, peephole
 Numeric Types
m y
d e
a
integers

Ac
rationals

y te
th B
floats

equality
M a
à binary representations exactness rounding

measures of closeness, approximate equality

t ©
i g h
yr
Decimals à alternative to floats exactness precision rounding

o p
C
complex numbers à cmath standard library
 Numeric Types - Booleans
m y
d e
associated Truth values à every object has one
c a
te A
precedence and short-circuiting

B y
a th
Boolean operators
M
à what they really do

© using in context of associated truth values

h t
yr i g
comparison operators à identity, value equalities ordering

o p
C
 Functions
m y
d e
higher-order functions
c a
te A
y
docstrings and annotations

th B
a
Lambdas

introspection
© M
h t
yr i g
functional programming à map, filter, zip

o p à reducing functions

C à partial functions
 Functions - Arguments
m y
d e
positional arguments
c a
te A
keyword-only arguments

B y
default values à caveats
a th
packing and unpacking
© M
h t
i g
variable positional arguments

yr
o p
variable keyword-only arguments

C
 Functions - Scopes and Closures
m y
d e
c a
global and local scopes

te A
nested scopes
B y
closures
a th
© M
nested closures

h t
yr i g
o p
C
 Decorators
m y
d e
decorators
c a
te A
y
nested decorators

parameterized decorators
th B
M a
©
stacked decorators

h t
i g
class decorators

yr
o p
decorator classes

C
applications à memoization, single dispatch, logging, timing
 Tuples as Data Structures
m y
d e
tuples are not just read-only lists
c a
te A
data structures

B y
packing and unpacking
a th
named tuples
© M
h t
g
augmenting named tuples

yr i
o p
C
 Modules, Packages and Namespaces
m y
d e
what are modules?
c a
te A
y
what are packages?

th B
a
how do the various imports work?

© M
how to manipulate namespaces using packages

h t
yr
zip archives
i g
o p
__main__

C
 Extras
m y
d e
will keep growing over time

c a
important new features of Python 3.6 and later
te A
B y
best practices

a th
© M
random collection of interesting stuff

h t
yr i g
additional resources

o p
send me your suggestions!

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Python Type Hierarchy focus on types covered in this course

m y
d e
Multi-Line Statements and Strings
c a
te A
Naming Conventions
B y
a th
Conditionals

© M
h t
Functions
yr i g
o pWhile
Loops
C For
Break, Continue, Try
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The following is a subset of the Python type hierarchy that we will cover in this course:

m y
d e
Numbers
c a
Integral Non-Integral

te A
y
Integers Floats (c doubles)
Booleans Complex

th B
a
Decimals

M
Fractions

©
h t
yr i g
o p
C
 Collections

m y
d e
c a
A
Sequences Sets Mappings
Mutable Immutable Mutable Immutable

y te
Lists Tuples Sets

th B
Frozen Sets Dictionaries
Strings

M a
t ©
i g h
p yr
C o
Callables Singletons
m y
d e
User-Defined Functions None
c a
Generators
A
NotImplemented

te
Classes

B y
Ellipsis (…)

h
Instance Methods
Class Instances (__call__())
a t
M
Built-in Functions (e.g. len(), open())

©
t
Built-in Methods (e.g. my_list.append(x))

h
yr i g
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Python Program
physical lines of code end with a physical newline character
m y
logical lines of code end with a logical NEWLINE token
d e
tokenized
c a
te A
B y
physical newline vs logical newline

a th
M
sometimes, physical newlines are ignored

©
t
in order to combine multiple physical lines

i g h
into a single logical line of code

p yr
terminated by a logical NEWLINE token

C o
Conversion can be implicit or explicit
Implicit

y
[1, [1, #item 1
Expressions in:

m
2, 2, #item 2

list literals: [ ]
3] 3 #item 3

d e
a
]

tuple literals: ( )
Ac
dictionary literals: { }
y te
th B
def my_func(a,

a
set literals: { } b, #comment

M
c):

©
function arguments / parameters print(a, b, c)

h t
yr i g
supports inline comments
my_func(10, #comment
20, 30)

o p
C
Explicit

y
You can break up statements over multiple lines explicitly, by using the \ (backslash ) character

m
d e
a
Multi-line statements are not implicitly converted to a single logical line.

Ac
if a if a \
y te
and b
and c:
and b \
and c:
th B
M a
t ©
Comments cannot be part of a statement, not even a multi-line statement.

i g h
yr
if a \

o
and c: p
and b \ #comment

C
Multi-Line String Literals

Multi-line string literals can be created using triple delimiters (' single or " double)
m y
d e
'''This is """This is
c a
A
a multi-line string''' a multi-line string"""

y te
h
Be aware that non-visible characters such as newlines,

t
tabs, etc. are actually part of the string – basically B
anything you type.

M a
t ©
You can use escaped characters (e.g. \n, \t), use string formatting, etc.

i g h
yr
A multi-line string is just a regular string.

o p
Multi-line strings are not comments, although they can be used as such,

C
especially with special comments called docstrings.
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Identifier names

m y
are case-sensitive my_var are different identifiers

d e
a
my_Var
ham
Ham
Ac
y te
must follow certain rules
th B
M a
t ©
should follow certain conventions

i g h
p yr
C o
Must
start with underscore ( _ ) or letter ( a-z A-Z )

m y
followed by any number of underscores ( _ ), letters ( a-z A-Z ), or digits ( 0-9 )

d e
var my_var index1 index_1 _var __var
a
__lt__

c
are all legal names

cannot be reserved words:

te A
None True False

B y
and or not

a th
if else elif

© M
t
for while break continue pass

def lambda
i gglobal
hnonlocal return yield

del in
p yr is assert class

try
C
import from
o
except finally

with
raise

as
Conventions

_my_var
This is a convention to indicate "internal use" or "private" objects

m
Objects named this way will not get imported by a statement such as :y
from module import *
d e
a
single underscore

Ac
te
__my_var Used to "mangle" class attributes – useful in inheritance chains

B y
double underscore

a th
__my_var__

© M
Used for system-defined names that have a special
meaning to the interpreter.

h t
Don't invent them, stick to the ones pre-defined by Python!

g
double underscore

yr i __init__

o p
C
x < y x.__lt__(y)
Other Naming Conventions from the PEP 8 Style Guide

Packages short, all-lowercase names. Preferably no underscores. utilities

m y
d e
Modules short, all-lowercase names. Can have underscores.

c a db_utils dbutils

Classes CapWords (upper camel case) convention

te A BankAccount

B y
Functions
h
lowercase, words separated by underscores (snake_case) open_account

a t
Variables
M
lowercase, words separated by underscores (snake_case)

©
account_id

Constants
h t
all-uppercase, words separated by underscores MIN_APR

yr i g
o p
https://www.python.org/dev/peps/pep-0008/ This is a should-read!

C
A foolish consistency is the hobgoblin of little minds
(Emerson)
 m y
memory references
d e
c a
what variables really are

te A
memory management
B y
reference counting
a t h
garbage collection

M
dynamic vs static typing
©
h
mutabilityt and immutability

y r i g
o p shared references

C variable equality
everything is an object
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
 m y
Python Memory Manager

d e
Memory

c a
0x1000
object 1

te A
y
0x1001

0x1002

th B
Memory Address
0x1003

M
0x1004 a
object 2
Heap

t ©0x1005 object 3

i g h …

p yr
o
…

C
 m y
d e
a
my_var_1 = 10 Memory

my_var_1
reference
0x1000 10
Ac
te
0x1000

y
0x1001

B
reference

h
my_var_2 0x1002 hello

t
0x1002

M a 0x1003

0x1004

©
my_var_2 = ‘hello’

t
0x1005

i g h …

yr
my_var_1 references the object at 0x1000

o p
my_var_2 references the object at 0x1002 …

C
 m y
d e
c a
A
In Python, we can find out the memory address
referenced by a variable by using the id() function.

y te
This will return a base-10 number. We can convert this

B
base-10 number to hexadecimal, by using the hex()

h
function.

a t
M
Example

©
a = 10

t
print(hex(id(a)))

h
yr i g
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
 reference
m y
count

d e
0x1000 1210

a
my_var = 10

Ac
te
my_var refe
ren
c

y
0x1
e
0x1000

B
000

h
10

a t e nce

M f er 0
other_var = my_var re 100

©
0x

t
other_var

i g h
p yr Reference Counting

C o Python Memory Manager

Finding the Reference Count

m y
d e
passing my_var to getrefcount() creates

a
sys.getrefcount(my_var)

c
an extra reference!

ctypes.c_long.from_address(address).value
te A
B y
a th
© MHere, we just pass the memory address (an integer),

t
not a reference – does not affect reference count

i g h
p yr
C o
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Circular References

m y
Object A Object B
d e
my_var
var_1 var_2
c a
te A Memory Leak!!

B y
a th
Garbage Collector

© M
- can be controlled programmatically using the gc module

h
- by default it is turned on
t
yr i g
- you may turn it off if you’re sure your code does not create circular references – but beware!!

o p
- runs periodically on its own (if turned on)

C
- you can call it manually, and even do your own cleanup
In general GC works just fine
but, not always…
m y
d e
c a
for Python < 3.4
te A
B y
h
If even one of the objects in the circular reference has a destructor [e.g. __del__() ]

a t
M
the destruction order of the objects may be important

©
but the GC does not know what that order should be

t
i g h
yr
so the object is marked as uncollectable

o p and the objects in the circular reference are not cleaned up memory leak

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Some languages (Java, C++, Swift) are statically typed

m y
d e
a
String myVar = “hello”;
data variable value

Ac
te
type name

String
B y String
0x1000

h
ref
myVar
a t hello

© M
h t
i g
myVar = 10; Does not work! myVar has been declared as a String, and cannot be

yr
assigned the integer value 10 later.

o p
C
myVar = “abc”; This is OK! “abc” is a String – so compatible type and assignment
works.
Python, in contrast, is dynamically typed.

m y
my_var = ‘hello’;

d e
c a
The variable my_var is purely a reference to a string object with value hello.
No type is “attached” to my_var.

te A
B y
h
my_var = 10;

a t
The variable my_var is now pointing to an integer object with value 10.

© M
t
0x1000

h ref String

i g
myVar

yr
hello

p
ref

C o Int

10
0x1234
We can use the built-in type() function to determine
the type of the object currently referenced by a variable.
m y
d e
c a
A
Remember: variables in Python do not have an inherent static type.

te
B y
Instead, when we call type(my_var)

a t h
© M
Python looks up the object my_var
and returns the type of the object
is referencing (pointing to),
at that memory location.

h t
y r i g
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
 m y
e
my_var = 10

int
c
0x1000
a d
my_var
10
te A
B y
my_var = 15
a th int 0x1234

© M 15

h t
yr i g
o p
C
 m y
e
my_var = 10

int
c
0x1000
a d
my_var
10
te A
B y
my_var = my_var + 5
a th int 0x1234

© M 15

h t
yr i g
o p
In fact, the value inside the int objects, can never be changed!
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
 type 0x1000
Consider an object in memory: Changing the data inside the
state
(data)
object is called modifying the

m
internal state of the object. y
d e
c a
A
memory address has not changed

te
B y
BankAccount

a th
0x1000
BankAccount
0x1000

M
my_account
Acct #: 12345 Acct #: 12345

©
Balance: 150 Balance: 500

h t
yr i g
o p internal state (data) has changed

C
Object was mutated
fancy way of saying the internal data has changed
 m y
d e
An object whose internal state can be changed, is called

c a
Mutable
te A
B y
a th
M
An object whose internal state
© Immutable
cannot be changed, is called

h t
y r i g
o p
C
 Examples in Python

m y
d e
Immutable
c a
Mutable

Numbers (int, float, Booleans, etc)

te
Lists A
y
• •

B
• Strings • Sets
•
•
Tuples
Frozen Sets
a th •
•
Dictionaries
User-Defined Classes
• User-Defined Classes

© M
h t
yr i g
o p
C
t = (1, 2, 3) Tuples are immutable: elements cannot be deleted, inserted, or replaced
m y
d e
In this case, both the container (tuple), and all its elements (ints) are immutable

c a
But consider this:

te A
a = [1, 2]

B y
h
Lists are mutable: elements can be deleted, inserted, or replaced
b = [3, 4]

a t
t = (a, b) t = ([1, 2], [3, 4])

© M
a.append(3)

h t
b.append(5)

yr i g
t = ([1, 2, 3], [3, 4, 5])

p
In this case, although the tuple is immutable, its elements are not.
o
C
The object references in the tuple did not change
but the referenced objects did mutate!
t = (1, 2, 3)
m y
e
tuple is immutable

c a d
te
these are references to immutable object (int)
A
B y
t = ([1, 2], [3, 4])
a th
tuple is immutable

© M
h t
g
these are references to a mutable object (list)

yr i
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
In Python, Strings (str) are immutable objects.

m y
Once a string has been created, the contents of the object can never be changed

d e
In this code: my_var = ‘hello’
c a
te A
the only way to modify the “value” of my_var is to re-assign my_var to another object

B y
a th 0x1000

my_var

© M hello

h t
yr i g 0x1234

p
abcd

C o
Immutable objects are safe from unintended side-effects

y
but watch out for immutable collection objects that contain mutable objects

m
d e
a
my_var’s reference is passed to process()

def process(s):
Scopes e A
c
s = s + ‘ world’
y t
B
return s
0x1000

a th module scope 0x1000

my_var = ‘hello’

© M my_var hello

process(my_var)
h t
yr i g
o p
print(my_var) process() scope
hello
0x1234

C hello
s
world
Mutable objects are not safe from unintended side-effects

m y
my_lists’s reference is passed to process()

d e
c a
A
def process(lst):

e
lst.append(100)
Scopes
0x1000
y t
my_list = [1, 2, 3]

th B
a
module scope 0x1000
1

M
process(my_list) my_list 2

©
3

print(my_list)
h t 100

yr i g
[1, 2, 3, 100]

o p process() scope
lst

C
Immutable collection objects that contain mutable objects

m y
my_tuple’s reference is passed to process()

d e
c a
A
def process(t):

e
t[0].append(3)
Scopes
0x1000
y t
my_tuple = ([1,2], ‘a’)

th B 0x1000

a
module scope

M
process(my_tuple) my_tuple [1,
[1,2,2]3]

©
‘a’

print(my_tuple)
h t
yr i g
([1, 2, 3], ‘a’)

o p process() scope
t

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The term shared reference is the concept of two variables referencing

y
the same object in memory (i.e. having the same memory address)

e m
d
0x1000
a = 10 a 10
c a
b = a
b

te A
B y
a th
def my_func(v):
© M 0x2000
… t
h t 20
t = 20

yr i g v

p
my_func(t)

C o
In fact, the following may surprise you:

0x1000

m y
e
a = 10 a 10
b = 10
b
c a d
te A
y
0x23FA
s1 = ‘hello’
s2 = ‘hello’
s1 hello

th B
s2

M a
t ©
In both these cases, Python’s memory manager decides to automatically re-use
the memory references!!

i g h We’ll revisit this again soon

y
Is this even safe?
p r Yes

C o
The integer object 10, and the string object ‘hello’ are immutable
– so it is safe to set up a shared reference
When working with mutable objects we have to be more careful

m y
e
1 0x1000

d
a = [1, 2, 3] a 2
b = a 3

c a
b.append(100)
b 100

te A
B y
th
With mutable objects, the Python memory manager will never create shared references
a
a = [1, 2, 3] a
© M 1
2
0x1000

b = [1, 2, 3]

h t 3

yr i g 1 0xF354

p
b 2

C o 3
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
We can think of variable equality in two fundamental ways:

m y
Memory Address Object State (data)
d e
c a
A
is ==
identity operator

y te
equality operator

var_1 is var_2
th B var_1 == var_2

M a
Negation
©
is not
t
!=

i g h
var_1 is not var_2 var_1 != var_2

p yr not(var_1 is var_2) not(var_1 == var_2)

C o
Examples

a is b
m y
e
a = 10
b = a
a == b

c a d
te A
a = ‘hello’ a is b
y
but as we’ll see later, don’t count on it!

B
b = ‘hello’

a
a == b
th
© M
a is b

t
a = [1, 2, 3]

i g h
b = [1, 2, 3] a == b

p yr
C o a = 10
b = 10.0
a is b

a == b
The None object

The None object can be assigned to variables to indicate that they are not set
m y
(in the way we would expect them to be), i.e. an “empty” value (or null pointer)

d e
c a
A
But the None object is a real object that is managed by the Python memory manager

y te
Furthermore, the memory manager will always use a shared reference when assigning

B
a variable to None

a = None
a
a th
M
0x1000
b = None

©
b None
c = None

h t c

yr i g
So we can test if a variable is “not set” or “empty” by comparing it’s memory

p
address to the memory address of None using the is operator

o
Ca is None x = 10
x is None
x is not None
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Throughout this course, we’ll encounter many data types.
•
•
Integers (int)
Booleans (bool)
m y
• Floats (float)
d e
• Strings (str)

c a
A
• Lists (list)

te
• Tuples (tuple)

y
• Sets (set)

B
• Dictionaries (dict)
• None (NoneType)

a th
We’ll also see other constructs:

© M
h t
g
• Operators (+, -, ==, is, …)
• Functions

yr i
p
• Classes
•

C o
Types

and many more…

But the one thing in common with all these things, is that
they are all objects (instances of classes)
def my_func():
m y
• Functions (function) …

d e
a
• Classes (class) [not just instances, but the class itself]

c
0x1000
• Types (type)

A
function
my_func

te
state

This means they all have a memory address!

B y
h
id(my_func)

a t 0x1000

M
As a consequence:

t ©
Any object can be assigned to a variable

i g h
including functions…
my_func is the name of the function

yr
Any object can be passed to a function my_func() invokes the function

o p including functions…

C
Any object can be returned from a function
including functions…
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Important Note:

A lot of what we discuss with memory management, garbage collection and optimizations,
m y
is usually specific to the Python implementation you use.

d e
In this course, we are using CPython, the standard (or reference) Python
c a
implementation (written in C).

te A
B y
But there are other Python implementations out there. These include:

a th
• Jython – written in Java and can import and use any Java class – in fact it even

•
© M
compiles to Java bytecode which can then run in a JVM
IronPython – this one is written in C# and targets .Net (and mono) CLR
•

h t
PyPy – this one is written in RPython (which is itself a statically-typed subset of Python

i g
written in C that is specifically designed to write interpreters)

yr
• and many more…

p
https://wiki.python.org/moin/PythonImplementations

C o
Earlier we saw:

a
0x1000

m y
a = 10 10

d e
a
b = 10
b

Ac
y te
But look at this:

th B
a
0x1001

M
a = 500
a 500
b = 500

©
In this case, although it would

h t be safe for Python to create

g
a shared reference, it does

i
0x1002

yr
not!
b 500

o p
C
What is going on?
Interning: reusing objects on-demand

m y
d e
At startup, Python (CPython), pre-loads (caches) a global list of integers in the range [-5, 256]

c a
A
Any time an integer is referenced in that range, Python will use the cached version of that object

te
y
Singletons Optimization strategy – small integers show up often

th B
a
When we write
a = 10

© M
Python just has to point to the existing reference for 10

h t
But if we write

yr i g
o
a = 257
p
C
Python does not use that global list and a new object
is created every time
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Some strings are also automatically interned – but not all!

m y
d e
As the Python code is compiled, identifiers are interned

c a
•
•
variable names
function names
te
Identifiers:
A
y
• must start with _ or a letter
• class names
• etc.

th B
• can only contain _, letters and numbers

M a
Some string literals may also be automatically interned:

t ©
• string literals that look like identifiers (e.g. ’hello_world’)

g h
• although if it starts with a digit, even though that is not a valid identifier,

i
yr
it may still get interned

o p But don’t count on it!!

C
Why do this?

It’s all about (speed and, possibly, memory) optimization.

m y
d e
Python, both internally, and in the code you write, deals with lots and lots of dictionary type lookups,
on string keys, which means a lot of string equality testing.
c a
Let’s say we want to see if two strings are equal:

te A
a = ‘some_long_string’
B y
b = ‘some_long_string’

a th
Using a == b, we need to compare the two strings character by character

© M
But if we know that ‘some_long_string’ has been interned, then a and b are the same string

h t
if they both point to the same memory address

yr i g
p
In which case we can use a is b instead – which compares two

o
integers (memory address)

C
This is much faster than the character by character comparison
Not all strings are automatically interned by Python

But you can force strings to be interned by using the sys.intern() method.
m y
d e
c a
import sys

te A
y
a is b à True
a = sys.intern(‘the quick brown fox’)
b = sys.intern(‘the quick brown fox’)

th B much faster than a == b

When should you do this?

M a
t ©
• dealing with a large number of strings that could have high repetition

i g h
e.g. tokenizing a large corpus of text (NLP)

yr
• lots of string comparisons

o p
C
In general though, you do not need to intern strings
yourself. Only do this if you really need to.
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
This is another variety of optimizations that can occur at compile time.

m y
d e
Constant expressions
c a
te A
y
numeric calculations

B
24 * 60

th
Python will actually pre-calculate 24 * 60 à 1440

a
short sequences length < 20
© M
h t (1, 2) * 5 à (1, 2, 1, 2, 1, 2, 1, 2, 1, 2)

yr i g 'abc' * 3 à abcabcabc

o p 'hello’ + ‘ world’ à hello world

C
but not 'the quick brown fox' * 10 (more than 20 characters)
Membership Tests: Mutables are replaced by Immutables

When membership tests such as:

m y
if e in [1, 2, 3]:
d e
c a
are encountered, the [1, 2, 3] constant, is replaced by its immutable counterpart

(1, 2, 3) tuple
te A
B y
• lists à tuples
• sets à frozensets
a th
© M
t
Set membership is much faster than list or tuple membership (sets are basically like dictionaries)

h
i
So, instead of writing:

yr g
p
if e in [1, 2, 3]: or if e in (1, 2, 3):

C
write o
if e in {1, 2, 3}:
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
 5
Four main types of numbers:
Python Type

Boolean truth values 0 (False), 1 (True) bool

m y
d e
Integer Numbers (ℤ)
c a
A
0, ±1, ±2, ±3, … int

#
y te
Rational Numbers (ℚ) $
#, $ ∈ ℤ, $ ≠ 0

th B fractions.Fraction

Real Numbers (ℝ) M ! a

0, -1, 0.125, , !, …
float

t © " decimal.Decimal

i g h
yr
Complex Numbers (ℂ) + + -. +, - ∈ ℝ complex

o p
C ℤ⊂ℚ⊂ℝ⊂ℂ
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The int data type

Ex: 0, 10, -100, 100000000, …

m y
How large can a Python int become (positive or negative)?
d e
c a
A
Integers are represented internally using base-2 (binary) digits, not decimal.

te
B y individual bits ( 0 | 1 )

___ ___ ___

1
___
0
___
0
___
1
___
1
___

a th
27 26 25 24 23 22

© M
21 20

128 64 32 16 8

h
4
t 2 1

yr i g
p
1 x 16 + 0 x 8 + 0 x 4 + 1 x 2 + 1 x 1 = 16 + 2 + 1 = 19

C o
(10011)2 = (19)10

Representing the decimal number 19 requires 5 bits

What's the largest (base 10) integer number that can be represented using 8 bits?

Let's assume first that we only care about non-negative integers

m y
1 1 1 1 1 1 1 1
d e
___ ___ ___ ___ ___ ___ ___ ___
a
128 + 64 + 32 + 16 + 8 + 4 + 2 + 1

c
27 26 25 24 23 22 21 20 = 255
= 28 - 1
te A
128 64 32 16 8 4 2 1

B y
a th
If we care about handling negative integers as well, then 1 bit is reserved to represent the sign of

M
the number, leaving us with only 7 bits for the number itself out of the original 8 bits

t ©
The largest number we can represent using 7 bits is 27 – 1 = 127

i g h
So, using 8 bits we are able to represent all the integers in the range [-127, 127]

p yr
Since 0 does not require a sign, we can squeeze out an extra number,

C o
and we end up with the range [-128, 127]
[-27, 27 – 1]
If we want to use 16 bits to store (signed) integers, our range would be:

2(16-1) = 215 = 32,768 Range: [-32,768 … 32,767]

m y
d e
c a
Similarly, if we want to use 32 bits to store (signed) integers, our range would be:

2(32-1) = 231 = 2,147,483,648

te A
Range: [-2,147,483,648 … 2,147,483,647]

B y
th
If we had an unsigned integer type, using 32 bits our range would be:

a
M
[0, 232] = [0 … 4,294,967,296]

In a 32-bit OS:
t ©
i g h
memory spaces (bytes) are limited by their address number à 32 bits

p yr
4,294,967,296 bytes of addressable memory

C o
= 4,294,967,296 / 1024 kB = 4,194,304 kB
= 4,194,304 / 1024 MB = 4,096 MB
= 4,096 / 1024 GB = 4 GB
 So, how large an integer can be depends on how many bits are used to store the number.

m y
Some languages (such as Java, C, …) provide multiple distinct integer data types that use a fixed
number of bits:

d e
byte signed 8-bit numbers -128, …, 127
c a
short signed 16-bit numbers -32,768, …, 32,767
te A
y
Java

int signed 32-bit numbers

th
-231 …, 231 - 1
B
long signed 64-bit numbers
M a
-263 …, 263 - 1

and more…
t ©
i g h
p yr
C o
Python does not work this way

m y
The int object uses a variable number of bits

d e
c a
A
Can use 4 bytes (32 bits), 8 bytes (64 bits), 12 bytes (96 bits), etc. Seamless to us

y te
B
[since ints are actually objects, there is a further fixed overhead per integer]

a th
Theoretically limited only by the amount of memory available

© M
h t
Of course, larger numbers will use more memory

yr i g
and standard operators such as +, *, etc. will run
slower as numbers get larger

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Integers support all the standard arithmetic operations: addition +
subtraction
multiplication
-
*
m y
division /
d e
c
exponents
a **

te A
y
But what is the resulting type of each operation?

int + int à int

th B
int - int à int
M a
t ©
int * int à int

i g h
int ** int

p yr
à int

C
int / int
o à float obviously
but, also
3 / 4
10 / 2
à 0.75
à 5
(float)
(float)
Two more operators in integer arithmetic

First, we revisit long integer division…

155 numerator
m y
4 denominator
d e
155 // 4

c a
A
38

te
4 155
12 155 ÷ 4 = 38 with remainder 3

B y
35
32 put another way:
a th
3

© M
155 = 4 * 38 + 3 155 = 4 * (155 // 4) + (155 % 4)

t
= 4 * 38 + 3
155 % 4

i g h
p yr
o
// is called floor division (div) % is called the modulo operator (mod)

C
and they always satisfy: n = d * (n // d) + (n % d)
What is floor division exactly?

First define the floor of a (real) number

m y
e
The floor of a real number a is the largest (in the standard number order) integer <= a

d
floor(3.14) à 3
c a
floor(1.9999) à 1
te A
floor(2) à 2
B y
a th
M
But watch out for negative numbers!

floor(-3.1) à -4 -4

t © -3

i g h -3.1

p yr
o
So, floor is not quite the same as truncation!

C a // b = floor(a / b)
a = b * (a // b) + a % b

m y
a = 135

d e
a
135 / 4 = 33.75 (33 ¾)

c
b = 4

135 // 4 à 33
te A
B y
h
135 % 4 à 3

a t
M
And, in fact: a = b * (a // b) + a % b

©
h
4 * (135 // 4) + (135 % 4)
t
= 4 * 33 + 3

yr i g
= 132 + 3

o p
C
= 135
Negative Numbers
Be careful, a//b, is not the integer portion of a / b, it is the floor of a / b

For a > 0 and b > 0, these are indeed the same thing
m y
d e
a
But beware when dealing with negative numbers!

a = -135
Ac
b = 4
-135 / 4 = -33.75 (-33 ¾)

y te
th B
-135 // 4

-135 % 4
à -34

à 1
135 // 4

M a
à 33

©
135 % 4 à 3

h t
And, in fact:
i g
a = b * (a // b) + a % b

yr
p
4 * (-135 // 4) + (-135 % 4)

o
C
= (4 * -34) + 1
= -136 + 1
= -135
Expanding this further…

a = 13 b = 4 a = -13 b = 4 a = 13 b = -4 a = -13

m yb = -4

13 / 4 à 3.25 -13 / 4 à -3.25 13 / -4 à -3.25

d e
-13 / -4 à 3.25

c a
13 // 4 à 3 -13 // 4 à -4 13 // -4 à -4

te A -13 // -4 à 3

13 % 4 à 1 -13 % 4 à 3
B
13 % -4 y à -3 -13 % -4 à -1

a th
© M
In each of these cases: a = b * (a // b) + a % b

h t
4 * (13 // 4) + 13 % 4
= 12 + 1 = 13

yr i g
4 * (-13 // 4) + -13 % 4
= -16 + 3 = -13
-4 * (13 // -4) + 13 % -4
= 16 + -3 = 13
-4 * (-13 // -4) + -13 % -4
= -12 + -1 = -13

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
An integer number is an object – an instance of the int class

The int class provides multiple constructors

m y
d e
a = int (10)
c a
a = int (-10)

te A
B y
th
Other (numerical) data types are also supported in the argument of the int constructor:

a
a = int(10.9)

© M truncation: a à 10

t
a = int(-10.9) truncation: a à -10
a = int(True)
i g h aà1

p yr
a = int(Decimal("10.9")) truncation: a à 10

C o
As well as strings (that can be parsed to a number)

a = int("10") a à 10
Number Base

int("123") à (123)10

m y
d e
a
When used with a string, constructor has an optional second parameter: base 2 <= base <= 36

Ac
If base is not specified, the default is base 10 – as in the example above

y te
int("1010", 2) à (10)10

th B
int("1010", base=2) à (10)10

int("A12F", base=16)
M
à (41263)10
a int("534", base=8) à (348)10

int("a12f", base=16)
t ©
à (41263)10 int("A", base=11) à (10)10

i g h
yr
int("B", 11) ValueError: invalid literal for int() with base 11: 'B'

o p
C
Reverse Process: changing an integer from base 10 to another base

built-in functions: bin() bin(10) à '0b1010'

m y
d e
oct() oct(10) à '0o12'
c a
te A
y
hex() hex(10) à '0xa'

th B
The prefixes in the strings help document the base of the number int('0xA', 16) à (10)10

M a
©
These prefixes are consistent with literal integers using a base prefix (no strings attached!)

t
a = 0b1010
i g
a à 10 h
a = 0o12
p yr
a à 10

C
a = 0xA o a à 10
 y
What about other bases? Custom code

n: number (base 10)

e m
b: base (target base)
c a d
te A
y
?
___ ?
___ ?
___ ?
___ ?
___ ?
___ ?
___ ?
___

b7 b6 b5 b4
th b3 Bb2 b1 b0

M a
t ©
i g h n = b * (n // b) + n % b

p yr à n = (n // b) * b + n % b

C o
 ?
___ ?
___ ?
___ ?
___
n = 232 b = 5 too big

y
53 52 51 50
?
___ ?
___ 46
___ 2
___

m
232 = (232 // 5) x 5 + 232 % 5 = 46 x 5 + 2
= [46 x 51] + [2 x 50]

d e
53 52 51 50
= [((46 // 5) x 5 + 46 % 5) x 51] + [2 x 50]

c a
A
too big
= [(9 x 5 + 1) x 51] + [2 x 50]

= [9 x 52] + [1 x 51] + [2 x 50]

y te ?
___ 9
___ 1
___ 2
___

= [((9 // 5) x 5 + 9 % 5) x 52] + [1 x 51] + [2 x 50]

th B 53 52 51 50

= [(1 x 5 + 4) x 52] + [1 x 51] + [2 x 50]

M a
©
1
___ 4
___ 1
___ 2
___

t
= [1 x 53] + [4 x 52] + [1 x 51] + [2 x 50]

h
53 52 51 50

g
div 3rd mod 2nd mod 1st mod

yr i
= [((1 // 5) x 5 + 1 % 5) x 53] + [4 x 52] + [1 x 51] + [2 x 50]

o p
= [(0 x 5 + 1) x 53] + [4 x 52] + [1 x 51] + [2 x 50]

C
= [0 x 54] + [1 x 53] + [4 x 52] + [1 x 51] + [2 x 50]
1
___
53
4
___
52
1
___
51
2
___
50
stop 4th mod 3rd mod 2nd mod 1st mod
Base Change Algorithm

n = base-10 number (>= 0) b = base (>= 2)

m y
d e
a
if b < 2 or n < 0: raise exception n = 232
digits à [1, 4, 1, 2]
if n == 0: return [0] b=5

Ac
digits = [ ]
y te
n = 1485
digits à [5, 12, 13]

B
while n > 0: b = 16
m = n % b
n = n // b
a th
digits.insert(0, m)

© M
h t
i g
This algorithm returns a list of the digits in the specified base b (a representation of n10 in base b)

yr
p
Usually we want to return an encoded number where digits higher than 9 use letters such as A..Z

o
C
We simply need to decide what character to use for the various digits in the base.
Encodings

Typically, we use 0-9 and A-Z for digits required in bases higher than 10
m y
d
But we don't have to use letters or even standard 0-9 digits to encode our number.e
c a
A
We just need to map between the digits in our number, to a character of our choice.

te
0 à 0 0 à 0 0 à a
y
Python uses 0-9 and a-z (case insensitive)

B
h
1 à 1 1 à b

t
1 à 1
and is therefore limited to base <= 36

a
… … …

M
9 à 9 10 à A 9 à i
11 à B 10 à #
10 à A …
t © 11 à !
11 à B

i g h
37 à a …

yr
… 38 à b 36 à *
36 à Z …

o p 62 à z

C
Your choice of characters to represent the digits, is your encoding map
Encodings

y
The simplest way to do this given a list of digits to encode, is to create a string with as many

m
e
characters as needed, and use their index (ordinal position) for our encoding map

base b (>=2)
c a d
map = ' … ' (of length b)

te A
digits = [ … ]
B y
encoding = map[digits[0]] + map[digits[1]] + …
a th
© M
Example: Base 12

h t 10

g
11

yr i
map = '0123456789ABC'
12

o p
C digits = [4, 11, 3, 12]

encoding = '4B3C'
Encoding Algorithm

y
digits = [ … ]
map = ' … '

e m
encoding = ''
c a d
for d in digits:
encoding += map[d] (a += b à a = a + b)
te A
B y
or, more simply:
a th
M
encoding = ''.join([map[d] for d in digits])

©
h t
we'll cover this in much more detail in the section on lists

yr i g
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Constructor
int(a) a is a numeric type such as float, Decimal, Fraction, bool, …
int(s, base=10)
m
s is a string, and base is the base used for the encoded string s
y
d e
Base Change Algorithm Encoding Algorithm
c a
n = base-10 number (>= 0) digits = [ … ]

te A
y
b = base (>= 2) map = ' … '

if b < 2 or n < 0: raise exception

th B
a
encoding = ''

M
if n == 0: return [0] for d in digits:
encoding += map[d]
digits = [ ]
t ©
while n > 0:

i g h or, more simply:

yr
encoding = ''.join([map[d] for d in digits])
m = n % b

p
n = n // b

o
digits.insert(0, m)

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Rational numbers are fractions of integer numbers

Ex:
1
−
22

m y
e
2 7

c a d
A
Any real number with a finite number of digits after the decimal point is also a rational number

0.45
45
0.123456789
y
123456789
te
B
à à
100 109

a th
So
8.3
4
is also rational
4
=
4
©
= M
8.3 83⁄10 83 1 83
× =
10 4 40

h t
8.3
yr i g8.3 83⁄10 83 10 83

p
as is since = = × =
1.4 1.4 14⁄10 10 14 14

C o
The Fraction Class

y
Rational numbers can be represented in Python using the Fraction class in the fractions module

m
d e
from fractions import Fraction
c a
x = Fraction(3, 4)

te A
y = Fraction(22, 7)
z = Fraction(6, 10)
B y
a th
M
Fractions are automatically reduced:

t ©
Fraction(6, 10) à Fraction(3, 5)

i g h
p yr
Negative sign, if any, is always attached to the numerator:

C o
Fraction(1, -4) à Fraction(-1, 4)
Constructors

Fraction(numerator=0, denominator=1)
m y
d e
a
Fraction(other_fraction)

Fraction(float)
Ac
Fraction(decimal)
y te
Fraction(string)
th B
M a
Fraction('10')

t ©
à Fraction(10, 1)

i g h
yr
Fraction('0.125') à Fraction(1, 8)

o p
Fraction('22/7') à Fraction(22, 7)

C
Standard arithmetic operators are supported: +, -, *, /
and result in Fraction objects as well

m y
e
2 1 2 1

d
× = = Fraction(2, 3) * Fraction(1, 2) à Fraction(1, 3)

a
3 2 6 3

2 1 4 3 7
Ac
te
+ = + = Fraction(2, 3) + Fraction(1, 2) à Fraction(7, 6)

y
3 2 6 6 6

th B
a
getting the numerator and denominator of Fraction objects:

M
t ©
x = Fraction(22, 7)

i g h
yr
x.numerator à 22

p
x.denominator

o
à7

C
float objects have finite precision ⇒ any float object can be written as a fraction!

Fraction(0.75) à Fraction(3, 4)
m y
Fraction(1.375) à Fraction(11, 8)
d e
c a
import math
te A
B y
h
x = Fraction(math.pi) à Fraction(884279719003555, 281474976710656)
y = Fraction(math.sqrt(2))
a t
à Fraction(6369051672525773, 4503599627370496)

©
Even though π and 2 are both irrational M
h t
⇒ i g
internally represented as floats

yr
p
finite precision real number

⇒
C o
expressible as a rational number
but it is an approximation
 Converting a float to a Fraction has an important caveat

m y
We'll examine this in detail in a later video on floats

d e
1
has an exact float representation
c a
8

te A
y
Fraction(0.125) à Fraction(1, 8)

th B
3
10
M a
does not have an exact float representation

t ©
h
Fraction(0.3) à Fraction(5404319552844595, 18014398509481984)

yr
format(0.3, '.5f') i g à 0.30000

o p
format(0.3, '.25f') à 0.2999999999999999888977698

C
Constraining the denominator

Given a Fraction object, we can find an approximate equivalent fraction

m y
e
with a constrained denominator
using the limit_denominator(max_denominator=1000000) instance method

c a d
i.e. finds the closest rational (which could be precisely equal)
with a denominator that does not exceed max_denominator
te A
B y
x = Fraction(math.pi)
a th
à Fraction(884279719003555, 281474976710656)

M
3.141592653589793

x.limit_denominator(10)
t ©
à Fraction(22, 7)

i g h 3.142857142857143

p yr
o
x.limit_denominator(100) à Fraction(311, 99)

C
3.141414141414141

x.limit_denominator(500) à Fraction(355, 113)

3.141592920353983
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The float class is Python's default implementation for representing real numbers

m y
the IEEE 754 double-precision binary float, also called binary64
d e
The Python (CPython) float is implemented using the C double type which (usually!) implements

c a
The float uses a fixed number of bytes à 8 bytes

te A
(but Python objects have some overhead too)

à 64 bits

B y à 24 bytes (CPython 3.6 64-bit)

These 64 bits are used up as follows:

a th
sign à 1 bit
© M exponent is -5

exponent à 11 bits
h t
à range [-1022, 1023] 1.5E-5 à 1.5 x 10-5
significant digits
yr i g
à 52 bits à 15-17 significant (base-10) digits

o p
C
significant digits à for simplicity, all digits except leading and trailing zeros

1.2345 1234.5 12345000000 0.00012345 12345e-50 1.2345e10

Representation: decimal

y
Numbers can be represented as base-10 integers and fractions:

0.75 à
% (
+ &'' à 7 ×10"# + 5 × 10"$
e m
2 significant digits
&'

c a d
A
+ ( ,
0.256 + &'' + &''' à 2 ×10 "#
+ 5 × 10 "$ "%
+ 6 × 10 3 significant digits

te
à
&'

B y
h
# % &
123.456 à 1×100 + 2 ×10 + 3 ×1 + + + 6 significant digits
!$ !$$

t
!$$$

a
à 1 x 10$ + 2 ×10# + 3 ×10& + 4 ×10"# + 5 × 10"$ + 6 × 10"%
+

© M
In general: 2 = 4 2' × 10'

h t
'()*

yr i g
p
+

o
sign = 0 for positive
2 = (−1),'-+ 4 2' × 10'

C
sign = 1 for negative
'()*
 +
Some numbers cannot be represented using a finite number of terms
2 = (−1),'-+ 4 2' × 10'

m y '()*

Obviously numbers such as

d e
? = 3.14159 …
c a
2 = 1.4142 …

te A
B y
but even some rational numbers

a th
1
= 0.333̇
© M
t
3

h
3 3 3
= + + + …

i
10 100 1000

yr g
o p
C
Representation: binary
Numbers in a computer are represented using bits, not decimal digits
à instead of powers of 10, we need to use powers of 2
m y
=
1 1
+
d e
a
0.11 . 2 4 = 0.5 + 0.25 !$ = 0.75 !$

c
!$

= 1× 2)! +1× 2). !$

te A
B y
h
Similarly,

0.1101 =
1 1 0 1
+ + +
a t
= 0.5 + 0.25 + 0.0625 = 0.8125

M
. 2 4 8 16 !$ !$
!$

t ©
= 1 × 2)! + 1 × 2). + 0 × 2)" + 1 × 2)#

h
!$

yr i g
This representation is very similar to the one we use with decimal numbers

p
but instead of using powers of 10, we use powers of 2

o
C
a binary representation
+

2 = (−1),'-+ 4 2' × 2'

'()*
 !
The same problem that occurs when trying to represent using a decimal expansion also
"
happens when trying to represent certain numbers using a binary expansion

1
m y
0.1 =
10
d e
Using binary fractions, this number does not have a finite representation

c a
A
0.1 !$ = 0. 0 0011 0011 0011 … .

te
0 0 0 1 1 0 0 1 1 0 0 1 1

y
= + + + + + + + + + + + + + …
base 10

2 4 8 16 32 64 128 256 512 1024 2048 4096 8192

=
1
+
1
+
1
+
1
+
1
+
1
+ …
th B
16 32 256 512 4096 8192
1 1 1
M1 a
1

©
= 0.0625 + + + + + + …
32 256 512 4096 8192
1
h
1
t 1 1
= 0.09375 +

yr i g+ +
256 512 4096 8192
+ + …

p
1 1 1

o
= 0.09765625 + + + + …
512 4096 8192

C
= 0.099609375 +
1
+
4096 8192
1
+ …

= 0.0999755859375 + …
So, some numbers that do have a finite decimal representation,
do not have a finite binary representation,

m y
e
and some do

c a d
0.75 !$ = 0.11 . finite

te A
y
exact float representation

B
0.8125 !$ = 0.1101 . finite

a th
© M
t
0.1 !$ = 0 0011 0011 0011 … . infinite approximate float representation

i g h
p yr
C o
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
In the previous video we saw that some decimal numbers (with a finite representation)
cannot be represented with a finite binary representation

This can lead to some "weirdness" and bugs in our code (but not a Python bug!!)
m y
d e
x = 0.1 + 0.1 + 0.1
c a
format(x, '.25f') à 0.3000000000000000444089210
y = 0.3
A
format(y, '.25f') à 0.2999999999999999888977698

te
x == y à False

B y
a th
Using rounding will not necessarily solve the problem either!

M
It is no more possible to exactly represent round(0.1, 1) than 0.1 itself

©
h t
round(0.1, 1) + round(0.1, 1) + round(0.1, 1) == round(0.3, 1) à False

yr i g
But it can be used to round the entirety of both sides of the equality comparison

o p
round(0.1 + 0.1 + 0.1, 5) == round(0.3, 5) à True

C
To test for "equality" of two different floats, you could do the following methods:

round both sides of the equality expression to the number of significant digits

m y
round(a, 5) == round(b, 5)

d e
c a
or, more generally, use an appropriate range (ε) within which two numbers are deemed equal

for some B > 0, D=E if and only if D−E <B

te A
def is_equal(x, y, eps)

B y return math.fabs(x-y) < eps

a th
M
This can be tweaked by specifying that the difference between the two numbers
be a percentage of their size à the smaller the number, the smaller the tolerance

t ©
h
i.e. are two numbers within x% of each other?

yr i g
But there are non-trivial issues with using these seemingly simple tests

o p
à numbers very close to zero vs away from zero

C
Using absolute tolerances…

x = 0.1 + 0.1 + 0.1

m y
17th digit after decimal pt

e
y = 0.3
print(format(x, '.20f'))
print(format(y, '.20f'))
à 0.30000000000000004441
à 0.29999999999999998890
c a d
Δ = 0.00000000000000005551

A
0.000000000000001

y te
a = 10000.1 + 10000.1 + 10000.1

th B 12th digit after decimal pt

a
b = 30000.3

M
print(format(a, '.20f')) à 30000.30000000000291038305
Δ = 0.00000000000363797881
print(format(b, '.20f')) à 30000.29999999999927240424

©
0.000000000000001

h t
yr i g
Using an absolute tolerance: abs_tol = 10-15 = 0.000000000000001

then
o p
C
math.fabs(x - y) < abs_tol à True

math.fabs(a - b) < abs_tol à False

Maybe we should use relative tolerances…

x = 0.1 + 0.1 + 0.1

y = 0.3
à tol = 0.000003000000000

m y
d e
a = 10000.1 + 10000.1 + 10000.1
a
à tol = 0.300003000000000

c
A
b = 30000.3

y
Using a relative tolerance: rel_tol = 0.001% = 0.00001 = 1e-5
te
h
i.e. maximum allowed difference between the two numbers,

t B
a
relative to the larger magnitude of the two numbers

M
tol = rel_tol * max(|x|, |y|)
t ©
i g h
yr
math.fabs(x - y) < tol à True

o p
C
math.fabs(a - b) < tol à True

Success! but is it really?

x = 0.0000000001 (1e-10)

y
y = 0

Using a relative tolerance: rel_tol = 0.1% = 0.0001 = 1e-3

e m
tol = rel_tol * max(|x|, |y|) à tol = rel_tol * |x|
c a d
à 1e-3 * 1e-10 = 1e-13

te A
math.fabs(x - y) < abs_tol à False

B y
a th
Using a relative tolerance technique does not work well for numbers close to zero!

M
So using absolute and relative tolerances, in isolation, makes it

©
t
difficult to get a one-size-fits-all solution

i g h
yr
We can combine both methods
calculating the absolute and relative tolerances

o p
and using the larger of the two tolerances

C
tol = max(rel_tol * max(|x|, |y|), abs_tol)

à PEP 485
The math module has that solution for us! à PEP 485

math.isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0)

m y
d
If you do not specify abs_tol, then it defaults to 0 and you will face the problem
e
a
we encountered in the last slide when comparing numbers close to zero.

c
x = 1000.0000001
te
a = 0.0000001A
y = 1000.0000002
y
b = 0.0000002

B
math.isclose(x, y) à True
a th math.isclose(a, b) à False

but

© M
h t
math.isclose(x, y, abs_tol=1e-5) à True math.isclose(a, b, abs_tol=1e-5) à True

yr i g
o p
C
Also works well in situations like this:

x = 1000.01

m y
e
y = 1000.02

math.isclose(x, y, rel_tol=1e-5, abs_tol=1e-5)

c
à True
a d
te A
a = 0.01

B y
h
b = 0.02

math.isclose(x, y, rel_tol=1e-5, abs_tol=1e-5)

a t à False

© M
h t
g
If you are going to be using this method, you should

yr i
play around with it for a while until you get a good feel

p
for how it works

C o
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Float à Integer data loss

m y
different ways to configure this data loss

d e
10.4
c a
10.5
te A
10.6 10? 11?
B y
10.0001
a th
10.9999

© M
h t
g
truncation

floor
yr i
p
data loss in all cases

C o
ceiling

rounding
pick your poison!
Truncation

y
truncating a float simply returns the integer portion of the number

i.e. ignores everything after the decimal point

e m
c a d
The math module provides us the trunc() function:

te A
import math

B y
math.trunc(10.4) à 10

a th
M
math.trunc(10.5) à 10
math.trunc(10.6) à 10

t ©
i g h
yr
math.trunc(-10.4) à -10

p
math.trunc(-10.5) à -10

C o
math.trunc(-10.6) à -10
The int Constructor

The Python int constructor will accept a float

m y
uses truncation when casting the float to an int
d e
c a
int(10.4) à 10
te A
B y
h
int(10.5) à 10
int(10.6) à 10
a t
int(-10.4) à -10
© M
int(-10.5) à -10
h t
yr
int(-10.6) à -10
i g
o p
C
Floor

y
Definition: the floor of a number is the largest integer less than (or equal to) the number

m
Gloor K = max O ∈ ℤ | O ≤ K

e
d
10 11 -11 -10

c a
x = 10.4

te A
x = -10.4

floor floor
B y
a th
For positive numbers, floor and truncation are equivalent but not for negative numbers!

© M
Recall also our discussion on integer division – aka floor division: //

h t
We defined floor division in combination with the mod operation

g
n = d * (n // d) + (n % d)

yr i
But in fact, floor division defined that way yields the same result

o p
as taking the floor of the floating point division

C
a // b == floor (a / b)
Floor

The math module provides us the floor() function:

m y
d e
a
import math
10 11
Ac
te
math.floor(10.4) à 10
math.floor(10.5) à 10
y
x = 10.4

B
math.floor(10.6) à 10
floor
a th
© M
t
-11 -10

h
math.floor(-10.4) à -11

yr i
math.floor(-10.5) à -11
g x = -10.4

o p
math.floor(-10.6) à -11
floor

C
Ceiling

y
Definition: the ceiling of a number is the smallest integer greater than (or equal to) the number

m
ceil K = min O ∈ ℤ | O ≥ K

e
10 11
c a d
math.ceil(10.4) à 11

te A
x = 10.4
B y
math.ceil(10.5) à 11

ceiling
th
math.ceil(10.6) à 11

a
© M
-11 -10

h t
g
math.ceil(-10.4) à -10

yr i math.ceil(-10.5) à -10
x = -10.4

o p math.ceil(-10.6) à -10

C ceiling
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The round() function

Python provides a built-in rounding function: round(x, n=0)

m y
This will round the number x to the closest multiple of 10)+
d e
c a
you might think of this as rounding to a certain number of digits after the decimal point

te A
which would work for positive n, but n can, in fact, also be negative!

B y
coerce a float to an integer number
a th
In addition to truncate, floor, and ceiling, we can therefore also use rounding (with n = 0) to

© M
t
If n is not specified, then it defaults to zero and round(x) will therefore return an int

i g h
yr
round(x) à int

o p
round(x, n) à same type as x

C
round(x, 0) à same type as x
n=0

round to the closest multiple of 10)$ = 1

m y
d e
c a
x = 1.23

te A
1 2

B y
h
round(1.23) à 1
0.23 0.77
a t
closest

© M
h t
yr i g
o p
C
n>0

n = 1 round to the closest multiple of 10)! = 0.1

m y
d e
c a
x = 1.23
te A
1.2 1.3
B y
th
round(1.23, 1) à 1.2

a
closest
0.03 0.07

© M
h t
yr i g
o p
C
n<0

n = -1 round to the closest multiple of 10)()!) = 10

m y
d e
c a
x = 18.2
te A
10 20
B y
th
round(18.2, -1) à 20

a
8.2 1.8
closest
© M
h t
yr i g
o p
C
Ties

y
x = 1.25
1.2 1.3 round(1.25, 1) = ???
e m
there is no closest value!!

c a d
A
0.05 0.05

We probably would expect round(1.25, 1) to be 1.3

y te rounding up / away from zero

th B
a
Similarly, we would expect round(-1.25, 1) to result in -1.3 rounding down / away from zero

© M
This type of rounding is called rounding to nearest, with ties away from zero

h t
But in fact:

yr i g
round(1.25, 1) à 1.2 towards 0

o p
round(1.35, 1) à 1.4 away from 0

C round(-1.25, 1) à -1.2 towards 0

round(-1.35, 1) à -1.4 away from 0

 Banker's Rounding

m y
IEEE 754 standard: rounds to the nearest value, with ties rounded to the nearest value with an
even least significant digit
d e
c a
x = 1.25
te A
1.2 1.3
B y
round(1.25, 1) à 1.2

a th
M
0.05 0.05

©
2 is even

h t
x = 1.35

yr i g
1.3
o p 1.4 round(1.35, 1) à 1.4

C 0.05 0.05
4 is even
n = -1 round to the closest multiple of 10)()!) = 10

x = 15
m y
10 20
d e
round(15, -1) à 20
c a
5 5
2 is even
te A
B y
x = 25
a th
20 30

© M
round(25, -1) à 20

h t
g
5 5

2 is even
yr i
o p
C
Why Banker's Rounding?

Less biased rounding than ties away from zero

m y
d e
c a
Consider averaging three numbers, and averaging the rounded value of each:

te A
y
0.5, 1.5, 2.5 à avg = 4.5 / 3 = 1.5

"standard" rounding: 1, 2, 3
th B
à avg = 6 / 3 = 2

M a
©
banker's rounding: 0, 2, 2 à avg = 4 / 3 = 1.3…

h t
yr i g
o p
C
If you really insist on rounding away from zero…

y
One common (and partially incorrect) way to round to nearest unit that often comes up

m
e
on the web is:

int(x + 0.5)
c a
10.3 à int(10.3 + 0.5) = int(10.8) = 10 d
te A
10.9 à int(10.9 + 0.5) = int(11.4) = 11

B y
10.5 à int(10.5 + 0.5) = int(11.0) = 11

but, this does not work for negative numbers

a th
© M
-10.3 à int(-10.3 + 0.5) = int(-9.8) = -9

h t
-10.9 à int(-10.9 + 0.5) = int(-10.4) = -10

yr i g -10.5 à int(-10.5 + 0.5) = int(-10.0) = -10

o p
C
Technically, this is also an acceptable rounding method
referred to as rounding towards + infinity
But this not rounding towards zero !!
 If you really insist on rounding away from zero…
The correct way to do it:

m y
e
sign(x) * int(abs(x)+0.5) 10.4 10.5 10.6 -10.4 -10.5 -10.6

+1 if K ≥ 0
sign(x) + +

c a + d- - -
sign K = W
−1 if K < 0 abs(x)+0.5 10.9

te A
11.0 11.1 10.9 11.0 11.5

!! Not the same as the

mathematical sgn
int(abs(x)+0.5)

B y
10 11 11 10 11 11
(signum) function!

a
sign(x) * int(abs(x)+0.5)
th 10 11 11 -10 -11 -11

M
= int(x + 0.5 * sign(x))

©
h t
Python does not have a sign function !

yr i g
We can however use the math.copysign() function to achieve our goal:

o p
C
copysign(x, y) returns the magnitude (absolute value) of x but with the sign of y

sign(x) = copysign(1, x)
 sign(x) * int(abs(x)+0.5)

m y
def round_up(x):
d e
from math import fabs, copysign
c a
A
return copysign(1, x) * int(fabs(x) + 0.5)

y te
A simpler way to code this:

th B
int(x + 0.5 * sign(x))

Ma
def round_up(x):

t ©
h
from math import copysign

i g
return int(x + copysign(0.5, x))

yr
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The decimal module (PEP 327)

float 0.1 à infinite binary expansion 0.1 = 0. 0 0011 0011 0011 … .

m y
e
!$

d
1 1 1 1 1 1

a
= + + + + + + …

c
16 32 256 512 4096 8192

à finite decimal expansion 0.1 !$ =

1

te A
y
10

th B
alternative to using the (binary) float type

M a
à avoids the approximation issues with floats

t
finite number of significant digits
©
à rational number (see videos on rationals)

i g h
yr
So why not just use the Fraction class?

o p
to add two fractions à common denominator

C à complex, requires extra memory

Why do we even care? Why not just use binary floats?

m
finance, banking, and any other field where exact finite representations are highly desirable y
d e
c a
let's say we are adding up all the financial transactions that took place over a certain time period

te A
y
amount = $100.01 1,000,000,000 transactions NYSE: 2-6 billion shares traded daily

th B
M
100.01 à 100.0100000000000051159076975 a
©
sum à $100010000000.00 (exact decimal)

t
i g h
$100009998761.1463928222656250000000000 (approximate binary float)

p yr
$1238.85… off!!

C o
Decimals have a context that controls certain aspects of working with decimals

precision during arithmetic operations

m y
rounding algorithm
d e
c a
This context can be global à the default context

te A
or temporary (local)
B y
à sets temporary settings without affecting the global settings

a th
M
import decimal

default context
t ©
à decimal.getcontext()

local context
i g h
yr
à decimal.localcontext(ctx=None)

p
creates a new context, copied from ctx

C o or from default if ctx not specified

returns a context manager (use a with statement)
 Precision and Rounding

y
ctx = decimal.getcontext() à context (global in this case)
ctx.prec à get or set the precision (value is an int)
e m
ctx.rounding
a
à get or set the rounding mechanism (value is a string)

c d
ROUND_UP rounds away from zero

te A
y
ROUND_DOWN rounds towards zero
ROUND_CEILING
B
rounds to ceiling (towards +∞)

th
ROUND_FLOOR
ROUND_HALF_UP
M a
rounds to floor (towards −∞)
rounds to nearest, ties away from zero

t
ROUND_HALF_DOWN
© rounds to nearest, ties towards zero

float rounding algorithm

i g h
ROUND_HALF_EVEN rounds to nearest, ties to even (least significant digit)

p yr
C o
Working with Global and Local Contexts

m y
Global

d e
decimal.getcontext().rounding = decimal.ROUND_HALF_UP
c a
A
//decimal operations performed here will use the current default context

te
B y
Local
a th
with decimal.localcontext() as ctx:
ctx.prec = 2
© M
h t
ctx.rounding = decimal.ROUND_HALF_UP

yr i g
//decimal operations performed here

p
//will use the ctx context

o
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Constructing Decimal Objects

The Decimal class is in the decimal module import decimal

m
from decimal import Decimal y
Decimal(x) x can be a variety of types
d e
c a
integers a = Decimal(10)

te A à 10

y
other Decimal object

strings
th
a = Decimal('0.1') B à 0.1
tuples
M a
a = Decimal((1, (3, 1, 4, 1, 5), -4)) à -3.1415

floats?
t ©
yes, but not usually done

i g h
yr
Decimal(0.1) à 0.100000000000000005551

o p
Since 0.1 does not have an exact binary float representation

C
it cannot be used to create an exact Decimal representation
of itself

à Use strings or tuples instead

Using the tuple constructor

1.23 à +123 x 10-2

m y
d e
-1.23 à -123 x 10-2
c a
(s, (d1, d2, d3, …), exp) Z(K) = W
0 O[ K ≥ 0

A
1 O[ K < 0

exponent

y te
sign
digits
t h B
Example: -3.1415
M a
à (1, (3, 1, 4, 1, 5), -4)

t ©
i g h
a = Decimal((1, (3, 1, 4, 1, 5), -4)) a à -3.1415

p yr
C o
Context Precision and the Constructor

Context precision affects mathematical operations

m y
Context precision does not affect the constructor
d e
c a
import decimal

te A
y
from decimal import Decimal

decimal.getcontext().prec = 2
th B
ß global (default) context now has precision set to 2

a = Decimal('0.12345')
M
a à 0.12345 a
b = Decimal('0.12345')
t ©
b à 0.12345
c = a + b

i g h a + b = 0.2469 c à 0.25

p yr
C o
Local vs Global Context

import decimal

m y
e
from decimal import Decimal

decimal.getcontext().prec = 6

c a d
a = Decimal('0.12345')
te A
b = Decimal('0.12345')
B y
print(a + b) à 0.24690
a th
ctx.prec = 2
© M
with decimal.localcontext() as ctx:

c = a + b
h t
print(c)

yr i g à 0.25

print(c)
o p à 0.25

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Some arithmetic operators don't work the same as floats or integers

// and % à also divmod()

m y
d e
The // and % operators still satisfy the usual equation:
c a
n = d * (n // d) + (n % d)

te A
But for integers, the // operator performs floor division
B y
à a // b = floor(a/b)

a th negative
numbers!

© M
For Decimals however, it performs truncated division à a // b = trunc(a/b)

h t
10 // 3 à 3

yr i g
Decimal(10) // Decimal(3) à Decimal(3)

-10 // 3 à -4

o p Decimal(-10) // Decimal(3) à Decimal(-3)

C
Boils down to the algorithm used to actually perform integer division

D dividend

m y
e
E divisor

- figure out the sign of the result

c a d
- use absolute values for divisor and dividend

te A
- keep subtracting b from a as long as a >= b

B y
- return the signed number of times this was performed

a th
10
3
res is +
10 - 3
= 7 7 >= 3
7 - 3

©
= 4 4 M >= 3
4 - 3
= 1 1 < 3 - STOP
return 3

h t
−10
3
res is -

yr i
10 - 3
g 7 - 3 4 - 3
return -3

p
= 7 7 >= 3 = 4 4 >= 3 = 1 1 < 3 - STOP

C o
this is basically the same as truncating the real division
trunc(10/3) à 3 trunc(-10/3) à -3
 But n = d * (n // d) + (n % d) is still satisfied

n = -135, d = 4

m y
d e
a
Integer Decimal

-135 // 4 -34
Ac -33

y te
-135 % 4 1

th B -3

d * (n // d) + (n % d)
M a
4 * (-34) + 1 = -135 4 * (-33) + (-3) = -135

t ©
i g h
p yr
C o
Other Mathematical Operations

y
The Decimal class defines a bunch of various mathematical operations, such as sqrt, logs, etc.

m
But not all functions defined in the math module are defined in the Decimal class
d e
c a
E.g. trig functions

te A
We can use the math module,
B y
a
but Decimal objects will first be cast to floats th
© M
– so we lose the whole precision mechanism that made us use Decimal objects in the first place!

h t
yr i g
Usually will want to use the math functions defined in the Decimal class if they are available

o p
C
decimal.getcontext().prec = 28

x = 0.01
m y
x_dec = Decimal('0.01')

d e
root = math.sqrt(x)
c a
root_mixed = math.sqrt(x_dec)

te A
root_dec = x_dec.sqrt()

B y
print(format(root, '1.27f'))
a th à 0.100000000000000005551115123
print(format(root_mixed, '1.27f'))
print(root_dec)
© M à 0.100000000000000005551115123
à 0.1

h t
yr i g
print(format(root * root, '.27f')) à 0.010000000000000001942890293

p
print(format(root_mixed * root_mixed, '.27f')) à 0.010000000000000001942890293

C o
print(root_dec * root_dec) à 0.01
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
There are some drawbacks to the Decimal class vs the float class

m y
- not as easy to code: construction via strings or tuples

d e
c a
A
- not all mathematical functions that exist in the math module have a Decimal counterpart

- more memory overhead

y te
th
- performance: much slower than floats (relatively) B
M a
t ©
i g h
p yr
C o
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The complex class

Literals

m y
Constructor: complex(x, y) x à real part x + yJ

d e
y à imaginary part
a
x + yj

c
(rectangular coordinates)

te A
Example: a = complex(1, 2)
B y
b = 1 + 2j
a th
a == b à True

© M
h t
yr i g
x and y (the real and imaginary parts) are stored as floats

o p
C
Some instance properties and methods

m y
.real à returns the real part
d e
.imag à returns the imaginary part
c a
.conjugate() à returns the complex conjugate
te A
B y
a th
M
d = 2 – 3j
d.real à 2

t ©
d.imag à -3

i g h
yr
d.conjugate() à 2 + 3j

p
C o
Arithmetic Operators

m y
The standard arithmetic operators (+, -, / , *, **) work as expected with complex numbers

d e
(1 + 2j) + (3 + 4j) à 4 + 6j
c a
(1 + 2j) * (3 + 4j) à 5 + 10j
te A
B y
Real and Complex numbers can be mixed:
a th
(1 + 2j) + 3 à 4 + 2j
© M
h t
i g
(1 + 2j) * 3 à 3 + 6j

yr
o p
// and % operators are not supported

C
Other operations

The == and != operators are supported

m y
Comparison operators such as <, >, <= and >= are not supported
d e
c a
Functions in the math module will not work
te A
B y
h
Use the cmath module instead

exponentials
a t
logs

© M
trigs and inverse trigs
h t
yr i g
hyperbolics and inverse hyperbolics

p
polar / rectangular conversions

o
C
isclose
Rectangular to Polar ℐd K
`
import cmath

m y ^

d e ℜb

a
cmath.phase(x) Returns the argument (phase) ^ of the complex number x
^ ∈ −?, ?
Ac
measured counter-clockwise from the real axis

abs(x) Returns the magnitude (`) of x

y te
th B
a = -1 + 0j
cmath.phase(a) à 3.1415… (?)
M a
abs(a) à 1

t ©
a = -1j

i g h
yr
1
cmath.phase(a) à -1.570… (− ) abs(a) à 1
.

o p
a = 1 + 1j

C
cmath.phase(a) à 0.785… ( )
1
# abs(a) à 1.414… ( 2)
Polar to Rectangular

import cmath

m y
d e
cmath.rect(r, phi)
c a
Returns a complex number (rectangular coordinates)

in polar coordinates

te A
equivalent to the complex number defined by (r, phi)

B y
cmath.rect(math.sqrt(2), math.pi/4)
a th
à 1 + 1j

© M 1.0000000000000002+1.0000000000000002j

h t
yr i g
o p
C
Euler's Identity

! 01 + 1 = 0
m y
d e
cmath.exp(cmath.pi * 1j) + 1

c a
à 1.2246467991473532e-16j

te A
binary floats tend to spoil the effect!

So, the next best thing:

B y
a th
cmath.isclose(cmath.exp(cmath.pi*1j) + 1, 0, abs_tol=0.0001) à True

© M
h t
Do note however the same issue with isclose() as we discussed in the float videos:

yr i g
cmath.isclose(cmath.exp(cmath.pi*1j) + 1, 0) à False

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The bool class PEP 285

Python has a concrete bool class that is used to represent Boolean values.

m y
d e
However, the bool class is a subclass of the int class
a
i.e. they posses all the properties and

c
methods of integers, and add some

issubclass(bool, int)
te A
specialized ones such as and, or, etc

y
à True

th B
isinstance(True, bool) à True
Two constants are defined: True and False

M a isinstance(True, int) à True

t ©
They are singleton objects of type bool

i g h
yr
bool 0xF345
True int

p
1

C o bool
int
0xA101
False 0
is vs ==

throughout the lifetime of your application

m y
Because True and False are singleton objects, they will always retain their same memory address

d e
c a
So, comparisons of any Boolean expression to True and False can be performed using
either the is (identity) operator, or the == (equality) operator

te A
a == True a is True

B y
where a is a bool object

a th
But since bool objects are also int objects, they can also be interpreted as the integers 1 and 0

int(True) à 1
© M
int(False) à 0

h t
yr i g
But: True and 1 are not the same objects id(True) ≠ id(1)

o p
False and 0 are not the same objects id(False) ≠ id(0)

CTrue is 1
True == 1
à False
à True
Booleans as Integers

m y
e
This can lead to "strange" behavior you may not expect!

True > False à True

c a d
te A
y
(1 == 2) == False à True (1 == 2) == 0 à True

th B
In fact, any integer operation will also work with booleans (//, %, etc)

True + True + True à 3

M a
(True + True + True) % 2 à 1

t ©
h
-True à -1

yr i
100 * False à 0g
o p
C
The Boolean constructor

m
The Boolean constructor bool(x) returns True when x is True, and False when x is False y
d e
Wow, that sounds like a useless constructor! But not at all!
c a
te A
B y
What really happens is that many classes contain a definition of how to cast instances of
themselves to a Boolean – this is sometimes called the truth value (or truthyness) of an object

a th (upcoming video)

© M
h t
Integers have a truth value defined according to this rule:

bool(0) à False

yr i g (0 is falsy)

o p
bool(x) à True for any int x ≠ 0 (x is truthy)

C
Examples

bool(0) à False
m y
d e
c a
A
bool(1) à True

bool(100) à True

y te
bool(-1) à True

th B
Ma
t ©
i g h
p yr
C o
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Objects have Truth Values

All objects in Python have an associated truth value

m y
We already saw this with integers
d e
(although to be fair, bool is a subclass of int)

c a
But this works the same for any object

te A
In general, the rules are straightforward

B y
a th
M
Every object has a True truth value, except:

• None
t ©
• False

i g h
yr
• 0 in any numeric type (e.g. 0, 0.0, 0+0j, …)

p
• empty sequences (e.g. list, tuple, string, …)
•
•
C o
empty mapping types (e.g. dictionary, set, …)
custom classes that implement a __bool__ or __len__
method that returns False or 0
which have a False truth value
Under the hood

Classes define their truth values by defining a special instance method:

m y
__bool__(self) (or __len__ )

d e
c a
A
Then, when we call bool(x) Python will actually executes x.__bool__()

te
or __len__ if __bool__ is not defined

B yif neither is defined, then True

Example: Integers
a th
def __bool__(self):
return self != 0
© M
h t
yr i g
When we call bool(100) Python actually executes 100.__bool__()

p
and therefore returns the result of 100 != 0

o
which is True

C
When we call bool(0) Python actually executes 0.__bool__()
and therefore returns the result of 0 != 0 which is False

We will cover this and many other special functions in a later section
Examples

bool([1, 2, 3]) à True

m y
bool([]) à False if my_list:

d e
a
# code block

c
bool(None) à False

bool('abc') à True
te
both not None and not empty A
code block will execute if and only if my_list is

bool('') à False
B y
this is equivalent to:

th
if my_list is not None and len(my_list) > 0:

a
# code block

M
bool(0) à False
bool(0 + 0j) à False

t ©
i g h
bool(Decimal('0.0') à False

yr
bool(-1) à True

o p
bool(1 + 2j) à True

C
bool(Decimal('0.1') à True
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The Boolean Operators: not, and, or

open False
m y
e
X Y not X X and Y X or Y
0
0
0
1
1
1
0
0
0
1
c a d
1 0 0 0 1

te A
closed True
1 1 0 1 1

B y
X
a th
X or Y
© M X and Y
Y
h t X Y

yr i g
p
True

C o
False
True
False True
False
Commutativity Distributivity
A or B == B or A A and (B or C) == (A and B) or (A and C)
A and B == B and A A or (B and C) == (A or B) and (A or C)

m y
Associativity
d e
A or (B or C) == (A or B) or C A or B or C
c a à (A or B) or C
A and (B and C) == (A and B) and C

te A
A and B and C à (A and B) and C

De Morgan's Theorem
B y left-to-right evaluation

a
not(A or B) == (not A) and (not B)
th
M
not(A and B) == (not A) or (not B)

©
h t
Miscellaneous

yr i g
not(x < y) == x >= y not(x <= y) == x > y

o p
not(x > y) == x <= y not(x >= y) == x < y

C
not(not A) == A
Operator Precedence

() True or True and False

m y
e
< > <= >= == != in is highest
à True or False à True

a d
precedence
not
and
Ac
(True or True) and False

or

y teà True and False à False

B
lowest

a th
M
When in doubt, or to be absolutely sure, use parentheses! True or (True and False)

©
Also, use parentheses to make your code more human readable!

t
i g h
yr
a < b or a > c and not x or y

o p
C
(a < b) or ((a > c) and (not x)) or y
Short-Circuit Evaluation
X

y
True
X Y X or Y
0 0 0
X or Y
e
True
m
0 1 1 Y

c a Y
d
1
1
0
1
1
1 A
if X is True, then X or Y will be True no matter the value of Y

te
y
So, X or Y will return True without evaluating Y if X is True

B
X Y X and Y
a th
0 0 0 X
© MY
X and Y
False Y
False
0 1 0

h t
1
1
0
1
0
1
yr i g if X is False, then X and Y will be False no matter the value of Y
So, X and Y will return False without evaluating Y if X is False

o p
C
Example

Scenario: There is some data feed that lists a stock symbol, and some financial data.

m y
d e
Your job is to monitor this feed, looking for specific stock symbols defined in some watch
list, and react only if the current stock price is above some threshold. Getting the current
stock price has an associated cost.
c a
te A
y
If Boolean expressions did not implement short-circuiting, you would probably write:

B
if symbol in watch_list:

a thsince calling the price() method has a cost,

M
if price(symbol) > threshold: you would only want to call it if the symbol was
on your watch list
# do something

t ©
i g h
yr
But because of short-circuit evaluation you could write this equivalently as:

o p
C
if symbol in watch_list and price(symbol) > threshold:
# do something
Example

y
null à None
name is a string returned from a nullable text field in a database
''

e m
perform some action if the first character of name is a digit (0-9)

c a
'abc'
d
if name[0] in string.digits:
te A
this code will break if name is None or an
# do something
B y
empty string

a th
M
because of short-circuiting and truth values

©
t
if name and name[0] in string.digits: if name is falsy (either None or an empty string)
# do something

i g h then

yr
name[0] in string.digits
is not evaluated

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Boolean Operators and Truth Values

X Y X and Y X or Y
y
Normally, Boolean operators are defined to operate on and

m
e
return Boolean values

d
0 0 0 0
0 1 0 1 True or False à True
c a
1 0 0 1 a = 2

te A
a > 0 and b < 5 à True

y
1 1 1 1 b = 3

th B
a
But every object in Python has a truth value (truthiness)

M
t ©
so, for any object X and Y, we could also write bool(X) and bool(Y) bool(X) or bool(Y)

i g h
In fact, we don't need to use bool()

yr
X and Y X or Y

o p
C
So, what is returned when evaluating these expressions?

A Boolean? No!
Definition of or in Python X Y X or Y

y
0 0 0

X or Y If X is truthy, returns X, otherwise returns Y

0

e m
1 1

c a d1
1
0
1
1
1
Does this work as expected when X and Y are Boolean values?

te A
B y
X Y Rule
th
Result

a
0 0

© M
X is False, so return Y 0

0 1
h t
X is False, so return Y 1

1 0
yr i g
X is True, so return X 1

o p
C X is True, so return X
1 1 1

If X is truthy, returns X, otherwise evaluates Y and returns it

Definition of and in Python X Y X and Y

y
0 0 0

X and Y If X is falsy, returns X, otherwise returns Y

0

e
1
m 0

c a d1
1
0
1
0
1
Does this work as expected when X and Y are Boolean values?

te A
B y
X Y Rule
th
Result

a
0 0

© M
X is False, so return X 0

0 1
h t
X is False, so return X 0

1 0
yr i g
X is True, so return Y 0

o p
C X is True, so return Y
1 1 1

If X is falsy, returns X, otherwise evaluates Y and returns it

Consequence: or

X or Y If X is truthy, returns X, otherwise evaluates and returns Y

m y
d e
a
X Y X or Y
None 'N/A' 'N/A'
Ac
'' 'N/A' 'N/A'
y te
'hello' 'N/A' 'hello'
th B
if s is None
M a a à N/A

©
a = s or 'N/A'

h t
if s is '' a à N/A

yr i gif s is a string
a à s

o p with characters

C
i.e. a will either be s or 'N/A' if s is None or an empty string
Example
We can expand this further:

m y
a = s1 or s2 or s3 or 'N/A'
d e
c a
In this case, a will be equal to the first truthy value

te A
(left to right evaluation)

and is guaranteed to have a value, since 'N/A' is truthy

B y
a th
Example

© M
t
We have an integer variable a that cannot be zero – if it is zero, we want to set it to 1.

h
a = a or 1

yr i g
o p
C
Consequence: and

X and Y If X is falsy, returns X, otherwise evaluates and returns Y

m y
d e
a
X Y X and Y
10 20/X 2
Ac
0 20/X 0
y te
th B
M a
Seems like we are able to avoid a division by zero error using the and operator

t ©
i g h
yr
x = a and total/a

o p
C
a = 10 à x = 10 and total/10 à total/10

a = 0 à x = 0 and total/0 à0
Example

Computing an average

m y
sum, n Sometimes n is non-zero, sometimes it is

d e
In either case: avg = n and sum/n
c a
te A
Example

B y
th
You want to return the first character of a string s, or an empty string if the string is None or empty

a
Option 1
© M
Option 2

h t
g
if s: return s and s[0] à doesn’t handle None case
return s[0]

yr i
p
else: return (s and s[0]) or ''

o
return ''

C
The Boolean not

not is a built-in function that returns a Boolean value

m y
d e
not x à True if x is falsy
c a
à False if x is truthy
te A
B y
a th
[] à falsy not [] à True

© M
h t
g
[1, 2] à truthy not [1, 2] à False

yr i
o p
None à falsy not None à True

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Categories of Comparison Operators

• binary operators
m y
• evaluate to a bool value
d e
c a
Identity Operations is is not
te A
compares memory address – any type

B y
a th compares values – different types OK,

M
Value Comparisons == !=
but must be compatible

t ©
Ordering Comparisons
i g h
< <= > >= doesn't work for all types

p yr
C o
Membership Operations in not in used with iterable types
Numeric Types
We will examine other types, including iterables, later in this course

m y
Value comparisons will work with all numeric types

d e
c a
Mixed types (except complex) in value and ordering comparisons is supported

te A
Note:
y
Value equality operators work between floats and Decimals, but as we have seen

B
before, using value equality with floats has some issues!

a th
10.0 == Decimal('10.0')
M
à True

©
0.1 == Decimal('0.1')

h t à False

yr i g
Decimal('0.125') == Fraction(1, 8) à True

o p
C
True == 1 à True

True == Fraction(3, 3) à True

Ordering Comparisons

Again, these work across all numeric types, except for complex numbers
m y
d e
1 < 3.14 à True
c a
te A
Fraction(22, 7) > math.pi à True

B y
a th
M
Decimal('0.5') <= Fraction(2, 3) à True

t ©
True < Decimal('3.14')

i g h à True

p yr
C o
Fraction(2, 3) > False à True
Chained Comparisons

m y
e
a == b == c à a == b and b == c

a < b < c à a < b and b < c

c a d
te A
1 == Decimal('1.0') == Fraction(1,1) à True

B y
1 == Decimal('1.5') == Fraction(3, 2)

a th à False

© M
t
1 < 2 < 3 à 1 < 2 and 2 < 3 à True

i g h
yr
1 < math.pi < Fraction(22, 7)

p
C o
à 1 < math.pi and math.pi < Fraction(22, 7)
à True
Chained Comparisons

a < b > c à a < b and b > c

m y
d e
5 < 6 > 2 à 5 < 6 and 6 > 2 à True

c a
5 < 6 > 10 à 5 < 6 and 6 > 10 à False

te A
B y
a < b < c < d
a th
à a < b and b < c and c < d

1 < 2 < 3 < 4

© M
à 1 < 2 and 2 < 3 and 3 < 4 à True

h t
1 < 10 > 4 < 5

yr i g à 1 < 10 and 10 > 4 and 4 < 5 à True

o p
C
if my_min == cnt < val > other <= my_max not in lst:
# do something
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Arguments vs Parameters

m y
Positional vs Keyword-Only Arguments
d e
c a
Optional Arguments via Defaults

te A
B y
Unpacking Iterables and Function Arguments

a th
Extended Unpacking
© M
h t
yr i g
Variable Number of Positional and Keyword-Only Arguments

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Semantics!

def my_func(a, b):

m y
e
In this context, a and b are called parameters of my_func
# code here

a d
Also note that a and b are variables, local to my_func

c
te A
When we call the function:
B y
x = 10
a th
x and y are called the arguments of my_func
y = 'a'

© M
Also note that x and y are passed by reference
my_func(x, y)
h t
yr i g i.e. the memory addresses of x and y are passed

o p
C
It's OK if you mix up these terms – everyone will understand what you mean!
x = 10 def my_func(a, b):
y = 'a' # code here

m y
my_func(x, y)
d e
c a
te A
y
Module Scope Function Scope

B
0xA13F

x 10

a th a

© M
0xE345
b

t
'a'

i g h
p yr
C o
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Positional Arguments

y
Most common way of assigning arguments to parameters: via the order in which they are passed

m
i.e. their position
d e
c a
def my_func(a, b):
te A
# code …

B y
a th
my_func(10, 20)
© M
à a = 10, b = 20

h t
my_func(20, 10)

yr i g
à a = 20, b = 10

o p
C
Default Values

A positional arguments can be made optional by specifying a default value for the
corresponding parameter
m y
d e
def my_func(a, b=100): my_func(10, 20)
c a
à a = 10, b = 20

A
# code …

te
my_func(5) à a = 5, b = 100

B y
Consider a case where we have three arguments, and we want to make one of them optional:

a th
How would we call this function without specifying a value

M
def my_func(a, b=100, c):
# code … for the second parameter?

t ©
i g h my_func(5, 25) ???

p yr
C o
If a positional parameter is defined with a default value

every positional parameter after it

must also be given a default value
Default Values

def my_func(a, b=5, c=10): my_func(1)

m
à a = 1, b = 5, c = 10 y
# code …

d e
a
my_func(1, 2) à a = 1, b = 2, c = 10

my_func(1, 2, 3)
Ac
à a = 1, b = 2, c = 3

y te
h B
But what if we want to specify the 1st and 3rd arguments, but omit the 2nd argument?

t
a
i.e. we want to specify values for a and c, but let b take on its default value?

M
à Keyword Arguments
t ©
(named arguments)

i g h
yr
my_func(a=1, c=2) à a = 1, b = 5, c = 2

o p
C
my_func(1, c=2) à a = 1, b = 5, c = 2
Keyword Arguments

m y
e
Positional arguments can, optionally, be specified by using the parameter name

d
whether or not the parameters have default values
c a
def my_func(a, b, c) my_func(1, 2, 3)
te A
B y
h
my_func(1, 2, c=3)

t
à a=1, b=2, c=3

a
my_func(a=1, b=2, c=3)

M
©
my_func(c=3, a=1, b=2)

h t
g
But once you use a named argument, all arguments thereafter must be named too

yr i
p
my_func(c=1, 2, 3)

o
C
my_func(1, b=2, 3) my_func(1, b=2, c=3)
my_func(1, c=3, b=2)
Keyword Arguments

All arguments after the first named (keyword) argument, must be named too
m y
d e
a
Default arguments may still be omitted

Ac
te
def my_func(a, b=2, c=3)

B y
my_func(1) à a=1, b=2, c=3
a th
my_func(a=1, b=5)
© M
à a=1, b=5, c=3

h t
my_func(c=0, a=1)

yr i g à a=1, b=2, c=0

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
A Side Note on Tuples

(1, 2, 3)

m y
What defines a tuple in Python, is not (), but ,
d e
c a
1, 2, 3 is also a tuple à (1, 2, 3)

te A
The () are used to make the tuple clearer

B y
To create a tuple with a single element:

a th
will not work as intended
M
(1) à int

t ©
h
1, or (1, ) à tuple

yr i g
o p
The only exception is when creating an empty tuple: () or tuple()

C
Packed Values

Packed values refers to values that are bundled together in some way
m y
d e
Tuples and Lists are obvious t = (1, 2, 3)
c a
te A
l = [1, 2, 3]

B y
Even a string is considered to be a packed value:

a ths = 'python'

© M
Sets and dictionaries are also packed values: set1 = {1, 2, 3}

h t d = {'a': 1, 'b': 2, 'c': 3}

yr i g
o p
In fact, any iterable can be considered a packed value

C
Unpacking Packed Values

m y
Unpacking is the act of splitting packed values into individual variables contained in a list or tuple

d e
a, b, c = [1, 2, 3] 3 elements in [1, 2, 3]
c a
à need 3 variables to unpack

te A
B y
th
this is actually a tuple of 3 variables: a, b and c

a
© M
a à 1 b à 2
t
c à 3

h
yr i g
p
The unpacking into individual variables is based on the relative positions of each element

C o
Does this remind you of how positional arguments were assigned to parameters in function calls?
Unpacking other Iterables

a, b, c = 10, 20, 'hello' à a = 10 b = 20 c = 'hello'

m y
d e
this is actually a tuple containing 3 values
c a
te A
a, b, c = 'XYZ' à a = 'X'

B yb = 'Y' c = 'Z'

instead of writing a = 10 we can write

a tha, b = 10, 20
b = 20

© M
h t
In fact, unpacking works with any iterable type

yr i g
p
for e in 10, 20, 'hello' à loop returns 10, 20, 'hello'

C o
for e in 'XYZ' à loop returns 'X', 'Y', 'Z'
Simple Application of Unpacking

swapping values of two variables a = 10 b = 20

m y
e
à
b = 20 a = 10

"traditional" approach
c a d
A
tmp
0x123

te
tmp = a
a

y
10
a = b
b = tmp
th B
a
0xF12

M
b 20

t ©
i g h
using unpacking

p yr this works because in Python, the entire

C
a, b = b, a
o RHS is evaluated first and completely
then assignments are made to the LHS
Unpacking Sets and Dictionaries
d = {'key1': 1, 'key2': 2, 'key3': 3}

m y
à e iterates through the keys: 'key1', 'key2', 'key3'
d e
a
for e in d

Ac
te
so, when unpacking d, we are actually unpacking the keys of d

B y
a, b, c = d

a th
à a = 'key1', b = 'key2', c='key3'
or
or
© M
à a = 'key2', b = 'key1', c='key3'
à a = 'key3', b = 'key1', c='key2'

h
etc…
t
yr i g
Dictionaries (and Sets) are unordered types.

o p
They can be iterated, but there is no guarantee the order of the results will match your literal!

C
In practice, we rarely unpack sets and dictionaries in precisely this way.
Example using Sets

s = {'p', 'y', 't', 'h', 'o', 'n'}

m y
d e
for c in s: à p
c a
print(c) t
h
te A
n
o
B y
y

a th
© M
t
a, b, c, d, e, f = s a = 'p'

i g h b = 't'

p yr c = 'h'

C o …
f = 'y'
 m y
d e
c a
te A
B y
a th
© M
* **

h t
yr i g
o p
C
The use case for * Much of this section applies to Python >= 3.5

We don't always want to unpack every single item in an iterable

m y
d e
a
We may, for example, want to unpack the first value, and then unpack the remaining values into
another variable

Ac
l = [1, 2, 3, 4, 5, 6]

y te
We can achieve this using slicing:
t
a = l[0]
h B
M a
b = l[1:]

t ©
or, using simple unpacking:

i g h a, b = l[0], l[1:] (aka parallel assignment)

p yr
C o
We can also use the * operator: a, *b = l

Apart from cleaner syntax, it also works with any iterable, not just sequence types!
Usage with ordered types

a, *b = [-10, 5, 2, 100] a = -10 b = [5, 2, 100]

m y
this is still a list!

d e
a
a, *b = (-10, 5, 2, 100) a = -10 b = [5, 2, 100] this is also a list!

Ac
te
a, *b = 'XYZ' a = 'X' b = ['Y', 'Z']

B y
The following also works:

a th
a, b, *c = 1, 2, 3, 4, 5

© M a = 1 b = 2 c = [3, 4, 5]

h t
yr i g
a, b, *c, d = [1, 2, 3, 4, 5] a = 1 b = 2 c = [3, 4] d = 5

o p
a, *b, c, d = 'python' a = 'p' b = ['y', 't', 'h']

C c = 'o' d = 'n'
The * operator can only be used once in the LHS an unpacking assignment

m y
For obvious reason, you cannot write something like this:

d e
c a
A
a, *b, *c = [1, 2, 3, 4, 5, 6]

y te
th B
Since both *b and *c mean "the rest", both cannot exhaust the remaining elements

M a
t ©
i g h
p yr
C o
Usage with ordered types

m y
We have seen how to use the * operator in the LHS of an assignment to unpack the RHS

a, *b, c = {1, 2, 3, 4, 5}
d e
c a
However, we can also use it this way:
te A
B y
l1 = [1, 2, 3]

a th
l2 = [4, 5, 6]

© M
t
l = [*l1, *l2] à l = [1, 2, 3, 4, 5, 6]

i g h
p
l1 = [1, 2, 3]
yr
l2 = 'XYZ'

C o
l = [*l1, *l2] à l = [1, 2, 3, 'X', 'Y', 'Z']
Usage with unordered types

Types such as sets and dictionaries have no ordering

m y
d e
s = {10, -99, 3, 'd'}
c a
print(s) à {10, 3, 'd', -99}

te A
B y
Sets and dictionary keys are still iterable, but iterating has no guarantee of preserving
the order in which the elements were created/added

a th
© M
But, the * operator still works, since it works with any iterable

h t
s = {10, -99, 3, 'd'}

yr i g
p
a, *b, c = s a = 10 b = [3, 'd'] c = -99

C o
In practice, we rarely unpack sets and dictionaries directly in this way.
Usage with unordered types

m y
It is useful though in a situation where you might want to create single collection containing all the
items of multiple sets, or all the keys of multiple dictionaries

d e
c a
A
d1 = {'p': 1, 'y': 2}

d2 = {'t': 3, 'h': 4}
y te
d3 = {'h': 5, 'o': 6, 'n': 7}
th B
Note that the key 'h' is in both d2 and d3

M a
l = [*d1, *d2, *d3]
t ©
à ['p', 'y', 't', 'h', 'h', 'o', 'n']

i g h (order is not guaranteed)

yr
s = {*d1, *d2, *d3} à {'p', 'y', 't', 'h, 'o', 'n'}

o p
C
The ** unpacking operator

When working with dictionaries we saw that * essentially iterated the keys

m y
d e
d = {'p': 1, 'y': 2, 't': 3, 'h': 4}

c a
a, *b = d a = 'p'

te A
b = ['y', 't', 'h']

y
(again, order is not guaranteed)

B
a th
M
We might ask the question: can we unpack the key-value pairs of the dictionary?

©
h t
Yes!

yr i g
o p
We need to use the ** operator

C
Using **

d1 = {'p': 1, 'y': 2}

m y
d2 = {'t': 3, 'h': 4}
d e
d3 = {'h': 5, 'o': 6, 'n': 7}
c a
Note that the key 'h' is in both d2 and d3

te A
d = {**d1, **d2, **d3}
B y
(note that the ** operator cannot be used in the LHS of an assignment)

a th
à {'p': 1, 'y': 2, 't': 3, 'h': 5, 'o': 6, 'n': 7}

© M
t
Note that the value of 'h' in d3 "overwrote" the first value of 'h' found in d2

h
yr i
(order not guaranteed) g
o p
C
Using **

m y
You can even use it to add key-value pairs from one (or more) dictionary into a dictionary literal:

d e
d1 = {'a': 1, 'b': 2}
c a
te A
{'a': 10, 'c': 3, **d1}
y
à {'a': 1, 'b': 2, 'c': 3}

B
{**d1, 'a': 10, 'c': 3}
a th
à {'a': 10, 'b': 2, 'c': 3}

© M
(order not guaranteed)
h t
yr i g
o p
C
Nested Unpacking

Python will support nested unpacking as well.

m y
l = [1, 2, [3, 4]] Here, the third element of the list is itself a list.

d e
We can certainly unpack it this way: a, b, c = l a = 1
c a b = 2 c = [3, 4]

te A
y
We could then unpack c into d and e as follows: d, e = c d = 3 e = 4

th B
Or, we could simply do it this way:

M a
a, b, (c, d) = [1, 2, [3, 4]] a = 1 b = 2

©
c = 3 d = 4

h t
i g
Since strings are iterables too:

yr
a, *b, (c, d, e) = [1, 2, 3, 'XYZ']

a = 1

o p b = [2, 3] c, d, e = 'XYZ'

C à c = 'X' d = 'Y' e = 'Z'

The * operator can only be used once in the LHS an unpacking assignment

m y
e
How about something like this then?

a, *b, (c, *d) = [1, 2, 3, 'python']

c a d
te A
y
Although this looks like we are using * twice in the same expression, the second * is actually in a

B
h
nested unpacking – so that's OK

a t
a = 1 b = [2, 3]
© M
c, *d = 'python'

h t
yr i g à c = 'p'

o p d = ['y', 't', 'h', 'o', 'n']

C
Try doing the same thing using slicing…
 m y
d e
c a
te A
B y
t
*args
a h
© M
h t
yr i g
o p
C
Recall from iterable unpacking

m y
e
a, b, c = (10, 20, 30) à a = 10 b = 20 c = 30

c a d
A
Something similar happens when positional arguments are passed to a function:

te
B y
def func1(a, b, c):

a th
M
# code

t ©
func1(10, 20, 30)

i g
à
h a = 10 b = 20 c = 30

p yr
C o
*args

Recall also: a, b, *c = 10, 20, 'a', 'b' à a=10 b=20

m y
c=['a', 'b']

d e
c a
Something similar happens when positional arguments are passed to a function:

te A
y
def func1(a, b, *c):
# code

th B this is a tuple, not a list

func1(10, 20, 'a', 'b') à

M
a=10
a b=20

t © c=('a', 'b')

i g h
yr
The * parameter name is arbitrary – you can make it whatever you want

o p
It is customary (but not required) to name it *args

C
def func1(a, b, *args):
# code
*args exhausts positional arguments

m y
You cannot add more positional arguments after *args

d e
c a
A
this is actually OK – covered in next lecture

te
y
def func1(a, b, *args, d):
# code

th B
M a
This will not work!

t ©
func1(10, 20, 'a', 'b', 100)

i g h
p yr
C o
Unpacking arguments

def func1(a, b, c):

m y
# code
d e
c a
l = [10, 20, 30]

te A
B y
This will not work: func1(l)

a th
© M
But we can unpack the list first and then pass it to the function

h t
func1(*l)

yr i g
à a = 10 b = 20 c = 30

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Keyword Arguments

m y
e
Recall that positional parameters can, optionally be passed as named (keyword) arguments

c a d
A
def func(a, b, c):

te
# code

B y
func(1, 2, 3) à
a th
a = 1, b = 2, c = 3

© M
t
func(a=1, c=3, b=2) à a = 1, b = 2, c = 3

i g h
p yr
Using named arguments in this case is entirely up to the caller.

C o
Mandatory Keyword Arguments

We can make keyword arguments mandatory.

m y
e
To do so, we create parameters after the positional parameters have been exhausted.

d
def func(a, b, *args, d):
c a
#code

te A
B y
In this case, *args effectively exhausts all positional arguments

a th
and d must be passed as a keyword (named) argument

func(1, 2, 'x', 'y', d = 100)

© M
h t
à a = 1, b = 2, args = ('x', 'y'), d = 100

yr i g
p
func(1, 2, d = 100)

o
C
à a = 1, b = 2, args = (), d = 100

func(1, 2) d was not a keyword argument

We can even omit any mandatory positional arguments:

def func(*args, d):

m y
#code

d e
c a
func(1, 2, 3, d=100) à args = (1, 2, 3), d = 100

te A
func(d=100) à args = (), d = 100
B y
a th
M
In fact we can force no positional arguments at all:

©
t
def func(*, d):

h
#code * indicates the "end" of positional arguments

yr i g
o p
func(1, 2, 3, d=100) à Exception

C
func(d=100) à d = 100
Putting it together

m y
def func(a, b=1, *args, d, e=True):
e
def func(a, b=1, *, d, e=True):

d
a
# code # code

Ac
te
a: mandatory positional argument (may be specified using a named

y
argument)

th B
b: optional positional argument (may be specified positionally, as a

a
named argument, or not at all), defaults to 1

M
args: catch-all for any (optional)

t © *: no additional positional arguments allowed

i g h
additional positional arguments

p yr
d: mandatory keyword argument

C o e: optional keyword argument, defaults to True

 m y
d e
c a
te A
B y
th
**kwargs
a
© M
h t
yr i g
o p
C
**kwargs

m y
*args is used to scoop up variable amount of remaining positional arguments

d e à tuple

c a
The parameter name args is arbitrary – * is the real performer here

te A
B y
h
**kwargs is used to scoop up a variable amount of remaining keyword arguments à dictionary

a t
The parameter name kwargs is arbitrary – ** is the real performer here

© M
h t
yr i g
**kwargs can be specified even if the positional arguments have not been exhausted

p
(unlike keyword-only arguments)

o
C
No parameters can come after **kwargs
Example

m y
def func(*, d, **kwargs):

d e
a
# code

Ac
func(d=1, a=2, b=3) à d = 1
y te
th B
kwargs = {'a': 2, 'b': 3}

M a
func(d=1)
t ©
à d = 1

i g h kwargs = {}

p yr
C o
Example

def func(**kwargs):
# code
m y
d e
func(a=1, b=2, c=3)
c a
à kwargs = {'a': 1, 'b': 2, 'c': 3}

func() à kwargs = {}

te A
B y
a th
def func(*args, **kwargs):
# code
© M
h t
i g
func(1, 2, a=10, b=20)

yr
à args = (1, 2)

p
kwargs = {'a': 10, 'b': 20}

C
func() o à args = ()
kwargs = {}
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Recap

m y
positional arguments
e
keyword-only arguments

d
c a
specific may have default values
A
after positional arguments have ben

te
exhausted

B y
*args collects, and exhausts
remaining positional
a th specific may have default values

arguments

© M
*
h t
indicates the end of **kwargs collects any remaining

yr i g
positional arguments
(effectively exhausts)
keyword arguments

o p
C
 scoops up any indicates no more scoops up any additional
additional positional positional args keyword args

y
args

e m
a, b, c=10 *args / * kw1, kw2=100
c a d
**kwargs

te A
B y
positional parameters

a th specific keyword-only args

M
can have default values can have default values

©
non-defaulted params are mandatory args non-defaulted params are mandatory args

h t
user may specify them using keywords user must specify them using keywords

yr i g if used, * or *args must also be used

o p
C
Examples

def func(a, b=10)

m y
d e
def func(a, b, *args)

c a
def func(a, b, *args, kw1, kw2=100)

te A
def func(a, b=10, *, kw1, kw2=100)
B y
a th
def func(a, b, *args, kw1, kw2=100, **kwargs)

© M
t
def func(a, b=10, *, kw1, kw2=100, **kwargs)

i g h
yr
def func(*args)

o p
def func(**kwargs)

C
def func(*args, **kwargs)
Typical Use Case: Python's print() function

m y
d e
c a
te A
B y
a th
© M
arbitrary number of positional arguments

t
*objects

i g h
yr
after that are keyword-only arguments

o p
they all have default values, so they are all optional

C
Typical Use Cases

Often, keyword-only arguments are used to modify the default behavior of a function

m y
such as the print() function we just saw
d e
c a
def calc_hi_lo_avg(*args, log_to_console=False):

te A
hi = int(bool(args)) and max(args)
lo = int(bool(args)) and min(args)
B y
avg = (hi + lo)/2
if log_to_console:
a th
return avg
© M
print("high={0}, low={1}, avg={2}".format(hi, lo, avg))

h t
yr i g
Other times, keyword-only arguments might be used to make

p
things clearer.

C o
Having many positional parameters can become confusing, and
extra care has to be taken to ensure the correct parameters are
passed in the correct sequence.
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
What happens at run-time…

When a module is loaded: all code is executed immediately

m y
d e
c a
Module Code

te A
a = 10

B y
the integer object 10 is created and a references it

a th
the function object is created, and func references it

M
def func(a):
print(a)

t ©
func(a)
i g hthe function is executed

p yr
C o
What about default values?

Module Code
m y
d e
def func(a=10):
c a
the function object is created, and func references it
print(a)

te A
the integer object 10 is evaluated/created

y
and is assigned as the default for a

th B
a
the function is executed
func()

M
by the time this happens, the default value for a has already been

©
h t
evaluated and assigned – it is not re-evaluated when the function is
called

yr i g
o p
C
 So what?

m y
Consider this:

d e
c a
We want to create a function that will write a log entry to the console with a user-specified event

A
date/time. If the user does not supply a date/time, we want to set it to the current date/time.

te
from datetime import datetime
B y
def log(msg, *, dt=datetime.utcnow()):
a th
print('{0}: {1}'.format(dt, msg)

© M
h t
i g
log('message 1') à 2017-08-21 20:54:37.706994 : message 1

p yr
C o
a few minutes later:

log('message 2') à 2017-08-21 20:54:37.706994 : message 2

Solution Pattern

We set a default for dt to None

m y
Inside the function, we test to see if dt is still None
d e
c a
if dt is None, set it to the current date/time

te A
recall that this is equivalent to:

y
otherwise, use what the caller specified for dt

B
if not dt:

a th dt = datetime.utcnow()

from datetime import datetime

© M
def log(msg, *, dt=None):

h t
i g
dt = dt or datetime.utcnow()

yr
print('{0}: {1}'.format(dt, msg)

o p
C
In general, always beware of using a mutable object (or a
callable) for an argument default
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
First-Class Objects

can be passed to a function as an argument

m y
d e
can be returned from a function
c a
can be assigned to a variable
te A
B y
th
can be stored in a data structure (such as list, tuple, dictionary, etc.)

a
© M
h t
g
Types such as int, float, string, tuple, list and many more are first-class objects.

yr i
o p
C
Functions (function) are also first-class objects
 Higher-Order Functions

m y
Higher-order functions are functions that:

d e
c a
take a function as an argument
te A
(e.g. the simple timer we wrote in the last section)

B y
and/or

a th
return a function
© M (plenty of that when we cover decorators in the next section)

h t
yr i g
o p
C
Topics in this section

m y
function annotations and documentation

d e
lambda expressions and anonymous functions
c a
te A
y
callables

th B
a
function introspection

© M
built-in higher order functions (such as sorted, map, filter)

h t
g
some functions in the functools module (such as reduce, all, any)

yr i
partials

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Docstrings

We have seen the help(x) function before

y
à returns some documentation (if available) for x

m
d e
We can document our functions (and modules, classes, etc) to achieve the same result using
docstrings à PEP 257
c a
te A
If the first line in the function body is a string (not an assignment, not a comment, just a string by
itself), it will be interpreted as a docstring
B y
def my_func(a):
a th help(my_func)
"documentation for my_func"
return a
© M à my_func(a)

h t documentation for my_func

yr i g
o p
Multi-line docstrings are achieved using…

C multi-line strings!
Where are docstrings stored? In the function's __doc__ property

def fact(n):

m y
e
"""Calculates n! (factorial function)

Inputs:
c a d
A
n: non-negative integer

te
Returns:

"""
the factorial of n

B y
...

a th
fact.__doc__

© M
à 'Calculates n! (factorial function)\n \n
n: non-negative integer\n Returns:\n
Inputs:\n
the

t
factorial of n\n '

h
help(fact)

yr i g
à fact(n)

p
Calculates n! (factorial function)

C o Inputs:
n: non-negative integer
Returns:
the factorial of n
Function Annotations

Function annotations give us an additional way to document our functions:

m y
à PEP 3107

d e
a
def my_func(a: <expression>, b: <expression>) -> <expression>:
pass

Ac
y te
return a * b
th B
def my_func(a: 'a string', b: 'a positive integer') -> 'a string':

M a
help(my_func)
t ©
à my_func(a:'a string', b:'a positive integer') -> 'a string'

i g h
p yr
my_func.__doc__ à empty string

C o
Annotations can be any expression

m y
e
def my_func(a: str, b: 'int > 0') -> str:
return a*b

c a d
def my_func(a: str, b: [1, 2, 3]) -> str:

te A
return a*b

B y
x = 3
a th
y = 5

© M
def my_func(a: str) -> 'a repeated ' + str(max(x, y)) + ' times':
return a*max(x, y)
h t
yr i g
help(my_func) à my_func(a:str) -> 'a repeated 5 times'

o p
C
Default values, *args, **kwargs

can still be used as before

m y
d e
c a
def my_func(a: str = 'xyz', b: int = 1) -> str:

te A
y
pass

th B
def my_func(a: str = 'xyz',
Ma
©
*args: 'additional parameters',
b: int = 1,

h t
g
**kwargs: 'additional keyword only params') -> str:
pass

yr i
o p
C
Where are annotations stored?

In the __annotations__ property of the function

m y
d e
à dictionary keys are the parameter names
c a
te A
for a return annotation, the key is return

values are the annotations

B y
a th
© M
t
def my_func(a: 'info on a', b: int) -> float:
pass

i g h
p yr
C o
my_func.__annotations__

à {'a': 'info on a', 'b': int, 'return': float}

Where does Python use docstrings and annotations?

It doesn't really!
m y
d e
Mainly used by external tools and modules

c a
te A
Example: apps that generate documentation from your code (Sphinx)

B y
a th
Docstrings and annotations are entirely optional, and do not "force" anything in our Python code

© M
t
We'll look at an enhanced version of annotations in an upcoming section on type hints

h
yr i g
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
What are Lambda Expressions?

We already know how to create functions using the def statement

m y
Lambda expressions are simply another way to create functions
d e
anonymous functions

c a
te A
the : is required, even for zero arguments
keyword
parameter list optional

B y
a th this expression is evaluated and
returned when the lambda

M
lambda [parameter list]: expression function is called

t © (think of it as the "body" of the function)

i g h
the expression returns a function object

p yr
that evaluates and returns the expression when it is called

C o
it can be assigned to a variable
passed as an argument to another function
it is a function, just like one created with def
Examples

lambda x: x**2
m y
d e
lambda x, y: x + y

c a
lambda : 'hello'

te A
lambda s: s[::-1].upper()
B y
a th
type(lambda x: x**2)
© M
à function

h t
yr i g
Note that these expressions are function objects, but are not "named"

o p
C
à anonymous functions

Lambdas, or anonymous functions, are NOT equivalent to closures

Assigning a Lambda to a Variable Name

my_func = lambda x: x**2

m y
d e
type(my_func) à function

c a
my_func(3) à9

te A
my_func(4) à 16
B y
a th
identical to: def my_func(x):
return x**2
© M
h t
yr i g
type(my_func) à function

o p my_func(3) à9

C my_func(4) à 16
Passing as an Argument to another Function

def apply_func(x, fn):

m y
return fn(x)

d e
c a
apply_func(3, lambda x: x**2) à 9
te A
B y
apply_func(2, lambda x: x + 5)

a th
à 7

© M
apply_func('abc', lambda x: x[1:] * 3) à bcbcbc

h t
equivalently:
yr i g
o
def fn_1(x):
p
C
return x[1:] * 3

apply_func('abc', fn_1) à bcbcbc

Limitations

The "body" of a lambda is limited to a single expression

m y
d e
no assignments lambda x: x = 5
a
lambda x: x = x + 5

c
te A
no annotations
def func(x: int):
return x**2
B y
lambda x:int : x*2

a th
single logical line of code
M
à line-continuation is OK, but still just one expression

©
h t
g
lambda x: x * \

yr i math.sin(x)

o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Functions are first-class objects

m y
They have attributes __doc__ __annotations__

d e
c a
We can attach our own attributes

te A
B y
def my_func(a, b):
return a + b
a th
© M
t
my_func.category = 'math'

i g h
my_func.sub_category = 'arithmetic'

p yr
C o
print(my_func.category)

print(my_func.sub_category)
à math

à arithmetic
The dir() function

m
dir() is a built-in function that, given an object as an argument, will return a list of validy
attributes for that object
d e
c a
dir(my_func)

te A
B y
h
['__annotations__', '__call__', '__class__', '__closure__',

a t
'__code__', '__defaults__', '__delattr__', '__dict__',
'__dir__', '__doc__', '__eq__', '__format__', '__ge__',

M
'__get__', '__getattribute__', '__globals__', '__gt__',

©
t
'__hash__', '__init__', '__init_subclass__', '__kwdefaults__',

h
'__le__', '__lt__', '__module__', '__name__',

i g
'__ne__', '__new__', '__qualname__', '__reduce__',

yr
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__',

p
'__str__', '__subclasshook__', 'category', 'sub_category']

o
C
Function Attributes: __name__, __defaults__, __kwdefaults__

m y
e
__name__ à name of function

__defaults__
c a
à tuple containing positional parameter defaults d
__kwdefaults__
te A
à dictionary containing keyword-only parameter defaults

B y
def my_func(a, b=2, c=3, *, kw1, kw2=2):
a th
pass

© M
h t
my_func.__name__
yr i g à my_func

o p
C
my_func.__defaults__ à (2, 3)

my_func.__kwdefaults__ à {'kw2': 2}
Function Attribute: __code__

def my_func(a, b=1, *args, **kwargs):

m y
i = 10

d e
a
b = min(i, b)

c
my_func.__code__
return a * b

A
à <code object my_func at 0x00020EEF … >

te
B y
This __code__ object itself has various properties, which include:

a th
co_varnames

© M
parameter and local variables

h t
my_func.__code__.co_varnames à ('a', 'b', 'args', 'kwargs', 'i')

yr i g
parameter names first, followed by local variable names

o p
C
co_argcount number of parameters
my_func.__code__.co_argcount à 2
does not count *args and **kwargs!
The inspect Module import inspect

ismethod(obj) isfunction(obj) isroutine(obj) and many others…

m y
d e
What's the difference between a function and a method?
c a
te A
Classes and objects have attributes – an object that is bound (to the class or the object)
An attribute that is callable, is called a method

B y
def my_func():
a th
func is bound to my_obj, an instance of MyClass
pass
M
isfunction(my_func) à True

©
def MyClass:

h t ismethod(my_func) à False

g
def func(self):
pass

yr i isfunction(my_obj.func) à False

o p
my_obj = MyClass() ismethod(my_obj.func) à True

C isroutine(my_func) à True
isroutine(my_obj.func) à True
Code Introspection

m y
We can recover the source code of our functions/methods

d e
c a
inspect.getsource(my_func)

te A
à a string containing our entire def statement, including

y
annotations, docstrings, etc

th B
a
We can find out in which module our function was created

M
t ©
g
inspect.getmodule(my_func)

i h à <module '__main__'>

p yr
o
inspect.getmodule(print) à <module 'builtins' (built-in)>

C
inspect.getmodule(math.sin) à <module 'math' (built-in)>
Function Comments

# setting up variable
m y
i = 10

d e
# TODO: Implement function
c a
# some additional notes
def my_func(a, b=1):
te A
# comment inside my_func

B y
h
pass

a t
inspect.getcomments(my_func)
© M
h t
i g
à '# TODO: Implement function\n# some additional notes'

p yr
C o
Many IDE's support the TODO comment to flag functions and other callables

Note that this is not the same as docstrings

Callable Signatures

à Signature instance
m y
e
inspect.signature(my_func)

Contains an attribute called parameters

c a d
te A
y
Essentially a dictionary of parameter names (keys), and metadata about the parameters (values)

B
keys à parameter name

a th
M
values à object with attributes such as name, default, annotation, kind

©
kind
h t
g
POSITIONAL_OR_KEYWORD

yr i
VAR_POSITIONAL

o p
KEYWORD_ONLY

C VAR_KEYWORD

POSITIONAL_ONLY
Callable Signatures

def my_func(a: 'a string',

m y
e
b: int = 1,
*args: 'additional positional args',
kw1: 'first keyword-only arg',
c a d
A
kw2: 'second keyword-only arg' = 10,

te
**kwargs: 'additional keyword-only args') -> str:
"""does something
or other"""
B y
pass

a th
© M
t
for param in inspect.signature(my_func).parameters.values():

i g h
print('Name:', param.name)

yr
print('Default:', param.default)
print('Annotation:', param.annotation)

o p
print('Kind:', param.kind)

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
What are callables?

m y
e
any object that can be called using the () operator callables always return a value

like functions and methods but it goes beyond just those two…
c a d
te A
y
many other objects in Python are also callable

th B
a
To see if an object is callable, we can use the built-in function: callable

M
©
callable(print) à True

h t
g
callable('abc'.upper) à True

yr i
callable(str.upper) à True

o p
C
callable(callable) à True

callable(10) à False
Different Types of Callables

built-in functions print len callable

m y
d e
a
built-in methods a_str.upper a_list.append

user-defined functions
Ac
created using def or lambda expressions

methods
y
functions bound to an object te
th B
classes
a
MyClass(x, y, z)
à

© M
__new__(x, y, z) à creates the new object

h t
à __init__(self, x, y, z)

yr i g à returns the object (reference)

o p
C
class instances if the class implements __call__ method

generators, coroutines, asynchronous generators

 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Higher order functions

m y
e
A function that takes a function as a parameter and/or returns a function as its return value

d
c a
Example: sorted

te A
B y
h
map

a t
modern alternative à list comprehensions and generator expressions
filter

© M
h t
yr i g
o p
C
The map function

m y
map(func, *iterables)

d e
c a
*iterables à a variable number of iterable objects

te A
B y
h
func à some function that takes as many arguments as there are iterable

a
objects passed to iterables
t
© M
h t
i g
map(func, *iterables) will then return an iterator that calculates the

yr
function applied to each element of the iterables

o p
C
The iterator stops as soon as one of the iterables has been exhausted
so, unequal length iterables can be used
Examples

l = [2, 3, 4]

m y
def sq(x):
d e
return x**2

c a
list(map(sq, l)) à [4, 9, 16]
te A
B y
a th
M
l1 = [1, 2, 3]
l2 = [10, 20, 30]

t ©
h
def add(x, y):
return x + y

yr i g
o p
list(map(add, l1, l2)) à [11, 22, 33]

C
list(map(lambda x, y: x + y, l1, l2)) à [11, 22, 33]
The filter function

m y
filter(func, iterable)

d e
c a
iterable à a single iterable

te A
B y
h
func à some function that takes a single argument

a t
M
filter(func, iterable) will then return an iterator that contains all the

©
t
elements of the iterable for which the function called on it is Truthy

i g h
p yr
If the function is None, it simply returns the elements of iterable that are Truthy

C o
Examples

l = [0, 1, 2, 3, 4]

m y
d e
a
list(filter(None, l)) à [1, 2, 3, 4]

Ac
y te
def is_even(n):
return n % 2 == 0
th B
M a
list(filter(is_even, l))
t ©
à [0, 2, 4]

i g h
p yr
C o
list(filter(lambda n: n % 2 == 0, l)) à [0, 2, 4]
The zip function zip(*iterables)

[1, 2, 3, 4]
m y
e
zip

d
(1, 10), (2, 20), (3, 30), (4, 40)

a
[10, 20, 30, 40]

Ac
y te
B
[1, 2, 3]

[10, 20, 30]

zip

a th
(1, 10, 'a'), (2, 20, 'b'), (3, 30, 'c')

['a', 'b', 'c']

© M
h t
[1, 2, 3, 4, 5]

yr i g zip

p
(1, 10), (2, 20), (3, 30)

o
[10, 20, 30]

C
Examples

l1 = [1, 2, 3]
m y
l2 = [10, 20, 30, 40]
d e
l3 = 'python'

c a
list(zip(l1, l2, l3))
A
à [(1, 10, 'p'), (2, 20, 'y'), (3, 30, 't')]

te
B y
a th
M
l1 = range(100)

©
l2 = 'abcd'

h t
g
list(zip(l1, l2)) à [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

yr i
o p
C
List Comprehension Alternative to map

l = [2, 3, 4]
m y
d e
def sq(x):

c a
A
return x**2
list(map(lambda x: x**2, l)) à [4, 9, 16]
list(map(sq, l))

y te
th B
result = []
M a
for x in l:
result.append(x**2)
t ©
result à [4, 9, 16]

i g h
p yr
o
[x**2 for x in l]

C
à [4, 9, 16]

[<expression> for <varname> in <iterable>]

List Comprehension Alternative to map

l1 = [1, 2, 3]
m y
l2 = [10, 20, 30]
d e
c a
list(map(lambda x, y: x + y, l1, l2)) à [11, 22, 33]

te A
B y
Remember:
a th
zip(l1, l2) à [(1, 10), (2, 20), (3, 30)]

[x + y for x, y in zip(l1, l2)]

© M à [11, 22, 33]

h t
yr i g
o p
C
List Comprehension Alternative to filter

m y
l = [1, 2, 3, 4]

d e
list(filter(lambda n: n % 2 == 0, l)) à [2, 4]
c a
te A
[x for x in l if x % 2 == 0] à [2, 4]
B y
a th
M
[<expression1> for <varname> in <iterable> if <expression2>]

©
h t
yr i g
o p
C
Combining map and filter

m y
e
l = range(10)
list(filter(lambda y: y < 25, map(lambda x: x**2, l)))

c a d
à [0, 1, 4, 9, 16]

te A
Using a list comprehension is much clearer:

B y
a th
© M
[x**2 for x in range(10) if x**2 < 25] à [0, 1, 4, 9, 16]

h t
yr i g
o p
C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Reducing Functions in Python

y
These are functions that recombine an iterable recursively, ending up with a single return value

m
Also called accumulators, aggregators, or folding functions.
d e
c a
te A
y
Example: Finding the maximum value in an iterable

a0, a1, a2, …, an-1

th B
max(a, b) à maximum of a and b
M a
result = a0
t ©
i g
result = max(result, a1)
h
yr
result = max(result, a2)

p
…

C o
result = max(result, an-1)

à max value in a0, a1, a2, …, an-1

Because we have not studied iterables in general, we will stay with the special case of sequences.
(i.e. we can use indexes to access elements in the sequence)

m y
Using a loop
d e
a
result = 5

c
l = [5, 8, 6, 10, 9]

te A
result = max(5, 8) = 8

max_value = lambda a, b: a if a > b else b

B y result = max(8, 6) = 8

def max_sequence(sequence):
a th result = max(8, 10) = 10

M
result = sequence[0]
for e in sequence[1:]: result = max(10, 9) = 10

t ©
result = max_value(result, e)

h
return result result à 10

yr i g
o p
C
Notice the sequence of steps:

l = [5, 8, 6, 10, 9]

m y
d e
c a
A
5

te
max(5, 8)
8

B y
max(8, 6)

a th
M
8

t ©
max(8, 10)

i g h 10

p yr max(10, 9)

C o 10

result à 10
To calculate the min:

l = [5, 8, 6, 10, 9]

m y
min_value = lambda a, b: a if a < b else b
d e
c a
A
def min_sequence(sequence):
All we really needed to do was to change

te
result = sequence[0]
the function that is repeatedly applied.
for e in sequence[1:]:
result = min_value(result, e)
B y
return result

a th
In fact we could write:

© M
def _reduce(fn, sequence):

h t
result = sequence[0]

yr i
for x in sequence[1:]: g
p
result = fn(result, x)

o
return result

C
_reduce(lambda a, b: a if a > b else b, l) à maximum
_reduce(lambda a, b: a if a < b else b, l) à minimum
Adding all the elements in a list

add = lambda a, b: a+b

m y
d e
a
result = 5

l = [5, 8, 6, 10, 9]
Ac
result = add(5, 8) = 13

y te
result = add(13, 6) = 19
def _reduce(fn, sequence):

th Bresult = add(19, 10) = 29

a
result = sequence[0]

M
for x in sequence[1:]: result = add(29, 9) = 38
result = fn(result, x)
return result
t © result à 38

i g h
_reduce(add, l)

p yr
C o
The functools module

m y
Python implements a reduce function that will handle any iterable, but works similarly to what
we just saw

d e
c a
from functools import reduce

te A
B y
h
l = [5, 8, 6, 10, 9]

a t
© M
reduce(lambda a, b: a if a > b else b, l) à max à 10

h t
i g
reduce(lambda a, b: a if a < b else b, l)

yr
à min à 5

o p
reduce(lambda a, b: a + b, l) à sum à 38

C
reduce works on any iterable

m y
reduce(lambda a, b: a if a < b else b, {10, 5, 2, 4}) à 2

d e
c a
reduce(lambda a, b: a if a < b else b, 'python')

te A à h

B y
th
reduce(lambda a, b: a + ' ' + b, ('python', 'is', 'awesome!'))

a
M
à 'python is awesome'

t ©
i g h
p yr
C o
Built-in Reducing Functions

Python provides several common reducing functions:

m y
d e
min min([5, 8, 6, 10, 9]) à5
c a
max max([5, 8, 6, 10, 9]) à 10
te A
B y
h
sum sum([5, 8, 6, 10, 9]) à 38

a t
any any(l) à

© M
True if any element in l is truthy

t
False otherwise

h
all
yr
all(l) à i g True if every element in l is truthy

o p False otherwise

C
Using reduce to reproduce any

l = [0, '', None, 100]

m y
d e
c a
A
result = bool(0) or bool('') or bool(None) or bool(100)

y te
Note: 0 or '' or None or 100 à 100 but we want our result to be True/False

th B
so we use bool()

a
Here we just need to repeatedly apply the or operator to the truth values of each element

M
©
result = bool(0) à False

h t
g
result = result or bool('') à False

yr i
p
result = result or bool(None) à False

C o
result = result or bool(100) à True

reduce(lambda a, b: bool(a) or bool(b), l) à True

Calculating the product of all elements in an iterable

No built-in method to do this

m y
d e
c a
But very similar to how we added all the elements in an iterable or sequence:

[1, 3, 5, 6]
te A
y
à 1 * 3 * 5 * 6

th B
reduce(lambda a, b: a * b, l)

M a
res = 1

t ©
res = res * 3 = 3

i g h
p yr
res = res * 5 = 3 * 5 = 15

C o
res = res * 6 = 15 * 6 = 90 = 1 * 3 * 5 * 6
Special case: Calculating n!

n! = 1 * 2 * 3 * … * n 5! = 1 * 2 * 3 * 4 * 5
m y
d e
range(1, 6) à 1, 2, 3, 4, 5
c a
te A
y
range(1, n+1) à 1, 2, 3, …, n

th B
a
To calculate n! we need to find the product of all the elements in range(1, n+1)

© M
h t
reduce(lambda a, b: a * b, range(1, 5+1)) à 5!

yr i g
o p
C
The reduce initializer

The reduce function has a third (optional) parameter: initializer

m y
(defaults to None)

d e
a
If it is specified, it is essentially like adding it to the front of the iterable.

c
te A
It is often used to provide some kind of default in case the iterable is empty.

l = []
B y
reduce(lambda x, y: x+y, l)
th
à exception

a
l = []

© M
t
reduce(lambda x, y: x+y, l, 1) à1

i g h
yr
l = [1, 2, 3]

p
reduce(lambda x, y: x+y, l, 1) à7

C
l = [1, 2, 3]o
reduce(lambda x, y: x+y, l, 100) à 106
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Reducing Function Arguments

def my_func(a, b, c):

m y
print(a, b, c)
d e
c a
def fn(b, c):
fn(20, 30)
te A
à 10, 20, 30
return my_func(10, b, c)

B y
f = lambda b, c: my_func(10, b, c)
a th
f(20, 30) à 10, 20, 30

© M
h t
yr i g
from functools import partial

o p
f = partial(my_func, 10)

C
f(20, 30) à 10, 20, 30
Handling more complex arguments

def my_func(a, b, *args, k1, k2, **kwargs):

m y
print(a, b, args, k1, k2, kwargs)
d e
c a
te A
def f(b, *args, k2, **kwargs):
B y
th
return my_func(10, b, *args, k1='a', k2=k2, **kwargs)

a
© M
h t
i g
f = partial(my_func, 10, k1='a')

yr
o p
C
Handling more complex arguments

m y
e
def pow(base, exponent):
return base ** exponent

c a d
te A
y
square = partial(pow, exponent=2)
cube = partial(pow, exponent=3)

th B
M a
t ©
h
square(5) à 25

cube(5) à 75
yr i g
o p
C
cube(base=5) à 75

!! square(5, exponent=3) à 75
Beware!!

y
You can use variables when creating partials
but there arises a similar issue to argument default values

e m
def my_func(a, b, c):
print(a, b, c)
c a d
te A
the memory address of 10 is baked in to the partial

a = 10
B y
f = partial(my_func, a)

a th
f(20, 30) à 10, 20, 30
© M
a now points to a different memory address

h t
g
but the partial still points to the original object (10)
a = 100
yr i
f(20, 30)
o p à 10, 20, 30

C
If a is mutable (e.g. a list), then it's contents can be changed
 m y
d e
c a
te A
B y
t
operator
a h
© M
h t
yr i g
o p
C
Functional Equivalents to Operators

In the last lecture we wrote code such as:

m y
d e
c a
A
l = [2, 3, 4]

reduce(lambda a, b: a * b, l)
y te
th B
a
We used a lambda expression to create a functional version of the * operator

M
t ©
This is something that happens quite often, so the operator module was created

i g h
p yr
o
This module is a convenience module.

C
You can always use your own functions and lambda expressions instead.
The operator module

Arithmetic Functions
m y
d e
add(a, b)
c a
te A
mul(a, b)

B y
pow(a, b)
a th
© M
mod(a, b)

h t
yr i g
p
floordiv(a, b)

C
neg(a)
o
and many more…
Comparison and Boolean Operators

m y
lt(a, b) gt(a, b) eq(a, b)
d e
c a
le(a, b) ge(a, b) ne(a, b)

te A
is_(a,b) is_not(a,b)

B y
a th
and_(a, b)

© M
or_(a,b)
h t
not_(a,b)
yr i g
o p
C
Sequence/Mapping Operators

m y
e
concat(s1, s2)

c a d
A
contains(s, val)

countOf(s, val)
y te
th B
getitem(s, i)

M a
setitem(s, i, val)
t ©
i g h mutable objects
variants that use slices

delitem(s, i)

p yr
C o
Item Getters

The itemgetter function returns a callable

m y
d e
c a
getitem(s, i)

te A
takes two parameters, and returns a value: s[i]

s = [1, 2, 3]

B y
getitem(s, 1) à 2

a th
itemgetter(i)
© M
returns a callable which takes one parameter: a sequence object

h t
yr i g
f = itemgetter(1)

o p
s = [1, 2, 3] s = 'python'

Cf(s) à 2 f(s) à 'y'

Item Getters

We can pass more than one index to itemgetter:

m y
d e
c a
l = [1, 2, 3, 4, 5, 6]
s = 'python'
te A
B y
f = itemgetter(1, 3, 4)

a th
f(l) à (2, 4, 5)
© M
h t
f(s)

yr i g
à ('y', 'h', 'o')

o p
C
Attribute Getters

m y
The attrgetter function is similar to itemgetter, but is used to retrieve object attributes

d e
It also returns a callable, that takes the object as an argument

c a
Suppose my_obj is an object with three properties:

te A
my_obj.a à 10
B y
my_obj.b à 20
my_obj.c à 30
a th
© M
f = attrgetter('a')

h t f(my_obj) à 10

yr i g
f = attrgetter('a', 'c') f(my_obj) à (10, 30)

o p
C
Can also call directly:

attrgetter('a', 'b', 'c')(my_obj) à (10, 20, 30)

Calling another Callable

Consider the str class that provides the upper() method:

m y
s = 'python' s.upper() à PYTHON

d e
c a
A
f = attrgetter('upper')

f(s) à returns the upper method of s

y te
h B
it is a callable, and can be called using ()

t
f(s)() à PYTHON
M a
©
attrgetter('upper')(s)() à PYTHON

t
i g h
Or, we can use the slightly simpler methodcaller function

p yr
methodcaller('upper')('python')

o
à PYTHON

C
Basically, methodcaller retrieves the named attribute and calls it as well

It can also handle more arguments, as we'll in the code

 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
 y
Variable Scopes local scope

global scope
e m
nonlocal scope

c a d
nested scopes

te A
B y
Closures what they are

a th (they are not equivalent to lambdas!)

M
closure scopes

©
h t
g
Decorators what they are

yr ihow they are related to closures

o p convenience of using @

C
Applications
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Scopes and Namespaces

When an object is assigned to a variable a = 10

m y
d e
that variable points to some object

c a
and we say that the variable (name) is bound to that object

te A
B y
That object can be accessed using that name in various parts of our code

But not just anywhere!

a th
© M
t
That variable name and it's binding (name and object) only "exist" in specific parts of our code

i g h
yr
the portion of code where that name/binding is defined, is called the lexical scope of the variable

o p
these bindings are stored in namespaces

C
(each scope has its own namespace)
The Global Scope

m y
The global scope is essentially the module scope.

d e
c a
A
It spans a single file only.

y te
th B
There is no concept of a truly global (across all the modules in our entire app) scope in Python.

M a
The only exception to this are some of the built-in globally available objects, such as:

True False None

t
dict
© print

i g h
yr
The built-in and global variables can be used anywhere inside our module

p
C o
including inside any function
 Global scopes are nested inside the built-in scope

m y
e
Built-in Scope name space
name
space
var1

c a 0xA345E
d
A
Module1 func1 0xFF34A

te
Scope name

y
space

th B
Module2
Scope name
M a
©
space

h t
yr i g
o p
If you reference a variable name inside a scope and Python does not find it in that scope's namespace

C
it will look for it in an enclosing scope's namespace
Examples

module1.py
m y
Python does not find True or print in the current (module/global) scope
print(True) So, it looks for them in the enclosing scope à built-in
d e
Finds them there
c a
A
à True

y te
module2.py

th B
Python does not find a or print in the current (module/global) scope

a
print(a) So, it looks for them in the enclosing scope à built-in

© M
Find print, but not a à run-time Name Error

h t
module3.py

yr i g
p
print = lambda x: 'hello {0}!'.format(x)

C o
s = print('world') Python finds print in the module scope
So it uses it!
s à hello world!
The Local Scope

m y
When we create functions, we can create variable names inside those functions (using assignments)
e.g. a = 10
d e
c a
A
Variables defined inside a function are not created until the function is called

te
B
Every time the function is called, a new scope is created
y
a th
Variables defined inside the function are assigned to that scope à Function Local scope

© M à Local scope

h t
i g
The actual object the variable references could be different

yr
each time the function is called

o p
C
(this is why recursion works!)
Example these names will considered local
to my_func

def my_func(a, b):

my_func

m y
e
a

d
c = a * b b
return c
c
c a
te A
B y
my_func
a th
M
my_func('z', 2) aà'z'
bà2

t
cà'zz'
© same names, different local scopes

i g h
p yr
C o
my_func(10, 5)
my_func
aà10
bà5
cà50
Nested Scopes

Scopes are often nested

Namespace lookups

m y
e
When requesting the object bound to a variable name:

d
Built-in Scope e.g. print(a)

c a
te A
Python will try to find the object bound to the variable
Module Scope

B y
h
• in current local scope first
Local
Scope
a t• works up the chain of enclosing scopes

M
Local
Scope
Local
t ©
Scope

i g h
p yr
C o
Example
built-in scope
module1.py True

m y
a = 10
global scope
aà10
d e
def my_func(b): my_func
c a
print(True)
print(a) local
te A
print(b) scope
bà300 local

B
scope y
bà'a'

my_func(300)
a th
my_func('a')
© M
h t
i g
Remember reference counting?

yr
p
When my_func(var) finishes running, the scope is gone too!

C o
and the reference count of the object var was bound to (referenced) is decremented

We also say that var goes out of scope

Accessing the global scope from a local scope

When retrieving the value of a global variable from inside a function, Python automatically

m y
e
searches the local scope's namespace, and up the chain of all enclosing scope namespaces

local à global à built-in

c a d
te
What about modifying a global variables value from inside the function?A
B y
a = 0
a th
assignment à Python interprets this as a local variable (at compile-time)

def my_func():
© M à the local variable a masks the global variable a

a = 100

h t built-in

i g
print(a)

yr
my_func() global aà0

p
à 100

o
my_func

C
print(a) à 0 local
aà100
The global keyword

We can tell Python that a variable is meant to be scoped in the global scope
m y
by using the global keyword

d e
c a
a = 0

te A
y
built-in
def my_func():
global a
aà0
th global
B
a
a = 100 my_func
my_func()

© M local
print(a) à 100
h t
yr i g
o p
C
Example

counter = 0
m y
d e
a
def increment():
global counter
counter += 1
Ac
increment()
y te
increment()

th B
a
increment()

print(counter) à 3

© M
h t
yr i g
o p
C
Global and Local Scoping
When Python encounters a function definition at compile-time

m y
e
it will scan for any labels (variables) that have values assigned to them (anywhere in the function)

a
if the label has not been specified as global, then it will be local

c d
te A
variables that are referenced but not assigned a value anywhere in the function will not be local,

y
and Python will, at run-time, look for them in enclosing scopes

a = 10
th B assignment

def func1():
M a
a is referenced only in entire function
at compile time à a non-local
def func4(): at compile time à a local
print(a)
print(a)
t © a = 100

i g h
assignment à when we call func4()

yr
def func2(): at compile time à a local print(a) results in a run-time error
a = 100

o p because a is local, and we are

C
def func3():
global a
assignment
at compile time à a global
referencing it before we have
assigned a value to it!

a = 100 (because we told Python a was global)

 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Inner Functions
We can define functions from inside another function:

m y
def outer_func():
d e
a
global

c
# some code

def inner_func(): local (outer_func)

te A Nested local scopes

# some code

B y
inner_func()

a th
local (inner_func)

© M
t
outer_func()

i g h
p yr
Both functions have access to the global and built-in scopes as well as their respective local scopes

C o
But the inner function also has access to its enclosing scope – the scope of the outer function

That scope is neither local (to inner_func) nor global – it is called a nonlocal scope
Referencing variables from the enclosing scope

Consider this example we have seen before:

m y
d e
module1.py

c a
a = 10

te A
def outer_func():
print(a)
B y
a th
M
outer_func() When we call outer_func, Python sees the reference to a

©
Since a is not in the local scope, Python looks in the enclosing (global) scope

t
i g h
p yr
C o
Referencing variables from the enclosing scope

Now consider this example:

m y
d e
module1.py

c a
def outer_func():
a = 10
te A
B y
h
def inner_func():
print(a)
a t
inner_func()

© M
h t
outer_func()

yr i g
p
When we call outer_func, inner_func is created and called

o
C
When inner_func is called, Python does not find a in the local (inner_func) scope

So it looks for it in the enclosing scope, in this case the scope of outer_func
Referencing variables from the enclosing scope

module1.py

m y
e
a = 10

def outer_func():
c a d
def inner_func():
te A
print(a)

B y
inner_func()

a th
outer_func()

© M
h t
When we call outer_func, inner_func is defined and called

yr i g
When inner_func is called, Python does not find a in the local (inner_func) scope

o p
C
So it looks for it in the enclosing scope, in this case the scope of outer_func

Since it does not find it there either, it looks in the enclosing (global) scope
Modifying global variables

y
We saw how to use the global keyword in order to modify a global variable within a nested scope

m
a = 10
d e
c a
def outer_func1():
global a
te A
a = 1000

B y
outer_func1()
print(a)
a th
M
à 1000

©
We can of course do the same thing from within a nested function
def outer_func2():
h t
def inner_func():
global a
yr i g
p
a = 'hello'

o
C
inner_func()

outer_func2()
print(a) à hello
Modifying nonlocal variables

Can we modify variables defined in the outer nonlocal scope?

m y
d e
a
def outer_func():
x = 'hello'

Ac
def inner_func():
x = 'python'
y te
th B
a
inner_func()

print(x)

© M
outer_func() à hello
h t
yr i g
When inner_func is compiled, Python sees an assignment to x

o p
C
So it determines that x is a local variable to inner_func

The variable x in inner_func masks the variable x in outer_func

Modifying nonlocal variables

y
Just as with global variables, we have to explicitly tell Python we are modifying a nonlocal variable

m
We can do that using the nonlocal keyword
d e
c a
def outer_func():

te A
y
x = 'hello'

def inner_func():
th B
nonlocal x
x = 'python'
M a
inner_func()
t ©
i g h
yr
print(x)

o
outer_func()
p à python

C
Nonlocal Variables

Whenever Python is told that a variable is nonlocal

m y
d e
it will look for it in the enclosing local scopes chain until it first encounters the specified variable
name

c a
A
Beware: It will only look in local scopes, it will not look in the global scope

te
def outer():
B y
x = 'hello'

a th global

M
def inner1(): local (outer) inner1

©
def inner2():
x
nonlocal x
x = 'python'
h t local (inner1)

g
inner2
inner2()

yr i local (inner2)

o
inner1()
p x

C
print(x)

outer() à python
Nonlocal Variables

But consider this example:

global
m y
d e
a
def outer():
x = 'hello'

Ac
local (outer)
x

te
def inner1(): local (inner1)

y
x = 'python' x
def inner2():

th B local (inner2)

a
nonlocal x
x

M
x = 'monty'
print('inner(before)', x) à python
inner2()
t ©
i g h
print('inner(after)', x) à monty

inner1()

p yr
o
print('outer', x) à hello

C
outer()
Nonlocal Variables

def outer():
global
m y
e
x = 'hello'

def inner1():
c a d
local (outer)
nonlocal x
x = 'python'

te A
local (inner1)
x

y
def inner2(): x
nonlocal x
x = 'monty'
th B local (inner2)

a
print('inner(before)', x) à python
inner2()
M
x

©
print('inner(after)', x) à monty

t
inner1()
i g h
yr
print('outer', x) à monty

o p
C
outer()
Nonlocal and Global Variables

x = 100
global
m y
e
def outer():
x = 'python'

c a d
local (outer)

A
def inner1(): x
x

te
nonlocal x
local (inner1)

y
x = 'monty' x
def inner2():
global x
th B local (inner2)
x = 'hello'

M a
print('inner(before)', x) à monty
x

inner2()

t ©
h
print('inner(after)', x) à monty

inner1()
yr i g
o p
print('outer', x) à monty

C
outer()

print(x) à hello
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Free Variables and Closures

y
Remember: Functions defined inside another function can access the outer (nonlocal) variables

m
def outer():
d e
a
this x refers to the one in outer's scope
x = 'python'

c
this nonlocal variable x is called a free variable

A
te
def inner():
when we consider inner, we really are looking at:

y
print("{0} rocks!".format(x))
• the function inner
inner()
th B
• the free variable x (with current value python)

outer() à python rocks!

M a This is called a closure

t ©
i g h
p yr
C o
Returning the inner function

y
What happens if, instead of calling (running) inner from inside outer, we return it?

def outer():
e
x is a free variable in inner
m
x = 'python'
a d
it is bound to the variable x in outer

c
A
this happens when outer runs

te
def inner(): (i.e. when inner is created)
print("{0} rocks!".format(x))

B y
this the closure
inner()
return inner
a th
when we return inner, we are actually "returning" the closure

© M
h t
We can assign that return value to a variable name: fn = outer()

fn()
yr i
à python rocks!g
o p
C
When we called fn
at that time Python determined the value of x in the extended scope
But notice that outer had finished running before we called fn – it's scope was "gone"
Python Cells and Multi-Scoped Variables

def outer(): Here the value of x is shared between two scopes:

m y
e
x = 'python' • outer

d
def inner():

a
• closure
print(x)
return inner
Ac
The label x is in two different scopes but always reference the same "value"

y
Python does this by creating a cell as an intermediary object
te
th B indirect reference

a
outer.x cell 0xA500 str 0xFF100

© M
0xFF100 python

inner.x

h t
yr i g
p
In effect, both variables x (in outer and inner), point to the same cell

C o
When requesting the value of the variable, Python will "double-hop" to get to the final value
Closures

y
You can think of the closure as a function plus an extended scope that contains the free variables

e m
The free variable's value is the object the cell points to – so that could change over time!

a d
Every time the function in the closure is called and the free variable is referenced:

c
A
Python looks up the cell object, and then whatever the cell is pointing to

te
def outer():

B y
h
a = 100 closure
x = 'python'
a t cell 0xA500 str 0xFF100

def inner():
© M 0xFF100 python

h
a = 10 # local variable
t indirect reference

i g
print("{0} rocks!".format(x))

yr
o
fn = outer() p
return inner

C
fn à inner + extended scope x
Introspection
def outer(): cell 0xA500 str 0xFF100

y
a = 100
0xFF100 python

m
x = 'python'

e
def inner():

d
a = 10 # local variable
indirect reference

a
print("{0} rocks!".format(x))

c
return inner

fn = outer()

te A
fn.__code__.co_freevars à ('x',)
y
(a is not a free variable)

B
fn.__closure__

a th
à (<cell at 0xA500: str object at 0xFF100>, )

© M
t
def outer():
x = 'python'
print(hex(id(x))
i g h 0xFF100 indirect reference
def inner():

p yr
print(hex(id(x)) 0xFF100 indirect reference

C o
print("{0} rocks!".format(x))
return inner

fn = outer()
fn()
Modifying free variables

def counter(): closure

m y
e
count = 0 count is a free variable

def inc():
it is bound to the cell count

c a d
nonlocal count
count += 1
te A
return count

B y
return inc fn
a th
à inc + count à 0

fn = counter()
© M
h t
fn() à 1

yr i g
count's (indirect) reference changed from the object 0 to the object 1

o p
fn()
C à 2
Multiple Instances of Closures

Every time we run a function, a new scope is created.

m y
If that function generates a closure, a new closure is created every time as well

d e
c a
def counter(): closure f1 = counter()
f2 = counter()
te A
y
count = 0

def inc():
f1()

th
à 1
B
nonlocal count
count += 1
M
f1()
f1() a à 2
à 3
f1 and f2 do not have
the same extended

©
return count scope

h t f2() à 1 they are different instances of the

g
return inc

yr i closure

o p the cells are different

C
Shared Extended Scopes

y
def outer():

count = 0
e m
count is a free variable – bound to count in the extended scope

def inc1():
c a d
nonlocal count
count += 1
te A
count is a free variable – bound to the same count
return count

B y
def inc2():
a th
M
nonlocal count
count += 1

©
returns a tuple containing both closures

t
return count

i g h
yr
return inc1, inc2

o p
f1, f2 = outer()

C
f1() à 1
f2() à 2
Shared Extended Scopes
You may think this shared extended scope is highly unusual… but it's not!

m y
e
def adder(n):
def inner(x):
return x + n
c a d
return inner
te A
B y
add_1 = adder(1)
add_2 = adder(2)
a th
Three different closures – no shared scopes
add_3 = adder(3)

© M
add_1(10) à 11
h t
add_2(10)

yr
à 12
i g
p
add_3(10) à 13

C o
Shared Extended Scopes

y
But suppose we tried doing it this way:
adders = []

e m
d
for n in range(1, 4):
adders.append(lambda x: x + n)

c a
A
n = 1: the free variable in the lambda is n, and it is bound to the n we created in the loop

te
y
n = 2: the free variable in the lambda is n, and it is bound to the (same) n we created in the loop

h B
n = 3: the free variable in the lambda is n, and it is bound to the (same) n we created in the loop

t
M a
Now we could call the adders in the following way:

adders[0](10) à 13

t ©
adders[1](10) à 13

i g h
yr
adders[2](10) à 13

o p
Remember, Python does not "evaluate" the free variable n until the adders[i] function is called

C
Since all three functions in adders are bound to the same n
by the time we call adders[0], the value of n is 3
(the last iteration of the loop set n to 3)
Nested Closures
def incrementer(n):
# inner + n is a closure
def inner(start):
m y
current = start

d e
a
# inc + current + n is a closure
def inc():
nonlocal current
Ac
current += n

y te
B
return current

return inc
a th
M
return inner

t ©
(inner)

i g h
yr
fn = incrementer(2) à fn.__code__.co_freevars à 'n' n=2
(inc)

o p
inc_2 = fn(100) à inc_2.__code__.co_freevars à 'current', 'n'

C
(calls inc)
inc_2() à 102 (current = 102, n=2)
current=100, n=2

inc_2() à 104 (current = 104, n=2)

 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Decorators Recall the simple closure example we did which allowed to us to maintain a
count of how many times a function was called:
def counter(fn):

m y
using *args, **kwargs means we can call

e
count = 0 any function fn with any combination of
def inner(*args, **kwargs):
nonlocal count
a d
positional and keyword-only arguments

c
A
count += 1

te
print('Function {0} was called {1} times'.format(fn.__name__, count)
return fn(*args, **kwargs)
return inner
B y
def add(a, b=0):
a th
return a + b

© M
t
add = counter(add)

g h
result = add(1, 2) à Function add was called 1 times

i
yr
à result = 3

p
We essentially modified our add function by wrapping it inside another

o
C
function that added some functionality to it
We also say that we decorated our function add with the function counter
And we call counter a decorator function
Decorators

In general a decorator function:

m y
e
• takes a function as an argument
•
•
returns a closure
the closure usually accepts any combination of parameters
c a d
•
•
runs some code in the inner function (closure)

te A
the closure function calls the original function using the arguments passed to the closure
• returns whatever is returned by that function call

B y
a th closure

M
outer function (fn)

t ©
i g h
inner function

yr
(*args, **kwargs)

o p does something…

C returns fn(*args, **kwargs)

Decorators and the @ Symbol

y
In our previous example, we saw that counter was a decorator
and we could decorate our add function using: add = counter(add)

e m
c a d
In general, if func is a decorator function, we decorate another function my_func using:

my_func = func(my_func)
te A
B y
th
This is so common that Python provides a convenient way of writing that:

a
@counter
def add(a, b):
@func

© M
def my_func(…):
return a + b
h t …

yr i
is the same as writing g is the same as writing

o p
def add(a, b): def my_func(…):

C
return a + b

add = counter(add)
…

my_func = func(my_func)
 def counter(fn):
Introspecting Decorated Functions
count = 0

y
Let's use the same count decorator. def inner(*args, **kwargs):
nonlocal count
count += 1

e m
d
print('{0} was called {1} times'.format(fn.__name__, count)

@counter return inner

c a
return fn(*args, **kwargs)

def mult(a, b, c=1):

"""
te A
returns the product of three values

B y remember we could equally have written:

h
"""

t
mult = counter(mult)

a
return a * b * c

© M
mult.__name__
t
à inner

h
not mult mult's name "changed" when we decorated it

help(mult)
yr i g they are not the same function after all

p
à Help on function inner in module __main__:

C o inner(*args, **kwargs)
We have also "lost" our docstring,
and even the original function signature
Even using the inspect module's signature does not yield better results
One approach to fixing this

m
We could try to fix this problem, at least for the docstring and function name as follows:y
d e
def counter(fn):
c a
count = 0
def inner(*args, **kwargs):
te A
nonlocal count

B y
h
count += 1

return fn(*args, **kwargs)

a t
print('Function {0} was called {1} times'.format(fn.__name__, count)

inner.__name__ = fn.__name__
inner.__doc__ = fn.__doc__
© M
return inner
h t
yr i g
p
But this doesn’t fix losing the function signature – doing so would be quite complicated

o
C
Instead, Python provides us with a special function that we can use to fix this
 The functools.wraps function

m
The functools module has a wraps function that we can use to fix the metadata of our inner
y
function in our decorator

d e
from functools import wraps

c a
In fact, the wraps function is itself a decorator

te A
y
but it needs to know what was our "original" function – in this case fn

B
def counter(fn):
a th
def counter(fn):
count = 0
def inner(*args, **kwargs):
© M count = 0
@wraps(fn)
nonlocal count
h t def inner(*args, **kwargs):
count += 1
print(count)
yr i g nonlocal count
count += 1

o p
return fn(*args, **kwargs)
inner = wraps(fn)(inner)
print(count)
return fn(*args, **kwargs)

C
return inner return inner
def counter(fn): @counter
count = 0 def mult(a:int, b:int, c:int=1):
@wraps(fn)
def inner(*args, **kwargs):
"""

m y
returns the product of three values
nonlocal count """
d e
count += 1
a
return a * b * c

c
A
print(count)

te
return fn(*args, **kwargs)

y
return inner

th B
help(mult)
a
à Help on function mult in module __main__:

M
mult(a:int, b:int, c:int=1)

©
returns the product of three values

t
i g h
yr
And introspection using the inspect module works as expected:

o p
inspect.signature(mult) à <Signature (a:int, b:int, c:int=1)>

C
You don't have to use @wraps, but it will make debugging easier!
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Decorator Parameters

m
In the previous videos we saw some built-in decorators that can handle some arguments:
y
d e
@wraps(fn) @lru_cache(maxsize=256)
c a
def inner():
…
def factorial(n):
…
te A function call

B y
th
This should look quite different from the decorators we have been creating and using:

a
@timed
def fibonacci(n): M
no function call

©
…

h t
yr i g
o p
C
The timed decorator

def timed(fn):
from time import perf_counter

m y
def inner(*args, **kwargs):
hardcoded value 10

d e
total_elapsed = 0
c a
A
for i in range(10):

te
start = perf_counter()
result = fn(*args, **kwargs)

B y
total_elapsed += (perf_counter() - start)
avg_elapsed = total_elapsed / 10
print(avg_elapsed)
a th
return result
return inner
© M
h t
@timed
yr i g
o p
def my_func(): OR my_func = timed(my_func)

C
…
One Approach extra parameter

def timed(fn, reps):

from time import perf_counter
m y
e
free variable

def inner(*args, **kwargs):

c a d
A
total_elapsed = 0

te
for i in range(reps):

y
start = perf_counter()

B
result = fn(*args, **kwargs)

th
total_elapsed += (perf_counter() - start)

a
avg_elapsed = total_elapsed / reps
print(avg_elapsed)
return result
© M
return inner

h t
yr i g
o p
my_func = timed(my_func, 10) @timed(10)

C
def my_func():
…
Rethinking the solution
@timed
def my_func(): my_func = timed(my_func)
m y
…

d e
c a
So, timed is a function that returns that inner closure that contains our original function

te A
In order for this to work as intended:
B y
@timed(10)
a th
M
def my_func():
…

t ©
i g h
timed(10) will need to return our original timed decorator when called

p
dec = timed(10)
yr timed(10) returns a decorator
@dec
C o
def my_func():
…
and we decorate our function with dec
 Nested closures to the rescue!

def outer(reps):
def timed(fn):
m y
from time import perf_counter

d e
c a
our original decorator

def inner(*args, **kwargs):

A
total_elapsed = 0 free variable bound to reps in outer

te
for i in range(reps):
start = perf_counter()

B y
h
result = fn(*args, **kwargs)

a t
total_elapsed += (perf_counter() - start)

M
avg_elapsed = total_elapsed / reps
print(avg_elapsed)

t
return result
© calling outer(n) returns our original decorator

h
return inner with reps set to n (free variable)

return timed
yr i g
o p
C
my_func = outer(10)(my_func) OR @outer(10)
def my_func():
…
Decorator Factories

The outer function is not itself a decorator

m y
d e
instead it returns a decorator when called

c a
te A
and any arguments passed to outer can be referenced (as free variables) inside our decorator

B y
a th
We call this outer function a decorator factory function

M
(it is a function that creates a new decorator each time it is called)

©
h t
yr i g
o p
C
And finally…

To wrap things up, we probably don't want out decorator call to look like:
m y
d e
@outer(10)

c a
A
def my_func():

te
…

B y
It would make more sense to write it this way:

a th
@timed(10)
def my_func():
© M
…
h t
yr i g
o p
All we need to do is change the names of the outer and timed functions

C
def timed(reps): this was outer

def dec(fn): this was timed

m y
e
from time import perf_counter

c a d
A
@wraps(fn) we can still use @wraps

te
def inner(*args, **kwargs):
total_elapsed = 0
for i in range(reps):
B y
start = perf_counter()

a th
result = fn(*args, **kwargs)

© M
total_elapsed += (perf_counter() - start)
avg_elapsed = total_elapsed / reps

h t
print(avg_elapsed)

i g
return result

yr
return inner

p
return dec

o
C
@timed(10)
def my_func():
…
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Tuples are…

read-only lists… at least that's how many introductions to Python will present tuples!!
m y
d e
This isn't wrong, but there's a lot more going on with tuples…
c a
te A
y
If you only think of tuples as read-only lists, you're going to miss out on some interesting ideas

th B
a
We really need to think of tuples also as data records position of value has meaning

© M
This is why we are going to start looking at tuples before we even cover sequence types

h t
yr i g
We are going to focus here on tuples as a data records or structures

o p
C
We will also look at named tuples
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Tuples vs Lists vs Strings

Tuples Lists Strings

m y
containers containers

d e
containers

order matters order matters

c a order matters

Heterogeneous / Homogeneous

te
Heterogeneous / Homogeneous
A Homogeneous

indexable indexable
B y indexable

iterable iterable
a th iterable

immutable

© M mutable immutable
fixed length
h t length can change fixed length

fixed order

yr i g order of elements can change fixed order

o p
cannot do in-place sorts can do in-place sorts

C cannot do in-place reversals can do in-place reversals

Immutability of Tuples

elements cannot be added or removed

m y
the order of elements cannot be changed
d e
c a
works well for representing data structures:

te A
Point: (10, 20)
B y
1st element is the x-coordinate

a th
2nd element is the y-coordinate

Circle: (0, 0, 10)

© M 1st element is the x-coordinate of the center

h t 2nd element is the y-coordinate of the center

g
3rd element is the radius

yr i
p
City: ('London', 'UK', 8_780_000) 1st element is the name of a city

C o 2nd element is the country

3rd element is the population

The position of the data has meaning

Tuples as Data Records

Think of a tuple as a data record where the position of the data has meaning
m y
d e
london = ('London', 'UK', 8_780_000)
c a
new_york = ('New York', 'USA', 8_500_000)

te A
beijing = ('Beijing', 'China', 21_000_000)

B y
a th
Because tuples, strings and integers are immutable, we are guaranteed

M
that the data and data structure for london will never change

©
h t
We can have a list of these tuples:

yr i g
p
cities = [('London', 'UK', 8_780_000),

C o
('New York', 'USA', 8_500_000),
('Beijing', 'China', 21_000_000)]
Extracting data from Tuples

Since tuples are sequences just like strings and lists, we can retrieve items by index

m y
london = ('London', 'UK', 8_780_000)
d e
c a
city = london[0] country = london[1]
A
population = london[2]

te
B y
cities = [('London', 'UK', 8_780_000),
('New York', 'USA', 8_500_000),
a th
M
('Beijing', 'China', 21_000_000)]

©
h t
yr
total_population = 0
for city in cities: i g
o p
total_population += city[2]

C
You'll notice how the list of cities is homogeneous (contains cities only)
But a city (the tuple) is heterogeneous
Extracting data from Tuples

We can also use tuple unpacking

m y
d e
c a
We actually already know how to do this – we covered this in the section on function arguments

new_york = ('New York', 'USA', 8_500_000)

te A
packed three values into a tuple
B y
a th
M
city, country, population = new_york

unpacked tuple
t ©
i g h
p yr
o
city, country, population = ('New York', 'USA', 8_500_000)

C
city, country, population = 'New York', 'USA', 8_500_000
Dummy Variables

m y
This is something you’re likely to run across when you look at Python code that uses tuple unpacking

d e
a
Sometimes, we are only interested in a subset of the data fields in a tuple, not all of them

c
te A
y
Suppose we are interested only in the city name and the population:

B
th
city, _, population = ('Beijing', 'China', 21_000_000)

a
© M
_ is actually a legal variable name – so there's nothing special about it

h t
yr i g
but by convention, we use the underscore to indicate this is a variable we don't care about

o p
C
in fact, we could just have used:
city, ignored, population = ('Beijing', 'China', 21_000_000)
Dummy Variables

It's also used in extended unpacking too

m y
record = ('DJIA', 2018, 1, 19, 25987.35, 26071.72, 25942.83, 26071.72)
d e
symbol, year, month, day, open, high, low, close = record
c a
te A
Let's say we are only interested in the symbol, year, month, day and close fields

B y
We could do it this way: symbol = record[0]
year = record[1]
a th looks really bad!

© M
month = record[2]
day = record[3]

h t
close = record[7]

yr i g
symbol, year, close = record[0], record[1], record[7] awful!

o p
C
symbol, year, month, day, *_, close = record

symbol, year, month, day, *ignored, close = record

 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Tuple as Data Structure

We have seen how we interpreted tuples as data structures

m y
The position of the object contained in the tuple gave it meaning
d e
c a
A
For example, we can represent a 2D coordinate as: (10, 20)

x
y y
te
th
If pt is a position tuple, we can retrieve the x and B
x, y = pt or x = pt[0]
y coordinates using:

M a y = pt[1]

©
So, for example, to calculate the distance of pt from the origin we could write:

t
i g h
dist = math.sqrt(pt[0] ** 2 + pt[1] ** 2)

p yr
C o
Now this is not very readable, and if someone sees this code they will have to know that pt[0]
means the x-coordinate and pt[1] means the y-coordinate.

This is not very transparent.

 Using a Class Instead

At this point, in order to make things clearer for the reader (not the compiler, the reader), we
might want to approach this using a class instead.

m y
class Point2D:
d e
def __init__(self, x, y):
self.x = x
c a
pt = Point2D(10, 20)

self.y = y

te A
distance = sqrt(pt.x ** 2 + pt.y ** 2)

class Stock:
B y
th
def __init__(self, symbol, year, month, day, open, high, low, close):
self.symbol = symbol
a
self.year = year
self.month = month
© M
self.day = day

h t
Class Approach Tuple Approach

g
self.open = open

yr
self.high = high
i djia.symbol djia[0]

p
self.low = low djia.open djia[4]

C o
self.close = close djia.close djia[7]

djia.high – djia.low djia[5] – djia[6]

Extra Stuff

At the very least we should implement the __repr__ method

m y
e
à Point(x=10, y=20)

We probably should implement the __eq__ method too

c a d
à Point(10, 20) == Point(10, 20) à True

te A
B y
class Point2D:
def __init__(self, x, y):
a th
self.x = x
self.y = y
© M
h t
g
def __repr__(self):

yr i
return f'Point2D(x={self.x}, y={self.y}'

o p
def __eq__(self, other):

C
if isinstance(other, Point2D):
return self.x == other.x and self.y == other.y
else:
return False
Named Tuples to the rescue

There are other reasons to seek another approach. I cover some of those in the coding video

m y
Amongst other things, Point2D objects are mutable – something we may not want!
d e
c a
te A
There's a lot to like using tuples to represent simple data structures
The real drawback is that we have to know what the positions mean, and remember this in our code

B y
th
If we ever need to change the structure of our tuple in our code (like inserting a value that we forgot)

a
M
most likely our code will break!

eric = ('Idle', 42)

t ©
eric = ('Eric', 'Idle', 42)

i g h Broken!!

yr
last_name, age = eric last_name, age = eric

o p
C
Class approach: last_name = eric.last_name
age = eric.age
Named Tuples to the rescue

So what if we could somehow combine these two approaches, essentially creating tuples
where we can, in addition, give meaningful names to the positions?
m y
d e
That's what namedtuples essentially do

c a
te A
They subclass tuple, and add a layer to assign property names to the positional elements

B y
Located in the collections standard library module

a th
from collections import namedtuple

© M
h t
yr i g
namedtuple is a function which generates a new class à class factory

o p
that new class inherits from tuple

C but also provides named properties to access elements of the tuple

but an instance of that class is still a tuple

Generating Named Tuple Classes

We have to understand that namedtuple is a class factory

m y
d e
When we use it, we are essentially creating a new class, just as if we had used class ourselves

c a
A
namedtuple needs a few things to generate this class:

• the class name we want to use

y te
h B
• a sequence of field names (strings) we want to assign, in the order of the elements in the tuple

t
M a
field names can be any valid variable name
except that they cannot start with an underscore

t ©
h
The return value of the call to namedtuple will be a class

yr i g
We need to assign that class to a variable name in our code so we can use it to construct instances

o p
C
In general, we use the same name as the name of the class that was generated

Point2D = namedtuple('Point2D', ['x', 'y'])

Generating Named Tuple Classes

Point2D = namedtuple('Point2D', ['x', 'y'])

m y
d e
c a
We can create instances of Point2D just as we would with any class (since it is a class)

pt = Point2D(10, 20)
te A
B y
th
The variable name that we use to assign to the class generated and returned by namedtuple is arbitrary

a
© M
Pt2D = namedtuple('Point2D', ['x', 'y'])
pt = Pt2D(10, 20)
h t
yr i g
o p
C
 0xFF300
Generating Named Tuple Classes Class:
Variable: MyClass

y
MyClass

class MyClass:

e m
d
pass
Variable: MyClassAlias

c a
MyClassAlias = MyClass

te A
instance_1 = MyClass()
y
instance_2 = MyClassAlias()

B
instantiates the same class

a th
M
Similarly

©
Pt2DAlias = namedtuple('Point2D', ['x', 'y'])

t
i g h
yr
0xFF900
Class:
Variable: Pt2DAlias

o p Point2D

C
This is the same concept as aliasing a function, or assigning a lambda
function to a variable name!
Generating Named Tuple Classes

There are many ways we can provide the list of field names to the namedtuple function

m y
• a list of string
d e
• a tuple of strings
c a
à in fact any sequence, just remember that order matters!

A
• a single string with the field names separated by whitespace or commas

te
B y
namedtuple('Point2D', ['x', 'y'])
namedtuple('Point2D', ('x', 'y'))
a th
© M
t
namedtuple('Point2D', 'x, y')

i g h
namedtuple('Point2D', 'x y')

p yr
C o
Instantiating Named Tuples

m y
After we have created a named tuple class, we can instantiate them just like an ordinary class

d e
c a
In fact, the __new__ method of the generated class uses the field names we provided as param names

te A
y
Point2D = namedtuple('Point2D', 'x y')

th B
We can use positional arguments:

pt1 = Point2D(10, 20)

M
10 à x
a 20 à y

t ©
i g h
yr
And even keyword arguments:

o p
pt2 = Point2D(x=10, y=20) 10 à x 20 à y

C
Accessing Data in a Named Tuple

m y
Since named tuples are also regular tuples, we can still handle them just like any other tuple

• by index
d e
• slice
c a
• iterate

te A
Point2D = namedtuple('Point2D', 'x y')
B y
a th
M
pt1 = Point2D(10, 20) isinstance(pt1, tuple) à True

t ©
g
x, y = pt1

i h
p yr
x = pt1[0]

C o for e in pt1:
print(e)
Accessing Data in a Named Tuple

m y
e
But now, in addition, we can also access the data using the field names:

c a d
A
Point2D = namedtuple('Point2D', 'x y')

te
pt1 = Point2D(10, 20)

B y
pt1.x à 10

a th
pt1.y à 20

© M
h t
Since namedtuple generated classes inherit from tuple class Point2D(tuple):

yr i g
pt1 is a tuple, and is therefore immutable
…

o p
C
pt1.x = 100 will not work!
The rename keyword-only argument for namedtuple

Remember that field names for named tuples must be valid identifiers, but cannot start

m y
e
with an underscore

This would not work:

c a
Person = namedtuple('Person', 'name age _ssn') d
te A
namedtuple has a keyword-only argument, rename
B y
(defaults to False)

th
that will automatically rename any invalid field name

a
uses convention:

© M
_{position in list of field names}

This will now work:

h t
yr i g
Person = namedtuple('Person', 'name age _ssn', rename=True)

o p
And the actual field names would be:

C name age _2
Introspection

We can easily find out the field names in a named tuple generated class
m y
d e
class property à _fields
c a
te A
y
Person = namedtuple('Person', 'name age _ssn', rename=True)

Person._fields
t
à ('name', 'age', '_2')
h B
M a
t ©
i g h
p yr
C o
Introspection

Remember that namedtuple is a class factory, i.e. it generates a class

We can actually see what the code for that class is, using the class property _source
m y
d e
Point2D = namedtuple('Point2D', 'x y')

c a
Point2D._source à

te A lots of code omitted

class Point2D(tuple):
'Point2D(x, y)'
B y
a th
def __new__(_cls, x, y):

© M
'Create new instance of Point2D(x, y)'

h t
return _tuple.__new__(_cls, (x, y))

yr i g
def __repr__(self):

o p
'Return a nicely formatted representation string'
return self.__class__.__name__ + '(x=%r, y=%r)' % self

C
x = _property(_itemgetter(0), doc='Alias for field number 0')

y = _property(_itemgetter(1), doc='Alias for field number 1')

Extracting Named Tuple Values to a Dictionary

Instance method: _asdict()

m y
d e
that creates a dictionary of all the named values in the tuple
c a
te A
y
Point2D = namedtuple('Point2D', 'x y')
pt1 = Point2D(10, 20)

th B
pt1._asdict()
M
à {'x': 10, 'y': 20} a
t ©
i g h
p yr
C o
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Named Tuples are Immutable

So how can we "change" one or more values inside the tuple?

m y
d e
c a
Just like with strings, we have to create a new tuple, with the modified values

te A
y
Point2D = namedtuple('Point2D', 'x y')

pt = Point2D(0, 0)

th B
M a
©
Suppose we need to change the value of the x coordinate:

h t
Simple approach:

yr i g
pt = Point2D(100, pt.y)

o p
C
Note that the memory address of pt has now changed
Drawback

This simple approach can work well, but it has a major drawback

m y
d e
a
Stock = namedtuple('Stock', 'symbol year month day open high low close')

Ac
djia = Stock('DJIA', 2018, 1, 25, 26_313, 26_458, 26_260, 26_393)

y te
Suppose we only want to change the close field

th B
djia = Stock(djia.symbol,
M a
djia.year,
djia.month,
t ©
djia.day,

i g h
yr
djia.open, painful!

p
djia.high,

C o djia.low,
26_394)
Maybe slicing or unpacking?

djia = Stock('DJIA', 2018, 1, 25, 26_313, 26_458, 26_260, 26_393)

m y
d e
current = djia[:7]
c a
current à ('DJIA', 2018, 1, 25, 26_313, 26_458, 26_260)

te A
B y
*current, _ = djia current à ['DJIA', 2018, 1, 25, 26_313, 26_458, 26_260]

djia = Stock(*current, 26_394)

a th
© M
We can also use the _make class method – but we need to create an iterable that contains all
the values first:
h t
yr i g
new_values = current + (26_394,) new_values = current.append(26_394)

o p
new_values à 'DJIA', 2018, 1, 25, 26_313, 26_458, 26_260, 26_394

C iterable

djia = Stock._make(new_values)
This still has drawbacks

djia = Stock('DJIA', 2018, 1, 25, 26_313, 26_458, 26_260, 26_393)

m y
What if we wanted to change a value in the middle, say day?
d e
c a
A
Cannot use extended unpacking (only one starred value in extending unpacking)

te
*pre, day, *post = djia

B y
makes no sense…

a th
M
Slicing will work: pre = djia[:3]
post = djia[4:]

t ©
i g h
yr
new_values = pre + (26,) + post

o p
new_values à ('DJIA', 2018, 1, 26, 26_313, 26_458, 26_260, 26_394)

C
djia = Stock(*new_values)
But even this still has drawbacks! 3 5

djia = Stock('DJIA', 2018, 1, 25, 26_313, 26_458, 26_260, 26_393)

m y
d e
How about modifying both the day and the high values?

c a
te A
y
new_values = djia[:3] + (26,) + djia[4:5] + (26_459,) + djia[6:]

B
djia = Stock(*new_values)
a th
© M
h t
This is just unreadable and extremely error prone!

yr i g
p
There has to be a better way!

o
C
The _replace instance method

Named tuples have a very handy instance method, _replace

m y
d e
a
It will copy the named tuple into a new one, replacing any values from keyword arguments

c
te A
The keyword arguments are simple the field names in the tuple and the new value

The keyword name must match an existing field name

B y
a th
M
Stock = namedtuple('Stock', 'symbol year month day open high low close')

©
t
djia = Stock('DJIA', 2018, 1, 25, 26_313, 26_458, 26_260, 26_393)

i g h
yr
djia = djia._replace(day=26, high=26_459, close=26_394)

o p
C
djia à 'DJIA', 2018, 1, 26, 26_313, 26_459, 26_260, 26_394

Note that the memory address of djia has now changed

Extending a Named Tuple

m y
e
Sometimes we want to create named tuple that extends another named tuple, appending one
or more fields

c a d
A
Stock = namedtuple('Stock', 'symbol year month day open high low close')

te
B y
th
We want to create a new named tuple class, StockExt that adds a single field, previous_close

a
© M
When dealing with classes, this is sometimes done by using subclassing.

h t
yr i g
But this not easy to do with named tuples

o p
C and there's a cleaner way of doing it anyway
Extending a Named Tuple

Point2D = namedtuple('Point2D', 'x y')

m y
d e
c a
Let's say we want to create a Point3D named tuple that has an extra parameter

Yes, the obvious, and simplest approach here is best:

te A
B y
h
Point3D = namedtuple('Point3D', 'x y z')

a t
M
But what happens if you have a lot of fields in the named tuple? Code is not as clean anymore…

©
h t
Stock = namedtuple('Stock', 'symbol year month day open high low close')

yr i g
StockExt = namedtuple('Stock', 'symbol year month day open high low close previous_close')

o p
C
How about re-using the existing field names in Stock?
Extending a Named Tuple

Stock = namedtuple('Stock', 'symbol year month day open high low close')
m y
d e
c a
A
Stock._fields à 'symbol', 'year', 'month', 'day', 'open', 'high', 'low', 'close'

y te
h B
We can then create a new named tuple by "extending" the _fields tuple

t
M a
new_fields = Stock._fields + ('previous_close', )

t ©
h
StockExt = namedtuple('StockExt', new_fields)

yr i g
o p
C
Extending a Named Tuple

m
We can also easily use an existing Stock instance to create a new StockExt instance y
with the same common values, adding in our new previous_close value:

d e
c a
Stock = namedtuple('Stock', 'symbol year month day open high low close')

te A
y
StockExt = namedtuple('StockExt', Stock._fields + ('previous_close', )

th B
djia = Stock('DJIA', 2018, 1, 25, 26_313, 26_458, 26_260, 26_393)

M a
©
djia_ext = StockExt(*djia, 26_000)

h t
or

yr i g
o p
djia_ext = StockExt._make(djia + (26_000, ))

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Default Docs for Named Tuples

When we create a named tuple class, default docstrings are created

m y
d e
Point2D = namedtuple('Point2D', 'x y')

c a
Point2D.__doc__ à Point2D(x, y)

te A
y
Point2D.x.__doc__ à Alias for field number 0
Point2D.y.__doc__
h B
à Alias for field number 1

t
M a
©
help(Point2D) à class Point2D(builtins.tuple)

h t
Point2D(x, y)

yr i g x

p
Alias for field number 0

C o y
Alias for field number 1
Overriding DocStrings

m
We can override the docstrings simply by specifying values for the __doc__ properties y
d e
a
(this is not unique to named tuples!)

Ac
te
Point2D.__doc__ = 'Represents a 2D Cartesian coordinate.'

Point2D.x.__doc__ = 'x coordinate'

B y
Point2D.y.__doc__ = 'y coordinate'
a th
© M
help(Point2D) à
h t
class Point2D(builtins.tuple)

yr i g Represents a 2D Cartesian coordinate.

o p x
x coordinate

C y
y coordinate
Default Values

y
The namedtuple function does not provide us a way to define default values for each field

Two approaches to this:

e m
Using a Prototype
c a d
te A
Create an instance of the named tuple with default values - the prototype

y
Create any additional instances of the named tuple using the prototype._replace method

B
th
You will need to supply a default for every field (can be None)

a
Using the __defaults__ property

© M
h t
Directly set the defaults of the named tuple constructor (the __new__ method)

yr i g
You do not need to specify a default for every field

o p
Remember that you cannot have non-defaulted parameters

C
after the first defaulted parameter

def func(a, b=10, c=20) def func(a, b=10, c)

Using a Prototype

Vector2D = namedtuple('Vector2D', 'x1 y1 x2 y2 origin_x origin_y')

m y
d e
a
vector_zero = Vector2D(x1=0, y1=0, x2=0, y2=0, origin_x=0, origin_y=0)

or
Ac
te
vector_zero = Vector2D(0, 0, 0, 0, 0, 0)

B y
h
vector_zero à Vector2D(x1=0, y1=0, x2=0, y2=0, origin_x=0, origin_y=0)

a t
© M
t
To construct a new instance of Vector2D we now use vector_zero._replace instead:

i g h
yr
v1 = vector_zero._replace(x1=10, y1=10, x2=20, y2=20)

o p
v1 à Vector2D(x1=10, y1=10, x2=20, y2=20, origin_x=0, origin_y=0)

C
Using __defaults__

def func(a, b=10, c=20):

m y
pass

d e
c a
func.__defaults__ à (10, 20)
te A
B y
a b c

a th
M
10 20

t ©
h
no default

yr i g
The __defaults__ property is writable

o p
C
So we can set it to a tuple of our choice

Just don't provide more defaults than parameters! (extras are ignored)
Using __defaults__

We need to provide defaults to the constructor of our named tuple class

m
__new__
y
d e
c
Vector2D = namedtuple('Vector2D', 'x1 y1 x2 y2 origin_x origin_y')
a
Vector2D.__new__.__defaults__ = (0, 0)
te A
x1 y1 x2 y2 origin_x origin_y

B y 0 0

a th
v1 = Vector2D(10, 10, 20, 20)

© M
t
v1 à Vector2D(x1=10, y1=10, x2=20, y2=20, origin_x=0, origin_y=0)

h
yr i g
o p
Isn't this cleaner than the prototype approach?!!

C
v1 = vector_zero._replace(x1=10, y1=10, x2=20, y2=20)
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Modules
What are modules exactly? à objects of type ModuleType
m y
d e
How does Python load modules?
c a
te A
y
How to import without the import statement

B
a th
M
Reloading modules à why we should not do it!

t ©
i g h
Import variants à import

yr
from … import …
from … import *

o p
Misconceptions

C
__main__

m y
in modules

d e
c a
A
as file names

y te
th B
Zip Archives
M a
t ©
importing from a zip archive

i g h
yr
zipping an entire Python app

p
C o
creating an executable Python app in bash
Packages
What is a package?
m y
d e
Why use them?
c a
How is it different from a module?
te A
B y
h
The role of __init__.py files in packages

a t
Implicit Namespace Packages
© M
t
(Python 3.3+)

i g h
yr
What are they?

o p
How do we create and use them?

C
vs standard packages
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Import variants

# module1.py

m y
import math
d e
sys.modules
c a
is math in sys.modules? math

te
<module object>
A
if not, load it and insert ref

B y
a th
M
add symbol math to module1's global namespace referencing the same object

©
h t module1.globals()

yr i g math <module object>

o p math symbol in namespace

C
(if math symbol already exists in module1's namespace, replace reference)
Import variants

# module1.py

m y
import math as r_math sys.modules

d e
math <module object>

c a
is math in sys.modules?

te A
if not, load it and insert ref

B y
a th
M
add symbol r_math to module1's global namespace referencing the same object

©
h t module1.globals()

yr i g r_math <module object>

o p r_math symbol in namespace

C
math symbol not in namespace

(if r_math symbol already exists in module1's namespace, replace reference)

Import variants

# module1.py

m y
from math import sqrt
d e
sys.modules
c a
is math in sys.modules? math

te
<module object>
A
if not, load it and insert ref

B y
a th
M
add symbol sqrt to module1's global namespace referencing math.sqrt

©
h t
yr i g module1.globals()

o p sqrt <math.sqrt object>

C math symbol not in namespace

(if sqrt symbol already exists in module1's namespace, replace reference)

Import variants

# module1.py

m y
from math import sqrt as r_sqrt
d e
sys.modules
c a
is math in sys.modules? math

te
<module object>
A
if not, load it and insert ref

B y
a th
M
add symbol r_sqrt to module1's global namespace referencing math.sqrt

©
h t
yr i g module1.globals()

o p r_sqrt <math.sqrt object>

C math symbol not in namespace

(if r_sqrt symbol already exists in module1's namespace, replace reference)

Import variants

# module1.py

m y
from math import *
d e
sys.modules
c a
is math in sys.modules? math

te
<module object>
A
if not, load it and insert ref

B y
a th
© M what "all" means can be
defined by the module being

h t
add "all" symbols defined in math to module1's global namespace imported

yr i g module1.globals()

o p pi
sin
<math.pi object>
<math.sin object>

C and many more… math symbol not in namespace

(if any symbols already exists in module1's namespace, replace their reference)
Commonality

In every case the math module was loaded into memory and referenced in sys.modules
m y
d e
c a
Running from math import sqrt

te A
B y
did not "partially" load math

a th
M
it only affected what symbols were placed in module1's namespace!

©
h t
yr i g
o p
Things may be different with packages, but for simple modules this is the behavior

C
Why from <module> import * can lead to bugs

m y
d e
a
# module1.py module1.globals()
from cmath import * sqrt <cmath.sqrt>

Ac
te
…

B y
a th
from math import *
M
module1.globals()

©
h tsqrt <math.sqrt>

g
…

yr i
o p
C
Efficiency

What's more efficient?

m y
e
import math
or from math import sqrt

c a d
importing à same amount of work
te A
B y
a th
M
calling math.sqrt(2)

t ©
h
sqrt(2)

yr i g
o p This first needs to find the sqrt symbol in math's namespace

C dict lookup à super fast!

 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
What we've seen so far…
Modules can be imported using

m y
the import statement importlib.import_module

d e
When a module is imported:
c a
system cache is checked first sys.modules
te A
à if in cache, just returns cached reference
otherwise:
B y
module has to be located (found) somewhere
a th finders e.g. sys.meta_path

© M
module code has to be retrieved (loaded) loaders returned by finder à ModuleSpec

h t
i g
"empty" module typed object is created

yr
o p
a reference to the module is added to the system cache sys.modules

C
module is compiled

module is executed à sets up the module's namespace (module.__dict__ is module.globals())

Module Finders

sys.meta_path à _frozen_importlib.BuiltinImporter
m y
finds built-ins, such as math

_frozen_importlib.FrozenImporter
d e
finds frozen modules

c a file-based modules

A
_frozen_importlib_external.PathFinder

PathFinder
y te
th B
Finds file-based modules based on sys.path and package __path__

sys.path à
M
['/home/fmb/my-app',
a
t ©
'/usr/lib/python36.zip',

i g h
'/usr/lib/python3.6',

yr
'/usr/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/dist-packages',

o p '/usr/lib/python3/dist-packages']

C
collections.__path__ à ['/usr/lib/python3.6/collections']
Module Properties

built-in import math

m y
d e
type(math) à module
c a
math.__spec__ à ModuleSpec(name='math',
te A
y
loader=<class '_frozen_importlib.BuiltinImporter'>

B
h
origin='built-in')

a t
math.__name__ à math

© M
math.__package__
h
à ''
t
yr i g
p
__file__ is not an attribute of math (built-ins only)

C o
Module Properties

standard library import fractions

m y
d e
type(fractions) à module
c a
fractions.__spec__
te
à ModuleSpec(name='fractions', A
B y
loader=<_frozen_importlib_external.SourceFileLoader
object at 0x7fa9bf7ff6d8>,

th
origin='/usr/lib/python3.6/fractions.py')

a
fractions.__name__
M
à fractions

©
fractions.__package__
h t
à ''

fractions.__file__
yr i g à /usr/lib/python3.6/fractions.py

o p
C
Note that fractions.__file__ was found by PathFinder in one of the paths listed in sys.path
Module Properties

custom module import module1

m y
d e
type(module1) à module
c a
module1.__spec__ à ModuleSpec(name='module1',
te A
B y
loader=<_frozen_importlib_external.SourceFileLoader
object at 0x7fd9f4c4ae48>,

th
origin='/home/fmb/my-app/module1.py')

a
module1.__name__ à module1

© M
module1.__package__
h t
à ''

module1.__file__
yr i g à /home/fmb/my-app/module1.py

o p
C
Note that module1.__file__ was found by PathFinder in one of the paths listed in sys.path
Some Notes Python docs:
https://docs.python.org/3/tutorial/modules.html
Python modules may reside
m y
https://docs.python.org/3/reference/import.html

PEP 302
d e
in the built-ins

c a
in files on disk

te A
y
they can even be pre-compiled, frozen, or even inside zip archives

B
a th
anywhere else that can be accessed by a finder and a loader

M
custom finders/loaders à database, http, etc

©
h t
yr i g
For file based modules (PathFinder):

o p
They must exist in a path specified in sys.path

C or in a path specified by <package>.__path__

 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Packages are Modules

m y
e
Packages are modules (but modules are not necessarily packages)

c a d
A
They can contain

modules
y te
packages
th B
(called sub-packages)

M a
t ©
h
If a module is a package, it must have a value set for __path__

yr i g
p
After you have imported a module, you can easily see if that module is a package by

o
C
inspecting the __path__ attribute (empty à module, non-empty à package)
Packages and File Systems

m y
e
Remember that modules do not have to be entities in a file system (loaders, finders)

c a d
A
By the same token, packages do not have to be entities in the file system

y te
B
Typically they are - just as typically modules are file system entities

a th
M
But packages represent a hierarchy of modules / packages

©
h t
i g
pack1.mod1

p yr
pack1.pack1_1.mod1_1

C o
dotted notation indicates the path hierarchy of modules / packages

and is usually found in __path__

Importing Nested Packages

If you have a statement in your top-level program such as:

m y
import pack1.pack1_1.module1
d e
c a
The import system will perform these steps:

te A
imports pack1
B y
imports pack1.pack1_1

a th
imports pack1.pack1_1.module1

© M
h t
g
The sys.modules cache will contain entries for: pack1

yr i pack1.pack1_1

o p pack1.pack1_1.module1

C
The namespace where the import was run contains: pack1
File System Based Packages

Although modules and packages can be far more generic than file system based entities,

m y
e
it gets complicated!

c a d
te A
If you're interested in this, then the first document you should read is PEP302

B y
a th
M
In this course we're going to stick to traditional file based modules and packages

©
h t
yr i g
o p
C
File Based Packages

à package paths are created by using file system directories and files
m y
d e
c a
A
Remember: a package is simply a module that can contain other modules/packages

te
B y
th
On a file system we therefore have to use directories for packages

a
© M
t
The directory name becomes the package name

h
yr i g
o p
So where does the code go for the package (since it is a module)?
__init__.py
C
__init__.py

m y
e
To define a package in our file system, we must:

create a directory whose name will be the package name

c a d
te A
y
create a file called __init__.py inside that directory

th B
M a
That __init__.py file is what tells Python that the directory is a package as opposed to

©
a standard directory

h t
yr i g
p
(if we don't have an __init__.py file, then Python creates an implicit namespace package)

C o
– we'll discuss that later
What happens when a file based package is imported?

app/
pack1/
m y
__init__.py
d e
module1.py
module2.py
c a
te A
y
import pack1

the code for pack1 is in __init__.py

th B
M a
that code is loaded, executed and cached in sys.modules with a key of pack1

t
it's just a module!
©
i g h
the symbol pack1 is added to our namespace referencing the same object

p yr
it's just a module!

C o
but, it has a __path__ property à file system directory path (absolute)

also has a __file__ property à file system path to __init__.py (absolute)

Nested Packages

Packages can contain modules as well as packages

m y
d e
c a
A
app/ module inside pack1 pack1.module1a

te
pack1/

y
__init__.py
module1a.py
module1b.py
th B
a
package inside pack1 pack1.pack1_1
pack1_1/
__init__.py
© M
t
module inside pack1.pack1_1 pack1.pack1_1.module1_1b

h
module1_1a.py

i
module1_1b.py

yr g
o p
C
__file__, __path__ and __package__ Properties

Modules have __file__ and __package__ properties

m y
d e
__file__ is the location of module code in the file system
c a
te A
B y
th
__package__ is the package the module code is located in

a
M
(an empty string if the module is located in the application root)

©
h t
yr i g
If the module is also a package, then it also has a __path__ property

o p
C
__path__ is the location of the package (directory) in the file system
app/
module.py

pack1/
m y
__init__.py module.__file__ à …/app/module.py
d e
module1a.py
module1b.py
module.__path__ à not set
c a
pack1_1/
module.__package__ à ''

te A
__init__.py
B y
module1_1a.py
module1_1b.py
th
pack1.__file__ à …/app/pack1/__init__.py

a
pack1.__path__ à …/app/pack1

© M
pack1.__package__ à pack1

h t
yr i g pack1.module1a.__file__ à …/app/pack1/module1a.py

o p pack1.module1a.__path__ à not set

C
pack1.module1a.__package__ à pack1
app/
module.py

pack1/
m y
__init__.py
d e
module1a.py
module1b.py
c a
pack1_1/
te A
__init__.py
B y
module1_1a.py
module1_1b.py
a th
© M
t
pack1.pack1_1.__file__ à …/app/pack1/pack1_1/__init__.py

g h
pack1.pack1_1.__path__ à …/app/pack1/pack1_1

i
yr
pack1.pack1_1.__package__ à pack1.pack1_1

o p
C
pack1.pack1_1.module1_1a.__file__ à …/app/pack1/pack1_1/module1_1a.py
pack1.pack1_1.module1_1a.__path__ à not set
pack1.pack1_1.module1_1a.__package__ à pack1.pack1_1
 What gets loaded during the import phase?

app/ import pack1.pack1_1.module1_1a

m y
e
module.py

pack1/
at the very least:

c a d
A
__init__.py pack1 is imported and added to sys.modules

te
module1a.py

y
module1b.py pack1_1 is imported and added to sys.modules

pack1_1/
th B
module1_1a is imported and added to sys.modules
__init__.py
module1_1a.py
M a
©
module1_1b.py but, modules can import other modules!

h t pack1.__init__.py could import other modules/packages

yr i g pack1_1.__init__.py could import other modules/packages

o p module1_1a.__init__.py could import other modules/packages

C
For example…
# pack1.__init__.py
app/
m y
e
import pack1.module1a
module.py

pack1/
import pack1.module1b

c a d
__init__.py
module1a.py
te A
y
import pack1.pack1_1.module1_1a
module1b.py

th
Just as before: B
a
pack1_1/
pack1 is imported and added to sys.modules

M
__init__.py
module1_1a.py
module1_1b.py
t © pack1_1 is imported and added to sys.modules

i g h module1_1a is imported and added to sys.modules

p yr
o
but now also:

C module1a is imported and added to sys.modules

module1b is imported and added to sys.modules
For example…
# pack1.__init__.py
app/
m y
e
import pack1.module1a
module.py

pack1/
import pack1.module1b

c a d
__init__.py
module1a.py import pack1
te A
module1b.py

B y
pack1_1/
th
pack1 is imported and added to sys.modules

a
M
__init__.py module1a is imported and added to sys.modules
module1_1a.py
module1_1b.py
t ©module1b is imported and added to sys.modules

i g h
p yr
C o
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Code Organization, Ease of Use…

Suppose you have 50 different functions and classes in your program

m y
d e
api.py
c a
(single file)

connect
execute_no_result
User
UserProfile A
audit_endpoint

te
execute_single_row Users

B yLogger

h
execute_multi_row
BlogPost
a t validate_email

M
normalize_string BlogPosts validate_phone
convert_str_to_bool validate_name
format_iso_date
t ©RouteTable
current_time_utc

i g h Configuration etc…

p yr
authenticate JSONEncoder

o
validate_token

C
get_permissions UnitTests
in one file???
authorize_endpoint
Start with Modules… better…

api/ but still unwieldy – everything is at the top level

m y
e
api.py too many imports: import dbutilities
dbutilities.py import

c
import
a d
jsonutilities
typeconversions
jsonutilities.py

te A
import
import
validations
authentication
typeconversions.py

B y import authorization

validations.py

a th import
etc…
users

authentication.py

© M
authorization.py

h t certain modules could be broken down further:

users.py

yr i g dbutilities à connections, queries

p
users à User, Users, UserProfile
blogposts.py

C o
logging.py certain modules belong "together":
authentication, authorization à security
unittests.py
So, Packages… api/

y
utilities/

m
api/ __init__.py
api.py database/

d e
a
__init__.py
dbutilities.py
c
connections.py
queries.py
A
te
jsonutilities.py

y
json/

B
typeconversions.py __init__.py

validations.py
a th encoders.py
decoders.py

authentication.py

© M security/
__init__.py
authorization.py
h t authentication.py

users.py

yr i g authorization.py
models/

o p
blogposts.py __init__.py

C
users/
logging.py __init__.py
user.py
unittests.py userprofile.py
Another Use Case

You have a module that implements 2 functions/classes for users of the module
m y
d e
a
Those two objects require 20 different helper functions and 2 additional helper classes

c
te A
y
From module developer's perspective:

h B
much easier to break the code down into multiple modules

t
M a
From module user's perspective:

t ©
g h
they just want a single import for the function and the class

i
p yr
i.e. it should look like a single module

C o
Module Developer's Perspective

m y
mylib/
function to be exported to user lives here
d e
__init__.py

c a
A
submod1.py
submod2.py

y te
B
subpack1
__init__.py

a th
class to be exported to user lives here

M
pack1mod1.py
pack1mod2.py

t ©
i g h
p yr
Smaller code modules, with a specific purpose, are easier to write, debug, test, and understand

C o
Module User's Perspective
mylib/

y
function to be exported to user lives here

m
__init__.py
submod1.py

d e
submod2.py

c a
subpack1

te A
y
__init__.py
class to be exported to user lives here
pack1mod1.py

th B
a
pack1mod2.py

User should not have to write:

© M from mylib.submod1 import my_func

h t from mylib.subpack1.pack1mod2 import MyClass

yr i g
o p
Much easier for user if they could write: from mylib import my_func, MyClass

C or, simply import mylib

mylib.my_func() mylib.MyClass()
Using __init__.py

m
We can use packages' __init__.py code to export (expose) just what's needed by our usersy
d e
c a
A
# mylib.__init__.py
Example:

y te
from mylib.submod1 import my_func

B
mylib/ from mylib.subpack1.pack1mod2 import MyClass
__init__.py

a th
M
submod1.py User uses it this way:
function to be exported

©
submod2.py import mylib

t
to user lives here

h
subpack1

g
mylib.my_func()

yr
__init__.py
i mylib.MyClass()

o p
pack1mod1.py
our internal implementation is "hidden"

C
pack1mod2.py class to be exported
to user lives here
We'll cover this in the next video
So, why Packages?

m y
ability to break code up into smaller chunks, makes our code:

d e
easier to write

c a
A
easier to test and debug just like books are broken down into

te
easier to read/understand chapters, sections, paragraphs, etc.

y
easier to document

th B
but they can still be "stitched" together

M a
©
hides inner implementation from users
makes their code
h t
yr i g
easier to write
easier to test and debug

o p easier to read/understand

C
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
What are Implicit Namespace Packages?

Namespace packages are package-like

m y
d e
directories
c a
may contain modules

te A
may contain nested regular packages
B y
a th
may contain nested namespace packages

M
but cannot contain __init__.py

©
h t
yr i g
These directories are implicitly made into these special types of packages

o p
C PEP 420
 Mechanics

m y
d e
a
utils/ utils/ does not contains __init__.py à namespace package
validators/
Ac
validators/ does not contain __init__.py à namespace package

te
boolean.py boolean.py is a file with a .py extension à module
date.py
B y
json/

a th
json/ contains __init__.py à regular package

M
__init__.py
serializers.py is a file with a .py extension

©
serializers.py à module
validators.py

h t
yr i g
o p
C
Regular vs Namespace Packages

m y
Regular Package Namespace Package

d e
type à module type
c a
à module

__init__.py à yes __init__.py

te A
à no

B y
__file__ à package __init__

a th
__file__ à not set

paths
©
à breaks if parent M paths à dynamic path computation

h t
directories change so OK if parent directories change

g
and absolute imports

yr i
are used (your import statements will still

p
need to be modified)

C
directory
o
single package lives in single single package can live in multiple (non-nested)
directories
in fact, parts of the namespace may even be
in a zip file
 app/
Example utils/

m y
validators/
boolean.py
namespace regular

d e common/

a
package package
utils
A c
common
__init__.py

e
validators/
type module
y t module boolean.py

th B
a
__name__ utils common

__repr__()
© M
<module utils (namespace)> <module common from '…/app/common'>

h t
__path__
i g
_Namespace(['…/app/utils'])

yr
['…/app/utils']

__file__

o p not set …/app/common/__init__.py

__package__C utils common

common.validators
à validators utils.validators
Import Examples

utils/
m y
validators/
d e
boolean.py
c a
A
import utils.validators.boolean

te
date.py

y
json/ from utils.validators import date
__init__.py

th B
import utils.validators.json.serializers

a
serializers.py
validators.py

© M
h t
yr i g
First familiarize yourself with regular packages.

o p
Once you are completely comfortable with them, check out namespace packages if you want

C
Read PEP 420 – that should definitely be your starting point
 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
What's in this section?

m y
e
Tips and tricks

c a d
A
Send me your suggestions!
Pythonic code

y te
B
Opiniated

a th
Things I find interesting

© M
h t
Will grow over time

yr i g
o p
C
This is NOT going to be discussions of 3rd party Library XYZ

(As of today PyPi has 128,291 packages!)

 m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
The Python documentation

That should be your top bookmark for Python https://docs.python.org

m y
Don't forget to make sure you are looking at your version of Python.
d e
3.6 or above please!

c a
te A
B y
a th
© M
h t
yr i g
o p
C
PEP – Python Enhancement Proposals

These are a fantastic resource to understand how certain things work in Python,

m y
e
and why they were implemented in a certain way..

c a d
Not all PEPs actually make it into Python. Some are rejected, deferred or even withdrawn.

te A
Reading the PEPs that have not been accepted also provides a lot of insight!

y
A lot of thought by many people go into these PEPs, whether they make it or not.

B
a th
© M Some PEPs are for language features

t
some are informational only

i g h
p yr
C
Index page
o https://www.python.org/dev/peps/ search on that page
But sometimes a web search such as: Python PEP Style Guide is more practical
PEP – Some Notable Ones

m y
PEP 8 – Style Guide and Idiomatic Python
d e
c a
PEP 20 – Zen of Python

te A
or just type import this in a Python console/Jupyter

PEP 484 – Type Hints

B y
a th
M
PEP 468 – Python 3.6 Release Schedule

©
t
PEP 537 – Python 3.7 Release Schedule or whatever release your interested in at the time

i g h they provide links to other PEPs relevant to the release

p yr
C o
And many many more, depending on what topic you’re interested in
Wikipedia

Great resource for explanations of general computer science concepts

m y
d e
c a
te A
B y
a th
© M
h t
yr i g
o p
C
Books
These are my favorite Python specific go to books
y
not in any particular order of importance!

m
d e
c a
Learning Python e A
y t
Mark Lutz
th B
M a
h t©
y r ig
o p
C
Books

m y
d e
c a
Fluent Python
te A
B y
Luciano Ramalho

a th
© M
h t
yr i g
o p
C
Books

m y
d e
c a
Python Cookbooke A
David Beazley & y t K. Jones
th B Brian

M a
h t©
y rig
o p
C
Books

m y
d e
c a
Effective Python: e A
y t
59 Specific Ways to Write Better Python
Brett Slatkin
t h B
M a
h t©
y rig
o p
C
Books

m y
d e
c a
Python in a Nutshell
te A
B y
Alex Martelli, Anna Ravenscroft & Steve Holden

a th
© M
h t
yr i g
o p
C
Other Online Resources I Regularly Use

Raymond Hettinger's Twitter Feed @raymondh just awesome!

m y
d e
example:
a
#python tip: zip() with star-arguments is

c
A
great for transposing 2-D data:

te
m = [(1, 2, 3), (4, 5, 6)]

y
list(zip(*m))

B
[(1, 4), (2, 5), (3, 6)]

YouTube
a
Lots of great videos on Python.th
© M
Look out for PyCon videos – these are fantastic!
Anything by GvR, Raymond Hettinger, Alex Martelli…

h t
And many more, including any library you're interested in

yr i g
o p
Planet Python Blog http://planetpython.org/
Google Searches!
C
Stack Overflow https://stackoverflow.com/