Introduction To Python Generators

24/9/2016
IntroductiontoPythongeneratorshowchoo
KNOWLEDGE FOR THE WORLD
Introduction to Python generators

by tyler (154)
Favorite 1
Share
Comment
Generators in Python are incredibly powerful yet often hard to understand. In this guide we'll
cover generators in depth. We'll talk about how and why to use them, the dierence between
generator functions and regular functions, the yield keyword, and provide examples.
This guide assumes you have a basic knowledge of Python (especially how regular functions
work).
Throughout this guide we are going to work towards solving a problem. Suppose we are tasked
with writing a function that accepts two parameters - a list of le names and a pattern to
match. We need to read from all of the les and return an iterable containing only the lines
that match the pattern.
IN THESE INTERESTS [?]
python
code
79 SUBSCRIBERS SUBSCRIBE
106 SUBSCRIBERS SUBSCRIBE
1 Regular functions
Regular functions are straight forward. Execution starts at the rst line and continues until it
reaches a return statement, exception, or the end of the function, which implies return None.
We might be tempted to solve our problem using a regular function. It may look something like
this:
deffind_matches(filenames,pattern):
matches=[]
forfnameinfilenames:
forlineinopen(fname):
ifpatterninline:
matches.append(line)
returnmatches
Then we could use our function like this:
https://howchoo.com/g/otcwnwe2ndb/introductiontopythongenerators
1/6
24/9/2016
files=['t1.txt','t2.txt','t3.txt']
formatchinfind_matches(files,'the'):
printmatch
Our nd_matches function is pretty easy to follow. We loop through the le names, open each
le, then loop through each line in the le and test it against the pattern. If it matches, we
append it to a list. Finally we return the list.
This meets all of our requirements, sort of.
Let's talk about what is happening here. When we call nd_matches we are passing control
over to this function. nd_matches starts executing at the rst line, does all of the work
required to build up our list of matches, and then returns the full list. When it returns, it
returns control back to the caller and completely nishes executing.
This is generally what we expect from a function. However, there are some problems. What if
we're dealing with extremely large log les? And what if there are a lot of them? Fortunately
for us, Python's open actually is ecient and doesn't load the entire le into memory so we're
safe there. But what if our matches list far exceeds the available memory on our machine? I
know what you're thinking, we could just buy more memory or a new computer altogether, but
that isn't very scalable. The right solution involves being more ecient.
Enter generators.
2 Generator functions and generator objects

A generator function looks very similar to a regular function, but there's one major dierence:
yield. When you include the yield keyword in a function, the function automatically becomes a
generator function.
This means that when we call the generator function, the rst thing it does is return a
generator object without beginning execution at all. When we call next, only then does it begin
execution, and it executes until it reaches a yield statement.
Here's a basic, useless example:
defuseless():
yield"KingArthur"
yield"BraveSirRobin"
yield"SirGalahadtheChaste"
u=useless()
printnext(u)
2/6
24/9/2016
#"KingArthur"
printnext(u)
#"BraveSirRobin"
printnext(u)
#"SirGalahadtheChaste"
You can see here that whenever you call next on the generator object, it continues executing
until it reaches another yield statement. So what happens if you call next again?
Well, it raises a StopIteration exception.
$pythonuseless.py
KingArthur
BraveSirRobin
SirGalahadtheChaste
Traceback(mostrecentcalllast):
File"useless.py",line10,in<module>
printnext(u)
StopIteration
What is interesting about the generator function is that even though control is passed back to
the caller, its state is frozen. Calling next simply resumes execution until it reaches another
yield statement.
The value of using a generator for our purpose is clear. We can now write a generator function
that yields one match at a time rather than loading up all of the matches in memory.
deffind_matches(filenames,pattern):
forfnameinfilenames:
forlineinopen(fname):
ifpatterninline:
yieldline
We can call it the same way and get the same apparent results.
files=['t1.txt','t2.txt','t3.txt']
formatchinfind_matches(files,'the'):
printmatch
The dierence is our generator can handle extremely large les and many of them. Without a
generator this would be extremely messy.
3/6
24/9/2016
tyler (154)
Software Engineer and creator of howchoo.
Favorite
Share
Comment
1 FAVORITE
Share this guide!

Like
Share
Tweet
You might also like:

Getting Started With Django Forms
by Dayne in django, python
How to use recursion in Python

by Dayne in python
How BEM works and why it's awesome

by Zach in webdev, css, code
Learn the slow (and fast) way to append elements to the DOM
by tyler in javascript, jquery, code
How to make iPython work in a virtualenv virtual environment

by tyler in python
Using Python descriptors

by tyler in python
Install pip on Mac OS X

by tyler in mac, python
Discuss this guide

4/6
24/9/2016
0Comments
Recommend
howchoo
Share
Login
SortbyBest
Startthediscussion
Bethefirsttocomment.
ALSOONHOWCHOO
WalnutCoatRack
AddvoicecontrolstoyourRaspberryPi
usingJasper
1comment4monthsago
7comments5monthsago
TylerThisagreatproject.
RulesYouThisiscool.Thanksforsharing.I
ranintoanerrorforthefirsttimerunning.
ValueError:STTEngine'Sphinx'isnot
available.(Duetomissing
UsingtheMonokaithemeinTerminal(OSX)
CustomKeyMappingsVim
4commentsayearago
1comment2yearsago
daynejones88IforgothowmuchIlovedthis
TylerThisguideisexcellent.Ilovetheidea
theme.
tousekeymappingstoquicklyaddfrequently
usedcodesnippets.
Subscribe d AddDisqustoyoursiteAddDisqusAdd
Privacy
/python
You.
Join howchoo
About your privacy
Us.
Our team
Together.
Follow us on Facebook
Follow us on Pinterest
5/6
24/9/2016
Contact us
Sponsor an interest
2016 howchoo, llc | Sitemap | DMCA requests
6/6

Introduction To Python Generators - Howchoo

Uploaded by

Introduction To Python Generators - Howchoo

Uploaded by

24/9/2016

106 SUBSCRIBERS SUBSCRIBE

Then we could use our function like this:

2 Generator functions and generator objects

Share this guide!

You might also like:

How to use recursion in Python

How BEM works and why it's awesome

How to make iPython work in a virtualenv virtual environment

Using Python descriptors

Install pip on Mac OS X

Discuss this guide

2016 howchoo, llc | Sitemap | DMCA requests

You might also like