Python-Guide by Kenneth

Python Guide Documentation
Release 0.0.1
Kenneth Reitz
October 15, 2014
Contents
Getting Started
1.1 Picking an Interpreter . . . . .
1.2 Installing Python on Mac OS X
1.3 Installing Python on Windows .
1.4 Installing Python on Linux . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
5
6
8
Writing Great Code

2.1 Structuring Your Project
2.2 Code Style . . . . . . .
2.3 Reading Great Code . .
2.4 Documentation . . . . .
2.5 Testing Your Code . . .
2.6 Common Gotchas . . .
2.7 Choosing a License . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
11
11
17
26
27
29
33
35
Scenario Guide
3.1 Network Applications . . .
3.2 Web Applications . . . . .
3.3 HTML Scraping . . . . . .
3.4 Command-line Applications
3.5 GUI Applications . . . . . .
3.6 Databases . . . . . . . . . .
3.7 Networking . . . . . . . . .
3.8 Systems Administration . .
3.9 Continuous Integration . . .
3.10 Speed . . . . . . . . . . . .
3.11 Scientific Applications . . .
3.12 Image Manipulation . . . .
3.13 XML parsing . . . . . . . .
3.14 Cryptography . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
37
37
38
43
45
46
47
48
49
53
55
58
60
61
62
Shipping Great Code

4.1 Packaging Your Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Freezing Your Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
63
63
65
Development Environment
5.1 Your Development Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
67
67
72
.
.
.
.
.
.
.
ii
Additional Notes
6.1 Introduction . . . . .
6.2 The Community . . .
6.3 Learning Python . . .
6.4 Documentation . . . .
6.5 News . . . . . . . . .
6.6 Contribute . . . . . .
6.7 License . . . . . . . .
6.8 The Guide Style Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
75
75
76
77
81
82
82
84
84
Python Guide Documentation, Release 0.0.1
Welcome to The Hitchhikers Guide to Python.

This guide is currently under heavy active development. If youd like to help, fork us on GitHub!
This opinionated guide exists to provide both novice and expert Python developers a best-practice handbook to the
installation, configuration, and usage of Python on a daily basis.
Contents
Contents
CHAPTER 1
Getting Started
This part of the guide focuses on setting up your Python environment.
1.1 Picking an Interpreter

1.1.1 The State of Python (2 vs 3)
When choosing a Python interpreter, one looming question is always present: Should I choose Python 2 or Python
3? The answer is not as obvious as one might think.
The basic gist of the state of things is as follows:
1. Python 2.7 has been the standard for a long time.
2. Python 3 introduced major changes to the language, which many developers are unhappy with.
3. Python 2.7 will receive neccessary security updates for a few years.
4. Python 3 is continually evolving, like Python 2 did in years past.
So, you can now see why this is not such an easy decision.
1.1.2 Recommendations
Ill be blunt:
Use Python 3 if...
You dont care.
You love Python 3.
You are indifferent towards 2 vs 3.
You dont know which one to use.
You embrace change.
Use Python 2 if...
You love Python 2 and are saddened by the future being Python 3.
The stability requirements of your software would be improved by a language and runtime that never changes.
Software that you depend on requires it.
1.1.3 So.... 3?
If youre choosing a Python interpreter to use, and arent opinionated, then I recommend you use the newest Python 3.x,
since every version brings new and improved standard library modules, security and bug fixes. Progress is progress.
Given such, only use Python 2 if you have a strong reason to, such as a Python 2 exclusive library which has no
adequate Python 3 ready alternative, or you (like me) absolutely love and are inspired by Python 2.
Check out Can I Use Python 3? to see if any software youre depending on will block your adoption of Python 3.
Further Reading
It is possible to write code that works on Python 2.6, 2.7, and 3.3. This ranges from trivial to hard depending upon the
kind of software you are writing; if youre a beginner there are far more important things to worry about.
1.1.4 Implementations
When people speak of Python they often mean not just the language but also the CPython implementation. Python is
actually a specification for a language that can be implemented in many different ways.
CPython
CPython is the reference implementation of Python, written in C. It compiles Python code to intermediate bytecode
which is then interpreted by a virtual machine. CPython provides the highest level of compatibility with Python
packages and C extension modules.
If you are writing open-source Python code and want to reach the widest possible audience, targeting CPython is best.
To use packages which rely on C extensions to function, CPython is your only implementation option.
All versions of the Python language are implemented in C because CPython is the reference implementation.
PyPy
PyPy is a Python interpreter implemented in a restricted statically-typed subset of the Python language called RPython.
The interpreter features a just-in-time compiler and supports multiple back-ends (C, CLI, JVM).
PyPy aims for maximum compatibility with the reference CPython implementation while improving performance.
If you are looking to increase performance of your Python code, its worth giving PyPy a try. On a suite of benchmarks,
its currently over 5 times faster than CPython.
PyPy supports Python 2.7. PyPy3 1 , released in beta, targets Python 3.
Jython
Jython is a Python implementation that compiles Python code to Java bytecode which is then executed by the JVM
(Java Virtual Machine). Additionally, it is able to import and use any Java class like a Python module.
If you need to interface with an existing Java codebase or have other reasons to need to write Python code for the JVM,
Jython is the best choice.
Jython currently supports up to Python 2.5.
1
2
http://pypy.org/compat.html
http://wiki.python.org/jython/JythonFaq/GeneralInfo#Is_Jython_the_same_language_as_Python.3F
Chapter 1. Getting Started
IronPython
IronPython is an implementation of Python for the .NET framework. It can use both Python and .NET framework
libraries, and can also expose Python code to other languages in the .NET framework.
Python Tools for Visual Studio integrates IronPython directly into the Visual Studio development environment, making
it an ideal choice for Windows developers.
IronPython supports Python 2.7.
PythonNet
Python for .NET is a package which provides near seamless integration of a natively installed Python installation with
the .NET Common Language Runtime (CLR). This is the inverse approach to that taken by IronPython (see above),
to which it is more complementary than competing with.
In conjunction with Mono, PythonNet enables native Python installations on non-Windows operating systems, such as
OS X and Linux, to operate within the .NET framework. It can be run in addition to IronPython without conflict.
PythonNet supports from Python 2.3 up to Python 2.7.
Properly Install Python
1.2 Installing Python on Mac OS X

The latest version of Mac OS X, Mavericks, comes with Python 2.7 out of the box. You do not
need to install or configure anything else to use Python. Having said that, I would strongly recommend that you install the tools and libraries described in the next section before you start building
Python applications for real-world use. In particular, you should always install Setuptools, as it
makes it much easier for you to use other third-party Python libraries. The version of Python that
ships with OS X is great for learning. Yet, its not good for development. The version shipped
with OS X may be out of date from the official current Python release, which is considered the
stable production version.
1.2.1 Doing it Right
Lets install a real version of Python.
Before installing Python, youll need to install GCC. GCC can be obtained by downloading XCode, the smaller Command Line
Tools (must have an Apple account) or the even smaller OSX-GCC-Installer package.
Note:
If you already have XCode installed, do not install OSX-GCC-Installer.
In
combination,
the
software
can
cause
issues
that
are
difficult
to
diagnose.
While OS X comes with a large number of UNIX utilities, those familiar with Linux systems will notice
one key component missing: a decent package manager. Homebrew fills this void. To install Homebrew,
simply run
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
The script will explain what changes it will make and prompt you before the installation begins. Once
youve installed Homebrew, insert the Homebrew directory at the top of your PATH environment variable.
You can do this by adding the following line at the bottom of your ~/.profile file
3
4
http://ironpython.codeplex.com/releases/view/81726
http://pythonnet.github.io/readme.html
1.2. Installing Python on Mac OS X
export PATH=/usr/local/bin:/usr/local/sbin:$PATH
Now, we can install Python 2.7:

$ brew install python
This will take a minute or two.
1.2.2 Setuptools & Pip

Homebrew installs Setuptools and pip for you.
Setuptools enables you to download and install any compliant Python software over a network (usually the
Internet) with a single command (easy_install). It also enables you to add this network installation
capability to your own Python software with very little work.
pip is a tool for easily installing and managing Python packages, that is recommended over
easy_install. It is superior to easy_install in several ways, and is actively maintained.
1.2.3 Virtualenv
After Setuptools & Pip, the next development tool that you should install is virtualenv. Use pip
$ pip install virtualenv
The virtualenv kit provides the ability to create virtual Python environments that do not interfere with
either each other, or the main Python installation. If you install virtualenv before you begin coding then
you can get into the habit of using it to create completely clean Python environments for each project.
This is particularly important for Web development, where each framework and application will have
many dependencies.
To set up a new Python environment, move into the directory where you would like to store the environment, and use the virtualenv utility to create the new environment.
$ virtualenv venv
To use an environment, run source venv/bin/activate. Your command prompt will change to
show the active environment. Once you have finished working in the current virtual environment, run
deactivate to restore your settings to normal.
Each new environment automatically includes a copy of pip, so that you can setup the third-party libraries
and tools that you want to use in that environment. Put your own code within a subdirectory of the
environment, however you wish. When you no longer need a particular environment, simply copy your
code out of it, and then delete the main directory for the environment.
A useful set of extensions to virtualenv is available in virtualenvwrapper, RTFD to find out more.
This page is a remixed version of another guide, which is available under the same license.
1.3 Installing Python on Windows

First, download the latest version of Python 2.7 from the official Website. If you want to be sure you
are installing a fully up-to-date version then use the Windows Installer link from the home page of the
Python.org web site .
6
The Windows version is provided as an MSI package. To install it manually, just double-click the file. The
MSI package format allows Windows administrators to automate installation with their standard tools.
By design, Python installs to a directory with the version number embedded, e.g. Python version 2.7 will
install at C:\Python27\, so that you can have multiple versions of Python on the same system without
conflicts. Of course, only one interpreter can be the default application for Python file types. It also does
not automatically modify the PATH environment variable, so that you always have control over which
copy of Python is run.
Typing the full path name for a Python interpreter each time quickly gets tedious, so add the directories for
your default Python version to the PATH. Assuming that your Python installation is in C:\Python27\,
add this to your PATH:
C:\Python27\;C:\Python27\Scripts\
You can do this easily by running the following in powershell:
[Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27\;C:\Python27\Scripts\", "Us
The second (Scripts) directory receives command files when certain packages are installed, so it is a
very useful addition. You do not need to install or configure anything else to use Python. Having said
that, I would strongly recommend that you install the tools and libraries described in the next section
before you start building Python applications for real-world use. In particular, you should always install
Setuptools, as it makes it much easier for you to use other third-party Python libraries.
1.3.1 Setuptools + Pip

The most crucial third-party Python software of all is Setuptools, which extends the packaging and installation facilities provided by the distutils in the standard library. Once you add Setuptools to your Python
system you can download and install any compliant Python software product with a single command. It
also enables you to add this network installation capability to your own Python software with very little
work.
To obtain the latest version of Setuptools for Windows, run the Python script available here: ez_setup.py
Youll now have a new command available to you: easy_install. It is considered by many to be deprecated, so we will install its replacement: pip. Pip allows for uninstallation of packages, and is actively
maintained, unlike easy_install.
To install pip, run the Python script available here: get-pip.py
1.3.2 Virtualenv
> pip install virtualenv
many dependencies.
To set up a new Python environment, change the working directory to wherever you want to store the
environment, and run the virtualenv utility in your projects directory
1.3. Installing Python on Windows
> virtualenv venv
To use an environment, run the activate.bat batch file in the Scripts subdirectory of that environment. Your command prompt will change to show the active environment. Once you have finished
working in the current virtual environment, run the deactivate.bat batch file to restore your settings
to normal.
Each new environment automatically includes a copy of pip in the Scripts subdirectory, so that you
can setup the third-party libraries and tools that you want to use in that environment. Put your own code
within a subdirectory of the environment, however you wish. When you no longer need a particular
environment, simply copy your code out of it, and then delete the main directory for the environment.
1.4 Installing Python on Linux

The latest versions of Ubuntu and Fedora come with Python 2.7 out of the box.
The latest versions of Redhat Enterprise (RHEL) and CentOS come with Python 2.6. Some older versions
of RHEL and CentOS come with Python 2.4 which is unacceptable for modern Python development.
Fortunately, there are Extra Packages for Enterprise Linux which include high quality additional packages
based on their Fedora counterparts. This repository contains a Python 2.6 package specifically designed
to install side-by-side with the systems Python 2.4 installation.
You do not need to install or configure anything else to use Python. Having said that, I would strongly
recommend that you install the tools and libraries described in the next section before you start building
Python applications for real-world use. In particular, you should always install Setuptools, as it makes it
much easier for you to use other third-party Python libraries.
1.4.1 Setuptools & Pip

The most crucial third-party Python software of all is Setuptools, which extends the packaging and installation facilities provided by the distutils in the standard library. Once you add Setuptools to your Python
system you can download and install any compliant Python software product with a single command. It
also enables you to add this network installation capability to your own Python software with very little
work.
To obtain the latest version of Setuptools for Linux, refer to the documentation available here: unixsetuptools
The new easy_install command you have available is considered by many to be deprecated, so we
will install its replacement: pip. Pip allows for uninstallation of packages, and is actively maintained,
unlike easy_install.
To install pip, simply open a command prompt and run
$ easy_install pip
1.4.2 Virtualenv
many dependencies.
To set up a new Python environment, change the working directory to where ever you want to store the
environment, and run the virtualenv utility in your projects directory
$ virtualenv venv
To use an environment, run source venv/bin/activate. Your command prompt will change to
show the active environment. Once you have finished working in the current virtual environment, run
deactivate to restore your settings to normal.
Each new environment automatically includes a copy of pip, so that you can setup the third-party libraries
and tools that you want to use in that environment. Put your own code within a subdirectory of the
environment, however you wish. When you no longer need a particular environment, simply copy your
code out of it, and then delete the main directory for the environment.
1.4. Installing Python on Linux
10
CHAPTER 2
Writing Great Code
This part of the guide focuses on best practices for writing Python code.
2.1 Structuring Your Project

By structure we mean the decisions you make concerning how your project best meets its objective. We need to
consider how to best leverage Pythons features to create clean, effective code. In practical terms, structure means
making clean code whose logic and dependencies are clear as well as how the files and folders are organized in the
filesystem.
Which functions should go into which modules? How does data flow through the project? What features and functions
can be grouped together and isolated? By answering questions like these you can begin to plan, in a broad sense, what
your finished product will look like.
In this section we take a closer look at Pythons module and import systems as they are the central elements to enforcing
structure in your project. We then discuss various perspectives on how to build code which can be extended and tested
reliably.
2.1.1 Structure is Key

Thanks to the way imports and modules are handled in Python, it is relatively easy to structure a Python project. Easy,
here, means that you do not have many constraints and that the module importing model is easy to grasp. Therefore,
you are left with the pure architectural task of crafting the different parts of your project and their interactions.
Easy structuring of a project means it is also easy to do it poorly. Some signs of a poorly structured project include:
Multiple and messy circular dependencies: if your classes Table and Chair in furn.py need to import Carpenter from workers.py to answer a question such as table.isdoneby(), and if conversely the class
Carpenter needs to import Table and Chair, to answer the question carpenter.whatdo(), then you have a
circular dependency. In this case you will have to resort to fragile hacks such as using import statements inside
methods or functions.
Hidden coupling: each and every change in Tables implementation breaks 20 tests in unrelated test cases
because it breaks Carpenters code, which requires very careful surgery to adapt the change. This means you
have too many assumptions about Table in Carpenters code or the reverse.
Heavy usage of global state or context: instead of explicitly passing (height, width, type, wood)
to each other, Table and Carpenter rely on global variables that can be modified and are modified on the fly
by different agents. You need to scrutinize all access to these global variables to understand why a rectangular
table became a square, and discover that remote template code is also modifying this context, messing with table
dimensions.
11
Spaghetti code: multiple pages of nested if clauses and for loops with a lot of copy-pasted procedural code
and no proper segmentation are known as spaghetti code. Pythons meaningful indentation (one of its most
controversial features) make it very hard to maintain this kind of code. So the good news is that you might not
see too much of it.
Ravioli code is more likely in Python: it consists of hundreds of similar little pieces of logic, often classes or
objects, without proper structure. If you never can remember if you have to use FurnitureTable, AssetTable or
Table, or even TableNew for your task at hand, you might be swimming in ravioli code.
2.1.2 Modules
Python modules are one of the main abstraction layers available and probably the most natural one. Abstraction layers
allow separating code into parts holding related data and functionality.
For example, a layer of a project can handle interfacing with user actions, while another would handle low-level
manipulation of data. The most natural way to separate these two layers is to regroup all interfacing functionality in
one file, and all low-level operations in another file. In this case, the interface file needs to import the low-level file.
This is done with the import and from ... import statements.
As soon as you use import statements you use modules. These can be either built-in modules such as os and sys,
third-party modules you have installed in your environment, or your projects internal modules.
To keep in line with the style guide, keep module names short, lowercase, and be sure to avoid using special symbols
like the dot (.) or question mark (?). So a file name like my.spam.py is one you should avoid! Naming this way will
interfere with the way Python looks for modules.
In the case of my.spam.py Python expects to find a spam.py file in a folder named my which is not the case. There is
an example of how the dot notation should be used in the Python docs.
If youd like you could name your module my_spam.py, but even our friend the underscore should not be seen often
in module names.
Aside from some naming restrictions, nothing special is required for a Python file to be a module, but you need to
understand the import mechanism in order to use this concept properly and avoid some issues.
Concretely, the import modu statement will look for the proper file, which is modu.py in the same directory as
the caller if it exists. If it is not found, the Python interpreter will search for modu.py in the path recursively and
raise an ImportError exception if it is not found.
Once modu.py is found, the Python interpreter will execute the module in an isolated scope. Any top-level statement
in modu.py will be executed, including other imports if any. Function and class definitions are stored in the modules
dictionary.
Then, the modules variables, functions, and classes will be available to the caller through the modules namespace, a
central concept in programming that is particularly helpful and powerful in Python.
In many languages, an include file directive is used by the preprocessor to take all code found in the file and
copy it into the callers code. It is different in Python: the included code is isolated in a module namespace, which
means that you generally dont have to worry that the included code could have unwanted effects, e.g. override an
existing function with the same name.
It is possible to simulate the more standard behavior by using a special syntax of the import statement: from modu
import *. This is generally considered bad practice. Using import * makes code harder to read and makes
dependencies less compartmentalized.
Using from modu import func is a way to pinpoint the function you want to import and put it in the global
namespace. While much less harmful than import * because it shows explicitly what is imported in the global
namespace, its only advantage over a simpler import modu is that it will save a little typing.
Very bad
12
Chapter 2. Writing Great Code
[...]
from modu import *
[...]
x = sqrt(4) # Is sqrt part of modu? A builtin? Defined above?
Better
from modu import sqrt
[...]
x = sqrt(4) # sqrt may be part of modu, if not redefined in between
Best
import modu
[...]
x = modu.sqrt(4)
# sqrt is visibly part of modus namespace
As mentioned in the Code Style section, readability is one of the main features of Python. Readability means to avoid
useless boilerplate text and clutter, therefore some efforts are spent trying to achieve a certain level of brevity. But
terseness and obscurity are the limits where brevity should stop. Being able to tell immediately where a class or
function comes from, as in the modu.func idiom, greatly improves code readability and understandability in all but
the simplest single file projects.
2.1.3 Packages
Python provides a very straightforward packaging system, which is simply an extension of the module mechanism to
a directory.
Any directory with an __init__.py file is considered a Python package. The different modules in the package are
imported in a similar manner as plain modules, but with a special behavior for the __init__.py file, which is used
to gather all package-wide definitions.
A file modu.py in the directory pack/ is imported with the statement import pack.modu. This statement will
look for an __init__.py file in pack, execute all of its top-level statements. Then it will look for a file named
pack/modu.py and execute all of its top-level statements. After these operations, any variable, function, or class
defined in modu.py is available in the pack.modu namespace.
A commonly seen issue is to add too much code to __init__.py files. When the project complexity grows, there
may be sub-packages and sub-sub-packages in a deep directory structure. In this case, importing a single item from a
sub-sub-package will require executing all __init__.py files met while traversing the tree.
Leaving an __init__.py file empty is considered normal and even a good practice, if the packages modules and
sub-packages do not need to share any code.
Lastly, a convenient syntax is available for importing deeply nested packages: import very.deep.module as
mod. This allows you to use mod in place of the verbose repetition of very.deep.module.
2.1.4 Object-oriented programming

Python is sometimes described as an object-oriented programming language. This can be somewhat misleading and
needs to be clarified.
In Python, everything is an object, and can be handled as such. This is what is meant when we say, for example, that
functions are first-class objects. Functions, classes, strings, and even types are objects in Python: like any objects, they
have a type, they can be passed as function arguments, they may have methods and properties. In this understanding,
Python is an object-oriented language.
2.1. Structuring Your Project
13
However, unlike Java, Python does not impose object-oriented programming as the main programming paradigm. It
is perfectly viable for a Python project to not be object-oriented, i.e. to use no or very few class definitions, class
inheritance, or any other mechanisms that are specific to object-oriented programming.
Moreover, as seen in the modules section, the way Python handles modules and namespaces gives the developer a
natural way to ensure the encapsulation and separation of abstraction layers, both being the most common reasons to
use object-orientation. Therefore, Python programmers have more latitude to not use object-orientation, when it is not
required by the business model.
There are some reasons to avoid unnecessary object-orientation. Defining custom classes is useful when we want to
glue together some state and some functionality. The problem, as pointed out by the discussions about functional
programming, comes from the state part of the equation.
In some architectures, typically web applications, multiple instances of Python processes are spawned to respond
to external requests that can happen at the same time. In this case, holding some state into instantiated objects,
which means keeping some static information about the world, is prone to concurrency problems or race-conditions.
Sometimes, between the initialization of the state of an object (usually done with the __init__() method) and the
actual use of the object state through one of its methods, the world may have changed, and the retained state may be
outdated. For example, a request may load an item in memory and mark it as read by a user. If another request requires
the deletion of this item at the same time, it may happen that the deletion actually occurs after the first process loaded
the item, and then we have to mark as read a deleted object.
This and other issues led to the idea that using stateless functions is a better programming paradigm.
Another way to say the same thing is to suggest using functions and procedures with as few implicit contexts and sideeffects as possible. A functions implicit context is made up of any of the global variables or items in the persistence
layer that are accessed from within the function. Side-effects are the changes that a function makes to its implicit
context. If a function saves or deletes data in a global variable or in the persistence layer, it is said to have a side-effect.
Carefully isolating functions with context and side-effects from functions with logic (called pure functions) allow the
following benefits:
Pure functions are deterministic: given a fixed input, the output will always be the same.
Pure functions are much easier to change or replace if they need to be refactored or optimized.
Pure functions are easier to test with unit-tests: There is less need for complex context setup and data cleaning
afterwards.
Pure functions are easier to manipulate, decorate, and pass-around.
In summary, pure functions, without any context or side-effects, are more efficient building blocks than classes and
objects for some architectures.
Obviously, object-orientation is useful and even necessary in many cases, for example when developing graphical
desktop applications or games, where the things that are manipulated (windows, buttons, avatars, vehicles) have a
relatively long life of their own in the computers memory.
2.1.5 Decorators
The Python language provides a simple yet powerful syntax called decorators. A decorator is a function or a class that
wraps (or decorates) a function or a method. The decorated function or method will replace the original undecorated
function or method. Because functions are first-class objects in Python, this can be done manually, but using the
@decorator syntax is clearer and thus preferred.
def foo():
# do something
def decorator(func):
# manipulate func
14
return func
foo = decorator(foo)
# Manually decorate
@decorator
def bar():
# Do something
# bar() is decorated
This mechanism is useful for separating concerns and avoiding external un-related logic polluting the core logic
of the function or method. A good example of a piece of functionality that is better handled with decoration is
memoization or caching: you want to store the results of an expensive function in a table and use them directly instead
of recomputing them when they have already been computed. This is clearly not part of the function logic.
2.1.6 Dynamic typing

Python is said to be dynamically typed, which means that variables do not have a fixed type. In fact, in Python,
variables are very different from what they are in many other languages, specifically statically-typed languages. Variables are not a segment of the computers memory where some value is written, they are tags or names pointing
to objects. It is therefore possible for the variable a to be set to the value 1, then to the value a string, then to a
function.
The dynamic typing of Python is often considered to be a weakness, and indeed it can lead to complexities and hardto-debug code. Something named a can be set to many different things, and the developer or the maintainer needs to
track this name in the code to make sure it has not been set to a completely unrelated object.
Some guidelines help to avoid this issue:
Avoid using the same variable name for different things.
Bad
a = 1
a = a string
def a():
pass # Do something
Good
count = 1
msg = a string
def func():
pass # Do something
Using short functions or methods helps reduce the risk of using the same name for two unrelated things.
It is better to use different names even for things that are related, when they have a different type:
Bad
items = a b c d # This is a string...
items = items.split( ) # ...becoming a list
items = set(items) # ...and then a set
There is no efficiency gain when reusing names: the assignments will have to create new objects anyway. However,
when the complexity grows and each assignment is separated by other lines of code, including if branches and loops,
it becomes harder to ascertain what a given variables type is.
Some coding practices, like functional programming, recommend never reassigning a variable. In Java this is done
with the final keyword. Python does not have a final keyword and it would be against its philosophy anyway. However,
2.1. Structuring Your Project
15
it may be a good discipline to avoid assigning to a variable more than once, and it helps in grasping the concept of
mutable and immutable types.
2.1.7 Mutable and immutable types

Python has two kinds of built-in or user-defined types.
Mutable types are those that allow in-place modification of the content. Typical mutables are lists and dictionaries:
All lists have mutating methods, like list.append() or list.pop(), and can be modified in place. The same
goes for dictionaries.
Immutable types provide no method for changing their content. For instance, the variable x set to the integer 6 has no
increment method. If you want to compute x + 1, you have to create another integer and give it a name.
my_list = [1, 2, 3]
my_list[0] = 4
print my_list # [4, 2, 3] <- The same list as changed
x = 6
x = x + 1
# The new x is another object
One consequence of this difference in behavior is that mutable types are not stable, and therefore cannot be used as
dictionary keys.
Using properly mutable types for things that are mutable in nature and immutable types for things that are fixed in
nature helps to clarify the intent of the code.
For example, the immutable equivalent of a list is the tuple, created with (1, 2). This tuple is a pair that cannot be
changed in-place, and can be used as a key for a dictionary.
One peculiarity of Python that can surprise beginners is that strings are immutable. This means that when constructing
a string from its parts, it is much more efficient to accumulate the parts in a list, which is mutable, and then glue (join)
the parts together when the full string is needed. One thing to notice, however, is that list comprehensions are better
and faster than constructing a list in a loop with calls to append().
Bad
# create a concatenated string from 0 to 19 (e.g. "012..1819")
nums = ""
for n in range(20):
nums += str(n)
# slow and inefficient
print nums
Good
nums = []
for n in range(20):
nums.append(str(n))
print "".join(nums) # much more efficient
Best
nums = [str(n) for n in range(20)]
print "".join(nums)
One final thing to mention about strings is that using join() is not always best. In the instances where you are
creating a new string from a pre-determined number of strings, using the addition operator is actually faster, but in
16
cases like above or in cases where you are adding to an existing string, using join() should be your preferred
method.
foo = foo
bar = bar
foobar = foo + bar # This is good
foo += ooo # This is bad, instead you should do:
foo = .join([foo, ooo])
Note: You can also use the % formatting operator to concatenate a pre-determined number of strings besides
str.join() and +. However, according to PEP 3101, the % operator became deprecated in Python 3.1 and will be
replaced by the str.format() method in the later versions.
foo = foo
bar = bar
foobar = %s%s % (foo, bar) # It is OK
foobar = {0}{1}.format(foo, bar) # It is better
foobar = {foo}{bar}.format(foo=foo, bar=bar) # It is best
2.1.8 Vendorizing Dependencies

2.1.9 Runners
2.1.10 Further Reading
http://docs.python.org/2/library/
http://www.diveintopython.net/toc/index.html
2.2 Code Style

If you ask Python programmers what they like most in Python, they will often say its high readability. Indeed, a high
level of readability is at the heart of the design of the Python language, following the recognised fact that code is read
much more often than it is written.
One reason for Python code to be easily read and understood is its relatively complete set of Code Style guidelines
and Pythonic idioms.
Moreover, when a veteran Python developer (a Pythonista) points to portions of code and says they are not Pythonic,
it usually means that these lines of code do not follow the common guidelines and fail to express the intent in what is
considered the best (hear: most readable) way.
On some border cases, no best way has been agreed upon on how to express an intent in Python code, but these cases
are rare.
2.2.1 General concepts

Explicit code
While any kind of black magic is possible with Python, the most explicit and straightforward manner is preferred.
2.2. Code Style
17
Bad
def make_complex(*args):
x, y = args
return dict(**locals())
Good
def make_complex(x, y):
return {x: x, y: y}
In the good code above, x and y are explicitly received from the caller, and an explicit dictionary is returned. The
developer using this function knows exactly what to do by reading the first and last lines, which is not the case with
the bad example.
One statement per line
While some compound statements such as list comprehensions are allowed and appreciated for their brevity and their
expressiveness, it is bad practice to have two disjoint statements on the same line of code.
Bad
print one; print two
if x == 1: print one
if <complex comparison> and <other complex comparison>:
# do something
Good
print one
print two
if x == 1:
print one
cond1 = <complex comparison>
cond2 = <other complex comparison>
if cond1 and cond2:
# do something
Function arguments
Arguments can be passed to functions in four different ways.
1. Positional arguments are mandatory and have no default values. They are the simplest form of arguments and
they can be used for the few function arguments that are fully part of the functions meaning and their order is natural.
For instance, in send(message, recipient) or point(x, y) the user of the function has no difficulty
remembering that those two functions require two arguments, and in which order.
In those two cases, it is possible to use argument names when calling the functions and, doing so, it is possible to
switch the order of arguments, calling for instance send(recipient=World, message=Hello) and
point(y=2, x=1) but this reduces readability and is unnecessarily verbose, compared to the more straightforward
calls to send(Hello, World) and point(1, 2).
2. Keyword arguments are not mandatory and have default values. They are often used for optional parameters sent
to the function. When a function has more than two or three positional parameters, its signature is more difficult to
18
remember and using keyword argument with default values is helpful. For instance, a more complete send function could be defined as send(message, to, cc=None, bcc=None). Here cc and bcc are optional, and
evaluate to None when they are not passed another value.
Calling a function with keyword arguments can be done in multiple ways in Python, for example it is possible to
follow the order of arguments in the definition without explicitly naming the arguments, like in send(Hello,
World, Cthulhu, God), sending a blind carbon copy to God. It would also be possible to name arguments in another order, like in send(Hello again, World, bcc=God, cc=Cthulhu). Those
two possibilities are better avoided without any strong reason to not follow the syntax that is the closest to the function
definition: send(Hello, World, cc=Cthulhu, bcc=God).
As a side note, following YAGNI principle, it is often harder to remove an optional argument (and its logic inside the
function) that was added just in case and is seemingly never used, than to add a new optional argument and its logic
when needed.
3. The arbitrary argument list is the third way to pass arguments to a function. If the function intention is better expressed by a signature with an extensible number of positional arguments, it can be defined with the *args constructs.
In the function body, args will be a tuple of all the remaining positional arguments. For example, send(message,
*args) can be called with each recipient as an argument: send(Hello, God, Mom, Cthulhu),
and in the function body args will be equal to (God, Mom, Cthulhu).
However, this construct has some drawbacks and should be used with caution. If a function receives a list of arguments
of the same nature, it is often more clear to define it as a function of one argument, that argument being a list or any sequence. Here, if send has multiple recipients, it is better to define it explicitly: send(message, recipients)
and call it with send(Hello, [God, Mom, Cthulhu]). This way, the user of the function can
manipulate the recipient list as a list beforehand, and it opens the possibility to pass any sequence, including iterators,
that cannot be unpacked as other sequences.
4. The arbitrary keyword argument dictionary is the last way to pass arguments to functions. If the function
requires an undetermined series of named arguments, it is possible to use the **kwargs construct. In the function
body, kwargs will be a dictionary of all the passed named arguments that have not been caught by other keyword
arguments in the function signature.
The same caution as in the case of arbitrary argument list is necessary, for similar reasons: these powerful techniques
are to be used when there is a proven necessity to use them, and they should not be used if the simpler and clearer
construct is sufficient to express the functions intention.
It is up to the programmer writing the function to determine which arguments are positional arguments and which are
optional keyword arguments, and to decide whether to use the advanced techniques of arbitrary argument passing. If
the advice above is followed wisely, it is possible and enjoyable to write Python functions that are:
easy to read (the name and arguments need no explanations)
easy to change (adding a new keyword argument does not break other parts of the code)
Avoid the magical wand
A powerful tool for hackers, Python comes with a very rich set of hooks and tools allowing to do almost any kind of
tricky tricks. For instance, it is possible to do each of the following:
change how objects are created and instantiated
change how the Python interpreter imports modules
it is even possible (and recommended if needed) to embed C routines in Python.
However, all these options have many drawbacks and it is always better to use the most straightforward way to achieve
your goal. The main drawback is that readability suffers greatly when using these constructs. Many code analysis
tools, such as pylint or pyflakes, will be unable to parse this magic code.
2.2. Code Style
19
We consider that a Python developer should know about these nearly infinite possibilities, because it instills confidence
that no impassable problem will be on the way. However, knowing how and particularly when not to use them is very
important.
Like a kung fu master, a Pythonista knows how to kill with a single finger, and never to actually do it.
We are all consenting adults
As seen above, Python allows many tricks, and some of them are potentially dangerous. A good example is that any
client code can override an objects properties and methods: there is no private keyword in Python. This philosophy,
very different from highly defensive languages like Java, which give a lot of mechanisms to prevent any misuse, is
expressed by the saying: We are all consenting adults.
This doesnt mean that, for example, no properties are considered private, and that no proper encapsulation is possible
in Python. Rather, instead of relying on concrete walls erected by the developers between their code and others,
the Python community prefers to rely on a set of conventions indicating that these elements should not be accessed
directly.
The main convention for private properties and implementation details is to prefix all internals with an underscore.
If the client code breaks this rule and accesses these marked elements, any misbehavior or problems encountered if
the code is modified is the responsibility of the client code.
Using this convention generously is encouraged: any method or property that is not intended to be used by client code
should be prefixed with an underscore. This will guarantee a better separation of duties and easier modification of
existing code; it will always be possible to publicize a private property, while privatising a public property might be a
much harder operation.
Returning values
When a function grows in complexity it is not uncommon to use multiple return statements inside the functions
body. However, in order to keep a clear intent and a sustainable readability level, it is preferable to avoid returning
meaningful values from many output points in the body.
There are two main cases for returning values in a function: the result of the function return when it has been processed
normally, and the error cases that indicate a wrong input parameter or any other reason for the function to not be able
to complete its computation or task.
If you do not wish to raise exceptions for the second case, then returning a value, such as None or False, indicating
that the function could not perform correctly might be needed. In this case, it is better to return as early as the incorrect
context has been detected. It will help to flatten the structure of the function: all the code after the return-because-oferror statement can assume the condition is met to further compute the functions main result. Having multiple such
return statements is often necessary.
However, when a function has multiple main exit points for its normal course, it becomes difficult to debug the returned
result, so it may be preferable to keep a single exit point. This will also help factoring out some code paths, and the
multiple exit points are a probable indication that such a refactoring is needed.
def complex_function(a, b, c):
if not a:
return None # Raising an
if not b:
return None # Raising an
# Some complex code trying to
# Resist temptation to return
if not x:
# Some Plan-B computation
20
exception might be better

exception might be better
compute x from a, b and c
x if succeeded
of x
return x
# One single exit point for the returned value x will help
# when maintaining the code.
2.2.2 Idioms
A programming idiom, put simply, is a way to write code. The notion of programming idioms is discussed amply at
c2 and at Stack Overflow.
Idiomatic Python code is often referred to as being Pythonic.
Although there usually is one and preferably only one obvious way to do it; the way to write idiomatic Python
code can be non-obvious to Python beginners. So, good idioms must be consciously acquired.
Some common Python idioms follow:
Unpacking
If you know the length of a list or tuple, you can assign names to its elements with unpacking. For example, since
enumerate() will provide a tuple of two elements for each item in list:
for index, item in enumerate(some_list):
# do something with index and item
You can use this to swap variables as well:

a, b = b, a
Nested unpacking works too:

a, (b, c) = 1, (2, 3)
In Python 3, a new method of extended unpacking was introduced by PEP 3132:

a, *rest = [1, 2, 3]
# a = 1, rest = [2, 3]
a, *middle, c = [1, 2, 3, 4]
# a = 1, middle = [2, 3], c = 4
Create an ignored variable

If you need to assign something (for instance, in Unpacking) but will not need that variable, use __:
filename = foobar.txt
basename, __, ext = filename.rpartition(.)
Note: Many Python style guides recommend the use of a single underscore _ for throwaway variables rather
than the double underscore __ recommended here. The issue is that _ is commonly used as an alias for the
gettext() function, and is also used at the interactive prompt to hold the value of the last operation. Using a
double underscore instead is just as clear and almost as convenient, and eliminates the risk of accidentally interfering
with either of these other use cases.
2.2. Code Style
21
Create a length-N list of the same thing

Use the Python list * operator:
four_nones = [None] * 4
Create a length-N list of lists

Because lists are mutable, the * operator (as above) will create a list of N references to the same list, which is not
likely what you want. Instead, use a list comprehension:
four_lists = [[] for __ in xrange(4)]
A common idiom for creating strings is to use str.join() on an empty string.

letters = [s, p, a, m]
word = .join(letters)
This will set the value of the variable word to spam. This idiom can be applied to lists and tuples.
Sometimes we need to search through a collection of things. Lets look at two options: lists and dictionaries.
Take the following code for example:
d = {s: [], p: [], a: [], m: []}
l = [s, p, a, m]
def lookup_dict(d):
return s in d
def lookup_list(l):
return s in l
Even though both functions look identical, because lookup_dict is utilizing the fact that dictionaries in Python are
hashtables, the lookup performance between the two is very different. Python will have to go through each item in the
list to find a matching case, which is time consuming. By analysing the hash of the dictionary, finding keys in the dict
can be done very quickly. For more information see this StackOverflow page.
2.2.3 Zen of Python

Also known as PEP 20, the guiding principles for Pythons design.
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases arent special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
22
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless youre Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, its a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- lets do more of those!
For some examples of good Python style, see this Stack Overflow question or these slides from a Python user group.
2.2.4 PEP 8
PEP 8 is the de-facto code style guide for Python.
Conforming your Python code to PEP 8 is generally a good idea and helps make code more consistent when working
on projects with other developers. There is a command-line program, pep8, that can check your code for conformance.
Install it by running the following command in your Terminal:
$ pip install pep8
Then run it on a file or series of files to get a report of any violations.

$ pep8 optparse.py
optparse.py:69:11: E401 multiple imports on one line
optparse.py:77:1: E302 expected 2 blank lines, found 1
optparse.py:88:5: E301 expected 1 blank line, found 0
optparse.py:222:34: W602 deprecated form of raising exception
optparse.py:347:31: E211 whitespace before (
optparse.py:357:17: E201 whitespace after {
optparse.py:472:29: E221 multiple spaces before operator
optparse.py:544:21: W601 .has_key() is deprecated, use in
2.2.5 Conventions
Here are some conventions you should follow to make your code easier to read.
Check if variable equals a constant
You dont need to explicitly compare a value to True, or None, or 0 - you can just add it to the if statement. See Truth
Value Testing for a list of what is considered false.
Bad:
if attr == True:
print True!
if attr == None:
print attr is None!
Good:
# Just check the value
if attr:
print attr is truthy!
# or check for the opposite
2.2. Code Style
23
if not attr:
print attr is falsey!
# or, since None is considered false, explicitly check for it
if attr is None:
print attr is None!
Access a Dictionary Element

Dont use the dict.has_key() method.
dict.get().
Instead, use x in d syntax, or pass a default argument to
Bad:
d = {hello: world}
if d.has_key(hello):
print d[hello]
# prints world
else:
print default_value
Good:
d = {hello: world}
print d.get(hello, default_value) # prints world
print d.get(thingy, default_value) # prints default_value
# Or:
if hello in d:
print d[hello]
Short Ways to Manipulate Lists

List comprehensions provide a powerful, concise way to work with lists. Also, the map() and filter() functions
can perform operations on lists using a different, more concise syntax.
Bad:
# Filter elements greater than 4
a = [3, 4, 5]
b = []
for i in a:
if i > 4:
b.append(i)
Good:
a
b
#
b
= [3, 4, 5]
= [i for i in a if i > 4]
Or:
= filter(lambda x: x > 4, a)
Bad:
# Add three to all list members.
a = [3, 4, 5]
for i in range(len(a)):
a[i] += 3
24
Good:
a
a
#
a
= [3, 4, 5]
= [i + 3 for i in a]
Or:
= map(lambda i: i + 3, a)
Use enumerate() keep a count of your place in the list.

a = [3, 4, 5]
for i, item in enumerate(a):
print i, item
# prints
# 0 3
# 1 4
# 2 5
The enumerate() function has better readability than handling a counter manually. Moreover, it is better optimized
for iterators.
Read From a File
Use the with open syntax to read from files. This will automatically close files for you.
Bad:
f = open(file.txt)
a = f.read()
print a
f.close()
Good:
with open(file.txt) as f:
for line in f:
print line
The with statement is better because it will ensure you always close the file, even if an exception is raised inside the
with block.
Line Continuations
When a logical line of code is longer than the accepted limit, you need to split it over multiple physical lines. The
Python interpreter will join consecutive lines if the last character of the line is a backslash. This is helpful in some
cases, but should usually be avoided because of its fragility: a white space added to the end of the line, after the
backslash, will break the code and may have unexpected results.
A better solution is to use parentheses around your elements. Left with an unclosed parenthesis on an end-of-line the
Python interpreter will join the next line until the parentheses are closed. The same behavior holds for curly and square
braces.
Bad:
my_very_big_string = """For a long time I used to go to bed early. Sometimes, \
when I had put out my candle, my eyes would close so quickly that I had not even \
time to say Im going to sleep."""
from some.deep.module.inside.a.module import a_nice_function, another_nice_function, \
yet_another_nice_function
2.2. Code Style
25
Good:
my_very_big_string = (
"For a long time I used to go to bed early. Sometimes, "
"when I had put out my candle, my eyes would close so quickly "
"that I had not even time to say Im going to sleep."
)
from some.deep.module.inside.a.module import (
a_nice_function, another_nice_function, yet_another_nice_function)
However, more often than not having to split long logical line is a sign that you are trying to do too many things at the
same time, which may hinder readability.
2.3 Reading Great Code

One of the core tenants behind the design of Python is creating readable code. The motivation behind this design is
simple: The number one thing that Python programmers do is read code.
One of the secrets of becoming a great Python programmer is to read, understand, and comprehend excellent code.
Excellent code typically follows the guidelines outlined in Code Style, and does its best to express a clear and concise
intent to the reader.
Included below is a list of recommended Python projects for reading. Each one of these projects is a paragon of Python
coding.
Howdoi Howdoi is a code search tool, written in Python.
Flask Flask is a microframework for Python based on Werkzeug and Jinja2. Its intended for getting started very
quickly and was developed with best intentions in mind.
Werkzeug Werkzeug started as simple collection of various utilities for WSGI applications and has become one
of the most advanced WSGI utility modules. It includes a powerful debugger, full-featured request and response
objects, HTTP utilities to handle entity tags, cache control headers, HTTP dates, cookie handling, file uploads,
a powerful URL routing system and a bunch of community-contributed addon modules.
Requests Requests is an Apache2 Licensed HTTP library, written in Python, for human beings.
Tablib Tablib is a format-agnostic tabular dataset library, written in Python.
Todo
Embed and explain YouTube video showing python code reading: http://www.youtube.com/watch?v=Jc8M9LoEuo This may require installing a Sphinx plugin.
https://bitbucket.org/birkenfeld/sphinxcontrib/src/a09f29fc16970f34350ca36ac7f229e00b1b1674/youtube?at=default
Todo
Include code examples of exemplary code from each of the projects listed. Explain why it is excellent code. Use
complex examples.
Todo
Explain techniques to rapidly identify data structures, algorithms and determine what the code is doing.
26
2.4 Documentation
Readability is a primary focus for Python developers, in both project and code documentation. Following some simple
best practices can save both you and others a lot of time.
2.4.1 Project Documentation

A README file at the root directory should give general information to both users and maintainers of a project. It
should be raw text or written in some very easy to read markup, such as reStructuredText or Markdown. It should
contain a few lines explaining the purpose of the project or library (without assuming the user knows anything about
the project), the url of the main source for the software, and some basic credit information. This file is the main entry
point for readers of the code.
An INSTALL file is less necessary with Python. The installation instructions are often reduced to one command, such
as pip install module or python setup.py install and added to the README file.
A LICENSE file should always be present and specify the license under which the software is made available to the
public.
A TODO file or a TODO section in README should list the planned development for the code.
A CHANGELOG file or section in README should compile a short overview of the changes in the code base for the
latest versions.
2.4.2 Project Publication

Depending on the project, your documentation might include some or all of the following components:
An introduction should show a very short overview of what can be done with the product, using one or two
extremely simplified use cases. This is the thirty-second pitch for your project.
A tutorial should show some primary use cases in more detail. The reader will follow a step-by-step procedure
to set-up a working prototype.
An API reference is typically generated from the code (see docstrings). It will list all publicly available interfaces, parameters, and return values.
Developer documentation is intended for potential contributors. This can include code convention and general
design strategy of the project.
Sphinx
Sphinx is far and away the most popular Python documentation tool. Use it. It converts reStructuredText markup
language into a range of output formats including HTML, LaTeX (for printable PDF versions), manual pages, and
plain text.
There is also great, free hosting for your Sphinx docs: Read The Docs. Use it. You can configure it with commit
hooks to your source repository so that rebuilding your documentation will happen automatically.
Note: Sphinx is famous for its API generation, but it also works well for general project documentation. This Guide
is built with Sphinx and is hosted on Read The Docs
2.4. Documentation
27
reStructuredText
Most Python documentation is written with reStructuredText. Its like Markdown with all the optional extensions built
in.
The reStructuredText Primer and the reStructuredText Quick Reference should help you familiarize yourself with its
syntax.
2.4.3 Code Documentation Advice

Comments clarify the code and they are added with purpose of making the code easier to understand. In Python,
comments begin with a hash (number sign) (#). In Python, docstrings describe modules, classes, and functions:
def square_and_rooter(x):
"""Returns the square root of self times self."""
...
In general, follow the comment section of PEP 8 (the Python Style Guide).
Commenting Sections of Code
Do not use triple-quote strings to comment code. This is not a good practice, because line-oriented command-line tools
such as grep will not be aware that the commented code is inactive. It is better to add hashes at the proper indentation
level for every commented line. Your editor probably has the ability to do this easily, and it is worth learning the
comment/uncomment toggle.
Docstrings and Magic
Some tools use docstrings to embed more-than-documentation behavior, such as unit test logic. Those can be nice, but
you wont ever go wrong with vanilla heres what this does.
Docstrings versus Block comments
These arent interchangeable. For a function or class, the leading comment block is a programmers note. The
docstring describes the operation of the function or class:
# This function slows down program execution for some reason.
def square_and_rooter(x):
"""Returns the square root of self times self."""
...
See also:
Further reading on docstrings: PEP 257
2.4.4 Other Tools

You might see these in the wild. Use Sphinx.
Pycco Pycco is a literate-programming-style documentation generator and is a port of the node.js Docco. It makes
code into a side-by-side HTML code and documentation.
Ronn Ronn builds unix manuals. It converts human readable textfiles to roff for terminal display, and also to HTML
for the web.
28
Epydoc Epydoc is discontinued. Use Sphinx instead.

MkDocs MkDocs is a fast and simple static site generator thats geared towards building project documentation with
Markdown.
2.5 Testing Your Code

Testing your code is very important.
Getting used to writing the testing code and the running code in parallel is now considered a good habit. Used wisely,
this method helps you define more precisely your codes intent and have a more decoupled architecture.
Some general rules of testing:
A testing unit should focus on one tiny bit of functionality and prove it correct.
Each test unit must be fully independent. Each of them must be able to run alone, and also within the test suite,
regardless of the order they are called. The implication of this rule is that each test must be loaded with a fresh
dataset and may have to do some cleanup afterwards. This is usually handled by setUp() and tearDown()
methods.
Try hard to make tests that run fast. If one single test needs more than a few millisecond to run, development
will be slowed down or the tests will not be run as often as desirable. In some cases, tests cant be fast because
they need a complex data structure to work on, and this data structure must be loaded every time the test runs.
Keep these heavier tests in a separate test suite that is run by some scheduled task, and run all other tests as often
as needed.
Learn your tools and learn how to run a single test or a test case. Then, when developing a function inside a
module, run this functions tests very often, ideally automatically when you save the code.
Always run the full test suite before a coding session, and run it again after. This will give you more confidence
that you did not break anything in the rest of the code.
It is a good idea to implement a hook that runs all tests before pushing code to a shared repository.
If you are in the middle of a development session and have to interrupt your work, it is a good idea to write a
broken unit test about what you want to develop next. When coming back to work, you will have a pointer to
where you were and get back on track faster.
The first step when you are debugging your code is to write a new test pinpointing the bug. While it is not
always possible to do, those bug catching tests are among the most valuable pieces of code in your project.
Use long and descriptive names for testing functions. The style guide here is slightly different than that of
running code, where short names are often preferred. The reason is testing functions are never called explicitly. square() or even sqr() is ok in running code, but in testing code you would have names such as
test_square_of_number_2(), test_square_negative_number(). These function names are
displayed when a test fails, and should be as descriptive as possible.
When something goes wrong or has to be changed, and if your code has a good set of tests, you or other
maintainers will rely largely on the testing suite to fix the problem or modify a given behavior. Therefore the
testing code will be read as much as or even more than the running code. A unit test whose purpose is unclear
is not very helpful is this case.
Another use of the testing code is as an introduction to new developers. When someone will have to work on
the code base, running and reading the related testing code is often the best they can do. They will or should
discover the hot spots, where most difficulties arise, and the corner cases. If they have to add some functionality,
the first step should be to add a test and, by this means, ensure the new functionality is not already a working
path that has not been plugged into the interface.
2.5. Testing Your Code
29
2.5.1 The Basics

Unittest
unittest is the batteries-included test module in the Python standard library. Its API will be familiar to anyone
who has used any of the JUnit/nUnit/CppUnit series of tools.
Creating test cases is accomplished by subclassing unittest.TestCase.
import unittest
def fun(x):
return x + 1
class MyTest(unittest.TestCase):
def test(self):
self.assertEqual(fun(3), 4)
As of Python 2.7 unittest also includes its own test discovery mechanisms.
unittest in the standard library documentation
Doctest
The doctest module searches for pieces of text that look like interactive Python sessions in docstrings, and then
executes those sessions to verify that they work exactly as shown.
Doctests have a different use case than proper unit tests: they are usually less detailed and dont catch special cases or
obscure regression bugs. They are useful as an expressive documentation of the main use cases of a module and its
components. However, doctests should run automatically each time the full test suite runs.
A simple doctest in a function:
def square(x):
"""Squares x.
>>> square(2)
4
>>> square(-2)
4
"""
return x * x
if __name__ == __main__:
import doctest
doctest.testmod()
When running this module from the command line as in python module.py, the doctests will run and complain
if anything is not behaving as described in the docstrings.
2.5.2 Tools
py.test
py.test is a no-boilerplate alternative to Pythons standard unittest module.
30
$ pip install pytest
Despite being a fully-featured and extensible test tool, it boasts a simple syntax. Creating a test suite is as easy as
writing a module with a couple of functions:
# content of test_sample.py
def func(x):
return x + 1
def test_answer():
assert func(3) == 5
and then running the py.test command

$ py.test
=========================== test session starts ============================
platform darwin -- Python 2.7.1 -- pytest-2.2.1
collecting ... collected 1 items
test_sample.py F
================================= FAILURES =================================
_______________________________ test_answer ________________________________
>
E
E
def test_answer():
assert func(3) == 5
assert 4 == 5
+ where 4 = func(3)
test_sample.py:5: AssertionError
========================= 1 failed in 0.02 seconds =========================
is far less work than would be required for the equivalent functionality with the unittest module!
py.test
Nose
nose extends unittest to make testing easier.
$ pip install nose
nose provides automatic test discovery to save you the hassle of manually creating test suites. It also provides numerous
plugins for features such as xUnit-compatible test output, coverage reporting, and test selection.
nose
tox
tox is a tool for automating test environment management and testing against multiple interpreter configurations
$ pip install tox
tox allows you to configure complicated multi-parameter test matrices via a simple ini-style configuration file.
tox
2.5. Testing Your Code
31
Unittest2
unittest2 is a backport of Python 2.7s unittest module which has an improved API and better assertions over the one
available in previous versions of Python.
If youre using Python 2.6 or below, you can install it with pip
$ pip install unittest2
You may want to import the module under the name unittest to make porting code to newer versions of the module
easier in the future
import unittest2 as unittest
class MyTest(unittest.TestCase):
...
This way if you ever switch to a newer Python version and no longer need the unittest2 module, you can simply change
the import in your test module without the need to change any other code.
unittest2
mock
unittest.mock is a library for testing in Python. As of Python 3.3, it is available in the standard library.
For older versions of Python:
$ pip install mock
It allows you to replace parts of your system under test with mock objects and make assertions about how they have
been used.
For example, you can monkey-patch a method:
from mock import MagicMock
thing = ProductionClass()
thing.method = MagicMock(return_value=3)
thing.method(3, 4, 5, key=value)
thing.method.assert_called_with(3, 4, 5, key=value)
To mock classes or objects in a module under test, use the patch decorator. In the example below, an external search
system is replaced with a mock that always returns the same result (but only for the duration of the test).
def mock_search(self):
class MockSearchQuerySet(SearchQuerySet):
def __iter__(self):
return iter(["foo", "bar", "baz"])
return MockSearchQuerySet()
# SearchForm here refers to the imported class reference in myapp,
# not where the SearchForm class itself is imported from
@mock.patch(myapp.SearchForm.search, mock_search)
def test_new_watchlist_activities(self):
# get_search_results runs a search and iterates over the result
self.assertEqual(len(myapp.get_search_results(q="fish")), 3)
Mock has many other ways you can configure it and control its behavior.
mock
32
2.6 Common Gotchas

For the most part, Python aims to be a clean and consistent language that avoids surprises. However, there are a few
cases that can be confusing to newcomers.
Some of these cases are intentional but can be potentially surprising. Some could arguably be considered language
warts. In general though, what follows is a collection of potentially tricky behavior that might seem strange at first
glance, but is generally sensible once youre aware of the underlying cause for the surprise.
2.6.1 Mutable Default Arguments

Seemingly the most common surprise new Python programmers encounter is Pythons treatment of mutable default
arguments in function definitions.
What You Wrote
def append_to(element, to=[]):
to.append(element)
return to
What You Might Have Expected to Happen

my_list = append_to(12)
print my_list
my_other_list = append_to(42)
print my_other_list
A new list is created each time the function is called if a second argument isnt provided, so that the output is:
[12]
[42]
What Does Happen

[12]
[12, 42]
A new list is created once when the function is defined, and the same list is used in each successive call.
Pythons default arguments are evaluated once when the function is defined, not each time the function is called (like
it is in say, Ruby). This means that if you use a mutable default argument and mutate it, you will and have mutated
that object for all future calls to the function as well.
What You Should Do Instead
Create a new object each time the function is called, by using a default arg to signal that no argument was provided
(None is often a good choice).
2.6. Common Gotchas
33
def append_to(element, to=None):

if to is None:
to = []
to.append(element)
return to
When the Gotcha Isnt a Gotcha

Sometimes you can specifically exploit (read: use as intended) this behavior to maintain state between calls of a
function. This is often done when writing a caching function.
2.6.2 Late Binding Closures

Another common source of confusion is the way Python binds its variables in closures (or in the surrounding global
scope).
What You Wrote
def create_multipliers():
return [lambda x : i * x for i in range(5)]
What You Might Have Expected to Happen

for multiplier in create_multipliers():
print multiplier(2)
A list containing five functions that each have their own closed-over i variable that multiplies their argument, producing:
0
2
4
6
8
What Does Happen

8
8
8
8
8
Five functions are created; instead all of them just multiply x by 4.

Pythons closures are late binding. This means that the values of variables used in closures are looked up at the time
the inner function is called.
Here, whenever any of the returned functions are called, the value of i is looked up in the surrounding scope at call
time. By then, the loop has completed and i is left with its final value of 4.
34
Whats particularly nasty about this gotcha is the seemingly prevalent misinformation that this has something to do
with lambdas in Python. Functions created with a lambda expression are in no way special, and in fact the same
exact behavior is exhibited by just using an ordinary def:
multipliers = []
for i in range(5):
def multiplier(x):
return i * x
multipliers.append(multiplier)
return multipliers
What You Should Do Instead

The most general solution is arguably a bit of a hack. Due to Pythons aforementioned behavior concerning evaluating
default arguments to functions (see Mutable Default Arguments), you can create a closure that binds immediately to
its arguments by using a default arg like so:
return [lambda x, i=i : i * x for i in range(5)]
Alternatively, you can use the functools.partial function:

from functools import partial
from operator import mul
return [partial(mul, i) for i in range(5)]
When the Gotcha Isnt a Gotcha

Sometimes you want your closures to behave this way. Late binding is good in lots of situations. Looping to create
unique functions is unfortunately a case where they can cause hiccups.
2.7 Choosing a License

Your source publication needs a license. In the US, if no license is specified, users have no legal right to download,
modify, or distribute. Furthermore, people cant contribute to your code unless you tell them what rules to play by.
Choosing a license is complicated, so here are some pointers:
Open source. There are plenty of open source licenses available to choose from.
In general, these licenses tend to fall into one of two categories:
1. licenses that focus more on the users freedom to do with the software as they please (these are the more
permissive open source licenses such as the MIT, BSD, & Apache).
2. licenses that focus more on making sure that the code itself including any changes made to it and distributed
along with it always remains free (these are the less permissive free software licenses such as the GPL and
LGPL).
The latter are less permissive in the sense that they dont permit someone to add code to the software and distribute it
without also including the source code for their changes.
2.7. Choosing a License
35
To help you choose one for your project, theres a license chooser, use it.
More Permissive
PSFL (Python Software Foundation License) for contributing to Python itself
MIT / BSD / ISC
MIT (X11)
New BSD
ISC
Apache
Less Permissive:
LGPL
GPL
GPLv2
GPLv3
A good overview of licenses with explanations of what one can, cannot, and must do using a particular software can
be found at tl;drLegal.
36
CHAPTER 3
Scenario Guide
This part of the guide focuses on tool and module advice based on different scenarios.
3.1 Network Applications

3.1.1 HTTP
The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web.
Requests
Pythons standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken.
It was built for a different time and a different web. It requires an enormous amount of work (even method overrides)
to perform the simplest of tasks.
Requests takes all of the work out of Python HTTP making your integration with web services seamless. Theres
no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP
connection pooling are 100% automatic, powered by urllib3, which is embedded within Requests.
Documentation
PyPi
GitHub
3.1.2 Distributed Systems

ZeroMQ
MQ (also spelled ZeroMQ, 0MQ or ZMQ) is a high-performance asynchronous messaging library aimed at use in
scalable distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware,
a MQ system can run without a dedicated message broker. The library is designed to have a familiar socket-style
API.
37
RabbitMQ
RabbitMQ is an open source message broker software that implements the Advanced Message Queuing Protocol
(AMQP). The RabbitMQ server is written in the Erlang programming language and is built on the Open Telecom
Platform framework for clustering and failover. Client libraries to interface with the broker are available for all major
programming languages.
Homepage
GitHub Organization
3.2 Web Applications

As a powerful scripting language adapted to both fast prototyping and bigger projects, Python is widely used in Web
applications development.
3.2.1 Context
WSGI
The Web Server Gateway Interface (or WSGI for short) is a standard interface between web servers and Python
web application frameworks. By standardizing behavior and communication between web servers and Python web
frameworks, WSGI makes it possible to write portable Python web code that can be deployed in any WSGI-compliant
web server. WSGI is documented in PEP 3333.
3.2.2 Frameworks
Broadly speaking, a web framework consists of a set of libraries and a main handler within which you can build custom
code to implement a web application (i.e. an interactive web site). Most web frameworks include patterns and utilities
to accomplish at least the following:
URL Routing Matches an incoming HTTP request to a particular piece of Python code to be invoked
Request and Response Objects Encapsulate the information received from or sent to a users browser
Template Engine Allows for separating Python code implementing an applications logic from the HTML (or other)
output that it produces
Development Web Server Runs an HTTP server on development machines to enable rapid development; often automatically reloads server-side code when files are updated
Django
Django is a batteries included web application framework. By providing many utilities and patterns out of the box,
Django aims to make it possible to build complex, database-backed web applications quickly, while encouraging best
practices in code written using it.
Django has a large and active community, and many pre-built re-usable modules that can be incorporated into a new
project as-is, or customized to fit your needs.
There are annual Django conferences in the United States and in Europe.
38
Chapter 3. Scenario Guide
Flask
Flask is a microframework for Python. Rather than aiming to provide everything you could possibly need, Flask
implements the most commonly-used core components of a web application framework, like URL routing, request and
response objects, and templates. As a user of Flask, it is therefore up to you to choose and integrate other components
you may need, such as database access or form generation and validation. For many popular modules, Extensions may
already exist to suit your needs.
Support for flask can best be found in its mailing list. Just shoot an email to flask@librelist.com and reply to the
confirmation email.
Werkzeug
Werkzeug is not actually a real framework, but rather a very powerful set of tools for building web applications. It
provides URL routing utilities, request and response objects and a basic development server. It is mostly used where
users need bigger flexibility for their application that is not commonly found in other web frameworks.
Support can be found on its mailing list.
Tornado
Tornado is a scalable, non-blocking web server and web application framework with a relative simple usage. Tornado
is known for its high performance. It was initially developed for friendfeed , a real time chat and blog system.
In the Jinja2 template engine example it is used to serve the rendered pages.
Pyramid
Pyramid lies somewhere between a big framework like Django and the microframeworks: It comes with a lot of
libraries and functionality and can thus not be considered lightweight. On the other hand, it does not provide all the
functionality Django does. Instead Pyramid brings basic support for most regular tasks and provides a great deal of
extensibility. Additionally, Pyramid has a huge focus on complete documentation. As a little extra it comes with the
Werkzeug Debugger which allows you to debug a running web application in the browser.
Support can also be found in the documentation.
3.2.3 Web Servers

Nginx
Nginx (pronounced engine-x) is a web server and reverse-proxy for HTTP, SMTP and other protocols. It is known
for its high performance, relative simplicity, and compatibility with many application servers (like WSGI servers).
It also includes handy features like load-balancing, basic authentication, streaming, and others. Designed to serve
high-load websites, Nginx is gradually becoming quite popular.
3.2.4 WSGI Servers

Stand-alone WSGI servers typically use less resources than traditional web servers and provide top performance 1 .
1
Benchmark of Python WSGI Servers
3.2. Web Applications
39
Gunicorn
Gunicorn (Green Unicorn) is a WSGI server used to serve Python applications. It is a Python interpretation of the
Ruby Unicorn server. Unicorn is designed to be lightweight, easy to use, and uses many UNIX idioms. Gunicorn is
not designed to face the internet it was designed to run behind Nginx which buffers slow requests and takes care of
other important considerations. A sample setup for Nginx + Gunicorn can be found in the Gunicorn help.
uWSGI
uWSGI is a full stack for building hosting services. In addition to process management, process monitoring, and
other functionality, uWSGI acts as an application server for various programming languages and protocols - including
Python and WSGI. uWSGI can either be run as a stand-alone web router, or be run behind a full web server (such as
Nginx or Apache). In the latter case, a web server can configure uWSGI and an applications operation over the uwsgi
protocol. uWSGIs web server support allows for dynamically configuring Python, passing environment variables and
further tuning. For full details, see uWSGI magic variables.
3.2.5 Server Best Practices

The majority of self hosted Python applications today are hosted with a WSGI server such as Gunicorn, either directly
or behind a lightweight web server such as nginx.
The WSGI servers serve the Python applications while the web server handles tasks better suited for it such as static
file serving, request routing, DDoS protection, and basic authentication.
3.2.6 Hosting
Platform-as-a-Service
Platform-as-a-Service (PaaS) is a type of cloud computing infrastructure which abstracts and manages infrastructure,
routing, and scaling of web applications. When using PaaS, application developers can focus on writing application
code rather than needing to be concerned with deployment details.
Most PaaS services offer a command-line interface that developers can use to set up and interrogate configuration, and
to deploy new releases of an application to the service.
PaaS services and their partners offer add-on functionality which is well integrated into the platform, such as database
hosting, email services, logging, scheduled and background tasks, billing and payment, etc.
Heroku
Herokus Cedar stack offers first class support for Python 2.7 applications.
Heroku allows you to run as many Python web applications as you like, 24/7 and free of charge. Heroku is best
described as a horizontal scaling platform. They start to charge you once you scale your application to run on more
than one Dyno (abstracted servers) at a time.
Heroku maintains articles on using Python with Heroku as well as step-by-step instructions on how to set up your first
application.
40
DotCloud
DotCloud supports WSGI applications and background/worker tasks natively on their platform. Web applications run
Python version 2.6, use nginx and uWSGI, and allow custom configuration of both for advanced users.
DotCloud uses a custom command-line API client which can work with applications managed in git repositories or
any other version control system.
DotCloud has a free plan with limited database size, and without extra services (caching. . . ).
See the DotCloud documentation on Python for more information and help getting started.
Gondor
Gondor is a PaaS specialized for deploying Django and Pinax applications. Gondor recommends Django version 1.6
and supports any WSGI application on Python version 2.7. Gondor can automatically configure your Django site if
you use local_settings.py for site-specific configuration information.
Gondor has a guide on deploying Django projects.
3.2.7 Templating
Most WSGI applications are responding to HTTP requests to serve content in HTML or other markup languages.
Instead of generating directly textual content from Python, the concept of separation of concerns advises us to use
templates. A template engine manages a suite of template files, with a system of hierarchy and inclusion to avoid
unnecessary repetition, and is in charge of rendering (generating) the actual content, filling the static content of the
templates with the dynamic content generated by the application.
As template files are sometimes written by designers or front-end developers, it can be difficult to handle increasing
complexity.
Some general good practices apply to the part of the application passing dynamic content to the template engine, and
to the templates themselves.
Template files should be passed only the dynamic content that is needed for rendering the template. Avoid the
temptation to pass additional content just in case: it is easier to add some missing variable when needed than
to remove a likely unused variable later.
Many template engines allow for complex statements or assignments in the template itself, and many allow some
Python code to be evaluated in the templates. This convenience can lead to uncontrolled increase in complexity,
and often make it harder to find bugs.
It is often necessary to mix JavaScript templates with HTML templates. A sane approach to this design is to
isolate the parts where the HTML template passes some variable content to the JavaScript code.
Jinja2
Jinja2 is a template engine which is similar to the Django template system with some extra features. It is a text-based
template language and thus can be used to generate any markup. It allows customization of filters, tags, tests and
globals, and unlike the template system implemented in the Django Framework, also allows calling functions. Jinja2
is released under the BSD license.
Here some important html tags in Jinja2:
3.2. Web Applications
41
{# This is a comment #}
{# The next tag is a variable output: #}
{{title}}
{# Tag for a block, can be replaced through inheritance with other html code #}
{% block head %}
<h1>This is the head!</h1>
{% endblock %}
{# Output of an array as an iteration #}
{% for item in list %}
<li>{{ item }}</li>
{% endfor %}
The next listings is an example of a web site in combination with the tornado web server. Tornado is not very
complicate to use.
# import Jinja2
from jinja2 import Environment, FileSystemLoader
# import Tornado
import tornado.ioloop
import tornado.web
# Load template file templates/site.html
TEMPLATE_FILE = "site.html"
templateLoader = FileSystemLoader( searchpath="templates/" )
templateEnv = Environment( loader=templateLoader )
template = templateEnv.get_template(TEMPLATE_FILE)
# List for famous movie rendering
movie_list = [[1,"The Hitchhikers Guide to the Galaxy"],[2,"Back to future"],[3,"Matrix"]]
# template.render() returns a string which contains the rendered html
html_output = template.render(list=movie_list,
title="Here is my favorite movie list")
# Handler for main page
class MainHandler(tornado.web.RequestHandler):
def get(self):
# Returns rendered template string to the browser request
self.write(html_output)
# Assign handler to the server root (127.0.0.1:PORT/)
application = tornado.web.Application([
(r"/", MainHandler),
])
PORT=8884
if __name__ == "__main__":
# Setup the server
application.listen(PORT)
tornado.ioloop.IOLoop.instance().start()
The base.html file can be used as base for all site pages which are for example implemented in the content block.
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<html xmlns="http://www.w3.org/1999/xhtml">
42
<head>
<link rel="stylesheet" href="style.css" />
<title>{{title}} - My Webpage</title>
</head>
<body>
<div id="content">
{# In the next line the content from the site.html template will be added #}
{% block content %}{% endblock %}
</div>
<div id="footer">
{% block footer %}
© Copyright 2013 by <a href="http://domain.invalid/">you</a>.
{% endblock %}
</div>
</body>
The next listing is our site page (site.html) loaded in the Python app which extends base.html. The content
block is automatically set into the corresponding block in the base.html page.
<!{% extends "base.html" %}
{% block content %}
<p class="important">
<div id="content">
<h2>{{title}}</h2>
<p>{{ list_title }}</p>
<ul>
{% for item in list %}
<li>{{ item[0]}} : {{ item[1]}}</li>
{% endfor %}
</ul>
</div>
</p>
{% endblock %}
References
3.3 HTML Scraping

3.3.1 Web Scraping
Web sites are written using HTML, which means that each web page is a structured document. Sometimes it would be
great to obtain some data from them and preserve the structure while were at it. Web sites dont always provide their
data in comfortable formats such as csv or json.
This is where web scraping comes in. Web scraping is the practice of using a computer program to sift through a web
page and gather the data that you need in a format most useful to you while at the same time preserving the structure
of the data.
3.3.2 lxml and Requests

lxml is a pretty extensive library written for parsing XML and HTML documents very quickly, even handling messed
up tags in the process. We will also be using the Requests module instead of the already built-in urllib2 module due to
improvements in speed and readability. You can easily install both using pip install lxml and pip install
requests.
3.3. HTML Scraping
43
Lets start with the imports:

from lxml import html
import requests
Next we will use requests.get to retrieve the web page with our data, parse it using the html module and save
the results in tree:
page = requests.get(http://econpy.pythonanywhere.com/ex/001.html)
tree = html.fromstring(page.text)
tree now contains the whole HTML file in a nice tree structure which we can go over two different ways: XPath and
CSSSelect. In this example, we will focus on the former.
XPath is a way of locating information in structured documents such as HTML or XML documents. A good introduction to XPath is on W3Schools .
There are also various tools for obtaining the XPath of elements such as FireBug for Firefox or the Chrome Inspector.
If youre using Chrome, you can right click an element, choose Inspect element, highlight the code, right click again
and choose Copy XPath.
After a quick analysis, we see that in our page the data is contained in two elements - one is a div with title buyername and the other is a span with class item-price:
<div title="buyer-name">Carson Busses</div>
<span class="item-price">$29.95</span>
Knowing this we can create the correct XPath query and use the lxml xpath function like this:
#This will create a list of buyers:
buyers = tree.xpath(//div[@title="buyer-name"]/text())
#This will create a list of prices
prices = tree.xpath(//span[@class="item-price"]/text())
Lets see what we got exactly:

print Buyers: , buyers
print Prices: , prices
Buyers: [Carson Busses, Earl E. Byrd, Patty Cakes,
Derri Anne Connecticut, Moe Dess, Leda Doggslife, Dan Druff,
Al Fresco, Ido Hoe, Howie Kisses, Len Lease, Phil Meup,
Ira Pent, Ben D. Rules, Ave Sectomy, Gary Shattire,
Bobbi Soks, Sheila Takya, Rose Tattoo, Moe Tell]
Prices: [$29.95, $8.37, $15.26, $19.25, $19.25,
$13.99, $31.57, $8.49, $14.47, $15.86, $11.11,
$15.98, $16.27, $7.50, $50.85, $14.26, $5.68,
$15.00, $114.07, $10.09]
Congratulations! We have successfully scraped all the data we wanted from a web page using lxml and Requests. We
have it stored in memory as two lists. Now we can do all sorts of cool stuff with it: we can analyze it using Python or
we can save it to a file and share it with the world.
Some more cool ideas to think about are modifying this script to iterate through the rest of the pages of this example
dataset, or rewriting this application to use threads for improved speed.
44
3.4 Command-line Applications

Command-line applications, also referred to as Console Applications, are computer programs designed to be used
from a text interface, such as a shell. Command-line applications usually accept various inputs as arguments, often
referred to as parameters or sub-commands, as well as options, often referred to as flags or switches.
Some popular command-line applications include:
Grep - A plain-text data search utility
curl - A tool for data transfer with URL syntax
httpie - A command line HTTP client, a user-friendly cURL replacement
git - A distributed version control system
mercurial - A distributed version control system primarily written in Python
3.4.1 Clint
clint is a Python module which is filled with very useful tools for developing command-line applications. It supports
features such as; CLI colors and indents, simple and powerful column printer, iterator based progress bars and implicit
argument handling.
3.4.2 Click
click is an upcoming Python package for creating command-line interfaces in a composable way with as little code as
possible. This Command-line Interface Creation Kit is highly configurable but comes with good defaults out of the
box.
3.4.3 docopt
docopt is a lightweight, highly Pythonic package that allows creating command-line interfaces easily and intuitively,
by parsing POSIX-style usage instructions.
3.4.4 Plac
Plac is a simple wrapper over the Python standard library argparse, which hides most of its complexity by using a
declarative interface: the argument parser is inferred rather than written down by imperatively. This module targets
especially unsophisticated users, programmers, sys-admins, scientists and in general people writing throw-away scripts
for themselves, who choose to create a command-line interface because it is quick and simple.
3.4.5 Cliff
Cliff is a framework for building command-line programs. It uses setuptools entry points to provide subcommands,
output formatters, and other extensions. The framework is meant to be used to create multi-level commands such as
subversion and git, where the main program handles some basic argument parsing and then invokes a sub-command
to do the work.
3.4. Command-line Applications
45
3.5 GUI Applications

Alphabetical list of GUI Applications.
3.5.1 Camelot
Camelot provides components for building applications on top of Python, SQLAlchemy and Qt. It is inspired by the
Django admin interface.
The main resource for information is the website: http://www.python-camelot.com and the mailing list
https://groups.google.com/forum/#!forum/project-camelot
3.5.2 Cocoa
Note: The Cocoa framework is only available on OS X. Dont pick this if youre writing a cross-platform application!
3.5.3 GTk
PyGTK provides Python bindings for the GTK+ toolkit. Like the GTK+ library itself, it is currently licensed under the
GNU LGPL. It is worth noting that PyGTK only currently supports the Gtk-2.X API (NOT Gtk-3.0). It is currently
recommended that PyGTK not be used for new projects and that existing applications be ported from PyGTK to
PyGObject.
3.5.4 Kivy
Kivy is a Python library for development of multi-touch enabled media rich applications. The aim is to allow for quick
and easy interaction design and rapid prototyping, while making your code reusable and deployable.
Kivy is written in Python, based on OpenGL and supports different input devices such as: Mouse, Dual Mouse, TUIO,
WiiMote, WM_TOUCH, HIDtouch, Apples products and so on.
Kivy is actively being developed by a community and free to use. It operates on all major platforms (Linux, OSX,
Windows, Android).
The main resource for information is the website: http://kivy.org
PyObjC
Note: Only available on OS X. Dont pick this if youre writing a cross-platform application.
PySide
PySide is a Python binding of the cross-platform GUI toolkit Qt.
http://developer.qt.nokia.com/wiki/PySideDownloads/
46
PyQt
Note: If your software does not fully comply with the GPL you will need a commercial license!
PyQt provides Python bindings for the Qt Framework (see below).
http://www.riverbankcomputing.co.uk/software/pyqt/download
3.5.5 PyjamasDesktop (pyjs Desktop)

PyjamasDesktop is a port of Pyjamas. PyjamasDesktop is application widget set for desktop and a cross-platform
framework. (After release v0.6 PyjamasDesktop is a part of Pyjamas (Pyjs)). Briefly, it allows the exact same Python
web application source code to be executed as a standalone desktop application.
Python Wiki for PyjamasDesktop.
The main website; pyjs Desktop.
3.5.6 Qt
Qt is a cross-platform application framework that is widely used for developing software with a GUI but can also be
used for non-GUI applications.
3.5.7 Tk
Tkinter is a thin object-oriented layer on top of Tcl/Tk. It has the advantage of being included with the Python
standard library, making it the most convenient and compatible toolkit to program with.
Both Tk and Tkinter are available on most Unix platforms, as well as on Windows and Macintosh systems. Starting
with the 8.0 release, Tk offers native look and feel on all platforms.
Theres a good multi-language Tk tutorial with Python examples at TkDocs. Theres more information available on
the Python Wiki.
3.5.8 wxPython
wxPython is a GUI toolkit for the Python programming language. It allows Python programmers to create programs
with a robust, highly functional graphical user interface, simply and easily. It is implemented as a Python extension
module (native code) that wraps the popular wxWidgets cross platform GUI library, which is written in C++.
Install (Stable) wxPython go to http://www.wxpython.org/download.php#stable and download the appropriate package for your OS.
3.6 Databases
3.6.1 DB-API
The Python Database API (DB-API) defines a standard interface for Python database access modules. Its documented
in PEP 249. Nearly all Python database modules such as sqlite3, psycopg and mysql-python conform to this interface.
Tutorials that explain how to work with modules that conform to this interface can be found here and here.
3.6. Databases
47
3.6.2 SQLAlchemy
SQLAlchemy is a commonly used database toolkit. Unlike many database libraries it not only provides an ORM layer
but also a generalized API for writing database-agnostic code without SQL.
$ pip install sqlalchemy
3.6.3 Django ORM

The Django ORM is the interface used by Django to provide database access.
Its based on the idea of models, an abstraction that makes it easier to manipulate data in Python.
The basics:
Each model is a Python class that subclasses django.db.models.Model.
Each attribute of the model represents a database field.
Django gives you an automatically-generated database-access API; see Making queries.
3.7 Networking
3.7.1 Twisted
Twisted is an event-driven networking engine. It can be used to build applications around many different networking protocols, including http servers and clients, applications using SMTP, POP3, IMAP or SSH protocols, instant
messaging and much more.
3.7.2 PyZMQ
PyZMQ is the Python binding for ZeroMQ, which is a high-performance asynchronous messaging library. One great
advantage of ZeroMQ is that it can be used for message queuing without a message broker. The basic patterns for this
are:
request-reply: connects a set of clients to a set of services. This is a remote procedure call and task distribution
pattern.
publish-subscribe: connects a set of publishers to a set of subscribers. This is a data distribution pattern.
push-pull (or pipeline): connects nodes in a fan-out / fan-in pattern that can have multiple steps, and loops. This
is a parallel task distribution and collection pattern.
For a quick start, read the ZeroMQ guide.
3.7.3 gevent
gevent is a coroutine-based Python networking library that uses greenlets to provide a high-level synchronous API on
top of the libev event loop.
48
3.8 Systems Administration

3.8.1 Fabric
Fabric is a library for simplifying system administration tasks. While Chef and Puppet tend to focus on managing
servers and system libraries, Fabric is more focused on application level tasks such as deployment.
Install Fabric:
$ pip install fabric
The following code will create two tasks that we can use: memory_usage and deploy. The former will output the
memory usage on each machine. The latter will ssh into each server, cd to our project directory, activate the virtual
environment, pull the newest codebase, and restart the application server.
from fabric.api import cd, env, prefix, run, task
env.hosts = [my_server1, my_server2]
@task
def memory_usage():
run(free -m)
@task
def deploy():
with cd(/var/www/project-env/project):
with prefix(. ../bin/activate):
run(git pull)
run(touch app.wsgi)
With the previous code saved in a file named fabfile.py, we can check memory usage with:
$ fab memory_usage
[my_server1] Executing task memory
[my_server1] run: free -m
[my_server1] out:
total
[my_server1] out: Mem:
6964
[my_server1] out: -/+ buffers/cache:
[my_server1] out: Swap:
0
used
1897
1509
0
free
5067
5455
0
shared
0
buffers
166
cached
222
[my_server2]
[my_server2]
[my_server2]
[my_server2]
[my_server2]
[my_server2]
used
902
148
1
free
764
1517
894
shared
0
buffers
180
cached
572
Executing task memory

run: free -m
out:
total
out: Mem:
1666
out: -/+ buffers/cache:
out: Swap:
895
and we can deploy with:

$ fab deploy
Additional features include parallel execution, interaction with remote programs, and host grouping.
Fabric Documentation
3.8.2 Salt
Salt is an open source infrastructure management tool. It supports remote command execution from a central point
3.8. Systems Administration
49
(master host) to multiple hosts (minions). It also supports system states which can be used to configure multiple servers
using simple template files.
Salt supports Python versions 2.6 and 2.7 and can be installed via pip:
$ pip install salt
After configuring a master server and any number of minion hosts, we can run arbitrary shell commands or use prebuilt modules of complex commands on our minions.
The following command lists all available minion hosts, using the ping module.
$ salt * test.ping
The host filtering is accomplished by matching the minion id, or using the grains system. The grains system uses static
host information like the operating system version or the CPU architecture to provide a host taxonomy for the Salt
modules.
The following command lists all available minions running CentOS using the grains system:
$ salt -G os:CentOS test.ping
Salt also provides a state system. States can be used to configure the minion hosts.
For example, when a minion host is ordered to read the following state file, it will install and start the Apache server:
apache:
pkg:
- installed
service:
- running
- enable: True
- require:
- pkg: apache
State files can be written using YAML, the Jinja2 template system or pure Python.
Salt Documentation
3.8.3 Psutil
Psutil is an interface to different system information (e.g. CPU, memory, disks, network, users and processes).
Here is an example to be aware of some server overload. If any of the tests (net, CPU) fail, it will send an email.
# Functions to get system values:
from psutil import cpu_percent, net_io_counters
# Functions to take a break:
from time import sleep
# Package for email services:
import smtplib
import string
MAX_NET_USAGE = 400000
MAX_ATTACKS = 4
attack = 0
counter = 0
while attack <= MAX_ATTACKS:
sleep(4)
counter = counter + 1
# Check the cpu usage
50
if cpu_percent(interval = 1) > 70:

attack = attack + 1
# Check the net usage
neti1 = net_io_counters()[1]
neto1 = net_io_counters()[0]
sleep(1)
neti2 = net_io_counters()[1]
neto2 = net_io_counters()[0]
# Calculate the bytes per second
net = ((neti2+neto2) - (neti1+neto1))/2
if net > MAX_NET_USAGE:
attack = attack + 1
if counter > 25:
attack = 0
counter = 0
# Write a very important email if attack is higher than 4
TO = "you@your_email.com"
FROM = "webmaster@your_domain.com"
SUBJECT = "Your domain is out of system resources!"
text = "Go and fix your server!"
BODY = string.join(("From: %s" %FROM,"To: %s" %TO,"Subject: %s" %SUBJECT, "",text), "\r\n")
server = smtplib.SMTP(127.0.0.1)
server.sendmail(FROM, [TO], BODY)
server.quit()
A full terminal application like a widely extended top which is based on psutil and with the ability of a client-server
monitoring is glance.
3.8.4 Ansible
Ansible is an open source system automation tool. The biggest advantage over Puppet or Chef is it does not require an
agent on the client machine. Playbooks are Ansibles configuration, deployment, and orchestration language and are
written in in YAML with Jinja2 for templating.
Ansible supports Python versions 2.6 and 2.7 and can be installed via pip:
$ pip install ansible
Ansible requires an inventory file that describes the hosts to which it has access. Below is an example of a host and
playbook that will ping all the hosts in the inventory file.
Here is an example inventory file: hosts.yml
[server_name]
127.0.0.1
Here is an example playbook: ping.yml

--- hosts: all
tasks:
- name: ping
action: ping
To run the playbook:

$ ansible-playbook ping.yml -i hosts.yml --ask-pass
3.8. Systems Administration
51
The Ansible playbook will ping all of the servers in the hosts.yml file. You can also select groups of servers using
Ansible. For more information about Ansible, read the Ansible Docs.
3.8.5 Chef
Todo
Write about Chef
Chef Documentation
3.8.6 Puppet
Puppet is an IT Automation and configuration management software from Puppet Labs that allows System Administrators to define the state of their IT Infrastructure, thereby providing an elegant way to manage their fleet of physical
and virtual machines.
Puppet is available both as an Open Source and an Enterprise variant. Modules are small,shareable units of code
written to automate or define the state of a system. Puppet Forge is a repository for modules written by the community
for Open Source and Enterprise Puppet.
Puppet Agents are installed on nodes whose state needs to be monitored or changed. A desginated server known as
the Puppet Master is responsible for orchastrating the agent nodes.
Agent nodes send basic facts about the system such as to the operating system, kernel, architecture, ip address, hostname etc. to the Puppet Master. The Puppet Master then compiles a catalog with information provided by the agents
on how each node should be configured and sends it to the agent. The agent enforces the change as prescribed in the
catalog and sends a report back to the Puppet Master.
Facter is an interesting tool that ships with Puppet that pulls basic facts about the System. These facts can be referenced
as a variable while writing your Puppet modules.
$ facter kernel
Linux
$ facter operatingsystem
Ubuntu
Writing Modules in Puppet is pretty straight forward. Puppet Manifests together form Puppet Modules. Puppet
manifest end with an extension of .pp. Here is an example of Hello World in Puppet.
notify { This message is getting logged into the agent node:
#As nothing is specified in the body the resource title
#the notification message by default.
}
Here is another example with system based logic. Note how the operatingsystem fact is being used as a variable
prepended with the $ sign. Similarly, this holds true for other facts such as hostname which can be referenced by
$hostname
notify{ Mac Warning:
message => $operatingsystem ? {
Darwin => This seems to be a Mac.,
default => I am a PC.,
},
}
52
There are several resource types for Puppet but the package-file-service paradigm is all you need for undertaking
majority of the configuration management. The following Puppet code makes sure that the OpenSSH-Server package
is installed in a system and the sshd service is notified to restart everytime the sshd configuration file is changed.
package { openssh-server:
ensure => installed,
}
file { /etc/ssh/sshd_config:
source
=> puppet:///modules/sshd/sshd_config,
owner
=> root,
group
=> root,
mode
=> 640,
notify
=> Service[sshd], # sshd will restart
# whenever you edit this
# file
require => Package[openssh-server],
}
service { sshd:
ensure
=> running,
enable
=> true,
hasstatus => true,
hasrestart=> true,
}
For more information checkout Puppet Labs Documentation
3.8.7 Blueprint
Todo
Write about Blueprint
3.8.8 Buildout
Todo
Write about Buildout
Buildout Website
3.9 Continuous Integration

3.9.1 Why?
Martin Fowler, who first wrote about Continuous Integration (short: CI) together with Kent Beck, describes the CI as
follows:
Continuous Integration is a software development practice where members of a team integrate their work
frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each
3.9. Continuous Integration
53
integration is verified by an automated build (including test) to detect integration errors as quickly as
possible. Many teams find that this approach leads to significantly reduced integration problems and
allows a team to develop cohesive software more rapidly.
3.9.2 Jenkins
Jenkins CI is an extensible continuous integration engine. Use it.
3.9.3 Buildbot
Buildbot is a Python system to automate the compile/test cycle to validate code changes.
3.9.4 Mule
Mule is a lightweight integration platform that enables you to connect anything, anywhere. You can use Mule to
intelligently manage message routing, data mapping, orchestration, reliability, security and scalability between nodes.
Plug other systems and applications into Mule and let it handle all the communication between systems, enabling you
to track and monitor everything that happens.
3.9.5 Tox
tox is an automation tool providing packaging, testing and deployment of Python software right from the console or
CI server. It is a generic virtualenv management and test command line tool which provides the following features:
Checking that packages install correctly with different Python versions and interpreters
Running tests in each of the environments, configuring your test tool of choice
Acting as a front-end to Continuous Integration servers, reducing boilerplate and merging CI and shell-based
testing.
3.9.6 Travis-CI
Travis-CI is a distributed CI server which builds tests for open source projects for free. It provides multiple workers
to run Python tests on and seamlessly integrates with GitHub. You can even have it comment on your Pull Requests
whether this particular changeset breaks the build or not. So if you are hosting your code on GitHub, travis-ci is a
great and easy way to get started with Continuous Integration.
In order to get started, add a .travis.yml file to your repository with this example content:
language: python
python:
- "2.6"
- "2.7"
- "3.2"
- "3.3"
# command to install dependencies
script: python tests/test_all_of_the_units.py
branches:
only:
- master
54
This will get your project tested on all the listed Python versions by running the given script, and will only build the
master branch. There are a lot more options you can enable, like notifications, before and after steps and much more.
The travis-ci docs explain all of these options, and are very thorough.
In order to activate testing for your project, go to the travis-ci site and login with your GitHub account. Then activate
your project in your profile settings and youre ready to go. From now on, your projects tests will be run on every
push to GitHub.
3.10 Speed
CPython, the most commonly used implementation of Python, is slow for CPU bound tasks. PyPy is fast.
Using a slightly modified version of David Beazleys CPU bound test code (added loop for multiple tests), you can see
the difference between CPython and PyPys processing.
# PyPy
$ ./pypy -V
Python 2.7.1 (7773f8fc4223, Nov 18 2011, 18:47:10)
[PyPy 1.7.0 with GCC 4.4.3]
$ ./pypy measure2.py
0.0683999061584
0.0483210086823
0.0388588905334
0.0440690517426
0.0695300102234
# CPython
$ ./python -V
Python 2.7.1
$ ./python measure2.py
1.06774401665
1.45412397385
1.51485204697
1.54693889618
1.60109114647
3.10.1 Context
The GIL
The GIL (Global Interpreter Lock) is how Python allows multiple threads to operate at the same time. Pythons
memory management isnt entirely thread-safe, so the GIL is required to prevent multiple threads from running the
same Python code at once.
David Beazley has a great guide on how the GIL operates. He also covers the new GIL in Python 3.2. His results show
that maximizing performance in a Python application requires a strong understanding of the GIL, how it affects your
specific application, how many cores you have, and where your application bottlenecks are.
C Extensions
The GIL
Special care must be taken when writing C extensions to make sure you register your threads with the interpreter.
3.10. Speed
55
3.10.2 C Extensions
Cython
Cython implements a superset of the Python language with which you are able to write C and C++ modules for Python.
Cython also allows you to call functions from compiled C libraries. Using Cython allows you to take advantage of
Pythons strong typing of variables and operations.
Heres an example of strong typing with Cython:
def primes(int kmax):
"""Calculation of prime numbers with additional
Cython keywords"""
cdef int n, k, i
cdef int p[1000]
result = []
if kmax > 1000:
kmax = 1000
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
result.append(n)
n = n + 1
return result
This implementation of an algorithm to find prime numbers has some additional keywords compared to the next one,
which is implemented in pure Python:
def primes(kmax):
"""Calculation of prime numbers in standard Python syntax"""
p= range(1000)
result = []
if kmax > 1000:
kmax = 1000
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
result.append(n)
n = n + 1
return result
Notice that in the Cython version you declare integers and integer arrays to be compiled into C types while also
creating a Python list:
56
def primes(int kmax):

"""Calculation of prime numbers with additional
Cython keywords"""
cdef int n, k, i
cdef int p[1000]
result = []
def primes(kmax):
"""Calculation of prime numbers in standard Python syntax"""
p= range(1000)
result = []
What is the difference? In the upper Cython version you can see the declaration of the variable types and the integer
array in a similar way as in standard C. For example cdef int n,k,i in line 3. This additional type declaration (i.e. integer)
allows the Cython compiler to generate more efficient C code from the second version. While standard Python code
is saved in *.py files, Cython code is saved in *.pyx files.
Whats the difference in speed? Lets try it!
import time
#activate pyx compiler
import pyximport
pyximport.install()
#primes implemented with Cython
import primesCy
#primes implemented with Python
import primes
print "Cython:"
t1= time.time()
print primesCy.primes(500)
t2= time.time()
print "Cython time: %s" %(t2-t1)
print ""
print "Python"
t1= time.time()
print primes.primes(500)
t2= time.time()
print "Python time: %s" %(t2-t1)
These lines both need a remark:

import pyximport
pyximport.install()
The pyximport module allows you to import *.pyx files (e.g., primesCy.pyx) with the Cython-compiled version
of the primes function. The pyximport.install() command allows the Python interpreter to start the Cython compiler
directly to generate C-code, which is automatically compiled to a *.so C-library. Cython is then able to import this
library for you in your Python code, easily and efficiently. With the time.time() function you are able to compare the
time between these 2 different calls to find 500 prime numbers. On a standard notebook (dual core AMD E-450 1.6
GHz), the measured values are:
Cython time: 0.0054 seconds
Python time: 0.0566 seconds
And here the output of an embedded ARM beaglebone machine:
3.10. Speed
57
Cython time: 0.0196 seconds

Python time: 0.3302 seconds
Pyrex
Shedskin?
Numba
Todo
Write about Numba and the autojit compiler for NumPy
3.10.3 Threading
Threading
Spawning Processes
Multiprocessing
3.11 Scientific Applications

3.11.1 Context
Python is frequently used for high-performance scientific applications. It is widely used in academia and scientific
projects because it is easy to write and performs well.
Due to its high performance nature, scientific computing in Python often utilizes external libraries, typically written
in faster languages (like C, or FORTRAN for matrix operations). The main libraries used are NumPy, SciPy and
Matplotlib. Going into detail about these libraries is beyond the scope of the Python guide. However, a comprehensive
introduction to the scientific Python ecosystem can be found in the Python Scientific Lecture Notes
3.11.2 Tools
IPython
IPython is an enhanced version of Python interpreter, which provides features of great interest to scientists. The inline
mode allow graphics and plots to be displayed in the terminal (Qt based version). Moreover, the notebook mode supports literate programming and reproducible science generating a web-based Python notebook. This notebook allows
you to store chunks of Python code along side the results and additional comments (HTML, LaTeX, Markdown). The
notebook can then be shared and exported in various file formats.
58
3.11.3 Libraries
NumPy
NumPy is a low level library written in C (and FORTRAN) for high level mathematical functions. NumPy cleverly
overcomes the problem of running slower algorithms on Python by using multidimensional arrays and functions that
operate on arrays. Any algorithm can then be expressed as a function on arrays, allowing the algorithms to be run
quickly.
NumPy is part of the SciPy project, and is released as a separate library so people who only need the basic requirements
can use it without installing the rest of SciPy.
NumPy is compatible with Python versions 2.4 through to 2.7.2 and 3.1+.
Numba
Numba is an Numpy aware Python compiler (just-in-time (JIT) specializing compiler) which compiles annotated
Python (and Numpy) code to LLVM (Low Level Virtual Machine) through special decorators. Briefly, Numba uses a
system that compiles Python code with LLVM to code which can be natively executed at runtime.
SciPy
SciPy is a library that uses NumPy for more mathematical functions. SciPy uses NumPy arrays as the basic data structure, and comes with modules for various commonly used tasks in scientific programming, including linear algebra,
integration (calculus), ordinary differential equation solving and signal processing.
Matplotlib
Matplotlib is a flexible plotting library for creating interactive 2D and 3D plots that can also be saved as manuscriptquality figures. The API in many ways reflects that of MATLAB, easing transition of MATLAB users to Python.
Many examples, along with the source code to re-create them, are available in the matplotlib gallery.
Pandas
Pandas is data manipulation library based on Numpy which provides many useful functions for accessing, indexing,
merging and grouping data easily. The main data structure (DataFrame) is close to what could be found in the R
statistical package; that is, heterogeneous data tables with name indexing, time series operations and auto-alignment
of data.
Rpy2
Rpy2 is a Python binding for the R statistical package allowing the execution of R functions from Python and passing
data back and forth between the two environments. Rpy2 is the object oriented implementation of the Rpy bindings.
PsychoPy
PsychoPy is a library for cognitive scientists allowing the creation of cognitive psychology and neuroscience experiments. The library handles presentation of stimuli, scripting of experimental design and data collection.
3.11. Scientific Applications
59
3.11.4 Resources
Installation of scientific Python packages can be troublesome, as many of these packages are implemented as Python
C extensions which need to be compiled. This section lists various so-called scientific Python distributions which
provide precompiled and easy-to-install collections of scientific Python packages.
Unofficial Windows Binaries for Python Extension Packages
Many people who do scientific computing are on Windows, yet many of the scientific computing packages are notoriously difficult to build and install on this platform. Christoph Gohlke however, has compiled a list of Windows
binaries for many useful Python packages. The list of packages has grown from a mainly scientific Python resource to
a more general list. If youre on Windows, you may want to check it out.
Anaconda
Continuum Analytics offers the Anaconda Python Distribution which includes all the common scientific Python packages as well as many packages related to data analytics and big data. Anaconda itself is free, and Continuum sells a
number of proprietary add-ons. Free licenses for the add-ons are available for academics and researchers.
Canopy
Canopy is another scientific Python distribution, produced by Enthought. A limited Canopy Express variant is
available for free, but Enthought charges for the full distribution. Free licenses are available for academics.
3.12 Image Manipulation

Todo
Add introduction about image manipulation and its Python libraries.
3.12.1 Python Imaging Library

The Python Imaging Library, or PIL for short, is the library for image manipulation in Python. Unfortunately, its
development has stagnated, with its last release in 2009.
Luckily for you, theres an actively-developed fork of PIL called Pillow <http://python-pillow.github.io/_ - its
easier to install, runs on all operating systems, and supports Python 3.
Installation
Before installing Pillow, youll have to install Pillows prerequisites. Find the instructions for your platform here.
After that, its straightforward:
$ pip install Pillow
60
3.13 XML parsing

3.13.1 untangle
untangle is a simple library which takes an XML document and returns a Python object which mirrors the nodes and
attributes in its structure.
For example, an XML file like this:
<?xml version="1.0"?>
<root>
<child name="child1">
</root>
can be loaded like this:

import untangle
obj = untangle.parse(path/to/file.xml)
and then you can get the child elements name like this:
obj.root.child[name]
untangle also supports loading XML from a string or an URL.
3.13.2 xmltodict
xmltodict is another simple library that aims at making XML feel like working with JSON.
An XML file like this:
<mydocument has="an attribute">
<and>
<many>elements</many>
<many>more elements</many>
</and>
<plus a="complex">
element as well
</plus>
</mydocument>
can be loaded into a Python dict like this:

import xmltodict
with open(path/to/file.xml) as fd:
obj = xmltodict.parse(fd.read())
and then you can access elements, attributes and values like this:
doc[mydocument][@has] # == uan attribute
doc[mydocument][and][many] # == [uelements, umore elements]
doc[mydocument][plus][@a] # == ucomplex
doc[mydocument][plus][#text] # == uelement as well
xmltodict also lets you roundtrip back to XML with the unparse function, has a streaming mode suitable for handling
files that dont fit in memory and supports namespaces.
3.13. XML parsing
61
3.14 Cryptography
3.14.1 Cryptography
Cryptography is an actively developed library that provides cryptographic recipes and primitives. It supports Python
2.6-2.7, Python 3.2+ and PyPy.
Cryptography is divided into two layers of recipes and hazardous materials (hazmat). The recipes layer provides
simple API for proper symmetric encryption and the hazmat layer provides low-level cryptographic primitives.
Installation
$ pip install cryptography
Example
Example code using high level symmetric encryption recipe:
from cryptography.fernet import Fernet
key = Fernet.generate_key()
cipher_suite = Fernet(key)
cipher_text = cipher_suite.encrypt(b"A really secret message. Not for prying eyes.")
plain_text = cipher_suite.decrypt(cipher_text)
3.14.2 PyCrypto
PyCrypto is another library, which provides secure hash functions and various encryption algorithms. It supports
Python version 2.1 through 3.3.
Installation
$ pip install pycrypto
Example
from Crypto.Cipher import AES
# Encryption
encryption_suite = AES.new(This is a key123, AES.MODE_CBC, This is an IV456)
cipher_text = encryption_suite.encrypt("A really secret message. Not for prying eyes.")
# Decryption
decryption_suite = AES.new(This is a key123, AES.MODE_CBC, This is an IV456)
plain_text = decryption_suite.decrypt(cipher_text)
62
CHAPTER 4
Shipping Great Code
This part of the guide focuses on deploying your Python code.
4.1 Packaging Your Code

Packaging your code is important.
Youll need to package your code first before sharing it with other developers.
The Python Packaging Guide provides an extensive guide on creating and maintaining Python packages.
4.1.1 For Python Developers

If youre writing an open source Python module, PyPI, more properly known as The Cheeseshop, is the place to host
it.
Pip vs. easy_install
Use pip. More details here
Personal PyPI
If you want to install packages from a source different from PyPI, (say, if your packages are proprietary), you can do
it by hosting a simple http server, running from the directory which holds those packages which need to be installed.
Showing an example is always beneficial
Say if you are after installing a package called MyPackage.tar.gz, and assuming this is your directory structure:
archive
MyPackage
* MyPackage.tar.gz
Go to your command prompt and type:
$ cd archive
$ python -m SimpleHTTPServer 9000
63
This runs a simple http server running on port 9000 and will list all packages (like MyPackage). Now you can install
MyPackage using any Python package installer. Using Pip, you would do it like:
$ pip install --extra-index-url=http://127.0.0.1:9000/ MyPackage
Having a folder with the same name as the package name is crucial here. I got fooled by that, one time. But if you
feel that creating a folder called MyPackage and keeping MyPackage.tar.gz inside that, is redundant, you can
still install MyPackage using:
$ pip install
http://127.0.0.1:9000/MyPackage.tar.gz
pypiserver
Pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages to easy_install or pip.
It includes helpful features like an administrative command (-U ) which will update all its packages to their latest
versions found on PyPI.
S3-Hosted PyPi
One simple option for a personal PyPi server is to use Amazon S3. A prerequisite for this is that you have an Amazon
AWS account with an S3 bucket.
1. Install all your requirements from PyPi or another source
2. Install pip2pi
pip install git+https://github.com/wolever/pip2pi.git
3. Follow pip2pi README for pip2tgz and dir2pi commands
pip2tgz packages/ YourPackage (or pip2tgz packages/ -r requirements.txt)
dir2pi packages/
4. Upload the new files
Use a client like Cyberduck to sync the entire packages folder to your s3 bucket
Make sure you upload packages/simple/index.html as well as all new files and directories
5. Fix new file permissions
By default, when you upload new files to the S3 bucket, they will have the wrong permissions set.
Use the Amazon web console to set the READ permission of the files to EVERYONE.
If you get HTTP 403 when trying to install a package, make sure youve set the permissions correctly.
6. All done
You can now install your package with pip install --index-url=http://your-s3-bucket/packages/simple
YourPackage
4.1.2 For Linux Distributions

Ubuntu
Fedora
Debian
64
Chapter 4. Shipping Great Code
Arch
Useful Tools
fpm
alien
4.2 Freezing Your Code

An alternative to shipping your code is freezing it shipping it as an executable with a bundled Python interpreter.
Many applications you use every day do this:
Dropbox
BitTorrent
...
Todo
Fill in Freezing Your Code stub
4.2.1 Comparison
Solutions and platforms/features supported:
SoluWintion
dows
bbFreeze yes
py2exe
yes
pyInyes
staller
cx_Freeze yes
py2app no
Linux OS
X
yes
yes
no
no
yes
yes
Python
3
no
yes
no
License
MIT
MIT
GPL
One-file
mode
no
yes
yes
Zipfile
import
yes
yes
no
Eggs pkg_resources
support
yes
yes
no
no
yes
no
yes
no
yes
yes
PSF
MIT
no
no
yes
yes
yes
yes
yes
yes
no
yes
Note: Freezing Python code on Linux into a Windows executable was only once supported in PyInstaller and later
dropped..
Note: All solutions need MS Visual C++ dll to be installed on target machine, except py2app. Only Pyinstaller makes
self-executable exe that bundles the dll when passing --onefile to Configure.py.
4.2.2 Windows
bbFreeze
Prerequisite is to install Python, Setuptools and pywin32 dependency on Windows.
Todo
Write steps for most basic .exe
4.2. Freezing Your Code
65
py2exe
Prerequisite is to install Python on Windows.
1. Download and install http://sourceforge.net/projects/py2exe/files/py2exe/
2. Write setup.py (List of configuration options):
from distutils.core import setup
import py2exe
setup(
windows=[{script: foobar.py}],
)
3. (Optionally) include icon

4. (Optionally) one-file mode
5. Generate .exe into dist directory:
$ python setup.py py2exe
6. Provide the Microsoft Visual C runtime DLL. Two options: globally install dll on target machine or distribute
dll alongside with .exe.
PyInstaller
Prerequisite is to have installed Python, Setuptools and pywin32 dependency on Windows.
Most basic tutorial
Manual
4.2.3 OS X
py2app
PyInstaller
4.2.4 Linux
bbFreeze
PyInstaller
66
Chapter 4. Shipping Great Code
CHAPTER 5
Development Environment
5.1 Your Development Environment

5.1.1 Text Editors
Just about anything that can edit plain text will work for writing Python code, however, using a more powerful editor
may make your life a bit easier.
Vim
Vim is a text editor which uses keyboard shortcuts for editing instead of menus or icons. There are a couple of plugins
and settings for the Vim editor to aid Python development. If you only develop in Python, a good start is to set the
default settings for indentation and line-wrapping to values compliant with PEP 8. In your home directory, open a file
called .vimrc and add the following lines:
set
set
set
set
set
set
set
textwidth=79
shiftwidth=4
tabstop=4
expandtab
softtabstop=4
shiftround
autoindent
"
"
"
"
"
"
"
lines longer than 79 columns will be broken

operation >> indents 4 columns; << unindents 4 columns
a hard TAB displays as 4 columns
insert spaces when hitting TABs
insert/delete 4 spaces when hitting a TAB/BACKSPACE
round indent to multiple of shiftwidth
align the new line indent with the previous line
With these settings, newlines are inserted after 79 characters and indentation is set to 4 spaces per tab. If you also use
Vim for other languages, there is a handy plugin called indent, which handles indentation settings for Python source
files.
There is also a handy syntax plugin called syntax featuring some improvements over the syntax file included in Vim
6.1.
These plugins supply you with a basic environment for developing in Python. To get the most out of Vim, you should
continually check your code for syntax errors and PEP8 compliance. Luckily PEP8 and Pyflakes will do this for you.
If your Vim is compiled with +python you can also utilize some very handy plugins to do these checks from within
the editor.
For PEP8 checking, install the vim-pep8 plugin, and for pyflakes you can install vim-pyflakes. Now you can map the
functions Pep8() or Pyflakes() to any hotkey or action you want in Vim. Both plugins will display errors at the
bottom of the screen, and provide an easy way to jump to the corresponding line. Its very handy to call these functions
whenever you save a file. In order to do this, add the following lines to your .vimrc:
67
autocmd BufWritePost *.py call Pyflakes()

autocmd BufWritePost *.py call Pep8()
If you are already using syntastic, you can set it to run Pyflakes on write and show errors and warnings in the quickfix
window. An example configuration to do that which also shows status and warning messages in the statusbar would
be:
set
set
set
let
let
statusline+=%#warningmsg#
statusline+=%{SyntasticStatuslineFlag()}
statusline+=%*
g:syntastic_auto_loc_list=1
g:syntastic_loc_list_height=5
Python-mode
Python-mode is a complex solution for working with Python code in Vim. It has:
Asynchronous Python code checking (pylint, pyflakes, pep8, mccabe) in any combination
Code refactoring and autocompletion with Rope
Fast Python folding
Virtualenv support
Search through Python documentation and run Python code
Auto PEP8 error fixes
And more.
SuperTab
SuperTab is a small Vim plugin that makes code completion more convenient by using <Tab> key or any other
customized keys.
Emacs
Emacs is another powerful text editor. It is fully programmable (lisp), but it can be some work to wire up correctly. A
good start if youre already an Emacs user is Python Programming in Emacs at EmacsWiki.
1. Emacs itself comes with a Python mode.
2. Python ships with an alternate version: python-mode.el
3. Fabin Ezequiel Gallinas python.el provides nice functionality and behavior out of the box
TextMate
TextMate brings Apples approach to operating systems into the world of text editors. By bridging UNIX
underpinnings and GUI, TextMate cherry-picks the best of both worlds to the benefit of expert scripters
and novice users alike.
68
Chapter 5. Development Environment
Sublime Text
Sublime Text is a sophisticated text editor for code, markup and prose. Youll love the slick user interface,
extraordinary features and amazing performance.
Sublime Text has excellent support for editing Python code and uses Python for its plugin API. It also has a diverse
variety of plugins, some of which allow for in-editor PEP8 checking and code linting.
Atom
Atom is a hackable text editor for the 21st century, built on atom-shell, and based on everything we love
about our favorite editors.
Atom is web native (HTML, CSS, JS), focusing on modular design and easy plugin development. It comes with
native package control and plethora of packages. Recommended for Python development is Linter combined with
linter-flake8.
5.1.2 IDEs
PyCharm / IntelliJ IDEA
PyCharm is developed by JetBrains, also known for IntelliJ IDEA. Both share the same code base and most of PyCharms features can be brought to IntelliJ with the free Python Plug-In. There are two versions of PyCharm: Professional Edition (Free 30-day trial) and Community Edition(Apache 2.0 License) with less features.
Eclipse
The most popular Eclipse plugin for Python development is Aptanas PyDev.
Komodo IDE
Komodo IDE is developed by ActiveState and is a commercial IDE for Windows, Mac, and Linux. KomodoEdit is the
open source alternative.
Spyder
Spyder is an IDE specifically geared toward working with scientific Python libraries (namely Scipy). It includes
integration with pyflakes, pylint and rope.
Spyder is open-source (free), offers code completion, syntax highlighting, a class and function browser, and object
inspection.
WingIDE
WingIDE is a Python specific IDE. It runs on Linux, Windows and Mac (as an X11 application, which frustrates some
Mac users).
WingIDE offers code completion, syntax highlighting, source browser, graphical debugger and support for version
control systems.
5.1. Your Development Environment
69
NINJA-IDE
NINJA-IDE (from the recursive acronym: Ninja-IDE Is Not Just Another IDE) is a cross-platform IDE, specially
designed to build Python applications, and runs on Linux/X11, Mac OS X and Windows desktop operating systems.
Installers for these platforms can be downloaded from the website.
NINJA-IDE is open-source software (GPLv3 licence) and is developed in Python and Qt. The source files can be
downloaded from GitHub.
Eric (The Eric Python IDE)
Eric is a full featured Python IDE offering sourcecode autocompletion, syntax highlighting, support for version control
systems, python 3 support, integrated web browser, python shell, integrated debugger and a flexible plug-in system.
Written in python, it is based on the Qt gui toolkit, integrating the Scintilla editor control. Eric is an open-source
software (GPLv3 licence) with more than ten years of active development.
5.1.3 Interpreter Tools

virtualenv
Virtualenv is a tool to keep the dependencies required by different projects in separate places, by creating virtual
Python environments for them. It solves the Project X depends on version 1.x but, Project Y needs 4.x dilemma,
and keeps your global site-packages directory clean and manageable.
virtualenv creates a folder which contains all the necessary executables to use the packages that a Python project would
need. An example workflow is given below.
Install virtualenv:
Create a virtual environment for a project:

$ cd my_project
$ virtualenv venv
virtualenv venv will create a folder in the current directory which will contain the Python executable files, and
a copy of the pip library which you can use to install other packages. The name of the virtual environment (in this
case, it was venv) can be anything; omitting the name will place the files in the current directory instead.
To start using the virtual environment, run:
$ source venv/bin/activate
The name of the current virtual environment will now appear on the left of the prompt (e.g.
(venv)Your-Computer:your_project UserName$) to let you know that its active. From now on,
any package that you install using pip will be placed in the venv folder, isolated from the global Python installation.
Install packages as usual:
$ pip install requests
To stop using an environment, simply type deactivate. To remove the environment, just remove the directory it
was installed into. (In this case, it would be rm -rf venv.)
70
Other Notes
Running virtualenv with the option --no-site-packages will not include the packages that are installed
globally. This can be useful for keeping the package list clean in case it needs to be accessed later. [This is the default
behavior for virtualenv 1.7 and later.]
In order to keep your environment consistent, its a good idea to freeze the current state of the environment packages.
To do this, run
$ pip freeze > requirements.txt
This will create a requirements.txt file, which contains a simple list of all the packages in the current environment, and their respective versions. Later it will be easier for a different developer (or you, if you need to re-create the
environment) to install the same packages using the same versions:
$ pip install -r requirements.txt
This can help ensure consistency across installations, across deployments, and across developers.
Lastly, remember to exclude the virtual environment folder from source control by adding it to the ignore list.
virtualenvwrapper
Virtualenvwrapper makes virtualenv a pleasure to use by wrapping the command line API with a nicer CLI.
$ pip install virtualenvwrapper
Put this into your ~/.bash_profile (Linux/Mac) file:

$ export VIRTUALENVWRAPPER_VIRTUALENV_ARGS=--no-site-packages
This will prevent your virtualenvs from relying on your (global) site packages directory, so that they are completely
separate. [Note: This is the default behavior for virtualenv 1.7 and later]
5.1.4 Other Tools

IDLE
IDLE is an integrated development environment that is part of Python standard library. It is completely written in
Python and uses the Tkinter GUI toolkit. Though IDLE is not suited for full-blown development using Python, it is
quite helpful to try out small Python snippets and experiment with different features in Python.
It provides the following features:
Python Shell Window (interpreter)
Multi window text editor that colorizes Python code
Minimal debugging facility
IPython
IPython provides a rich toolkit to help you make the most out of using Python interactively. Its main components are:
Powerful Python shells (terminal- and Qt-based).
A web-based notebook with the same core features but support for rich media, text, code, mathematical expressions and inline plots.
5.1. Your Development Environment
71
Support for interactive data visualization and use of GUI toolkits.

Flexible, embeddable interpreters to load into your own projects.
Tools for high level and interactive parallel computing.
$ pip install ipython
BPython
bpython is an alternative interface to the Python interpreter for Unix-like operating systems. It has the following
features:
In-line syntax highlighting.
Readline-like autocomplete with suggestions displayed as you type.
Expected parameter list for any Python function.
Rewind function to pop the last line of code from memory and re-evaluate.
Send entered code off to a pastebin.
Save entered code to a file.
Auto-indentation.
Python 3 support.
$ pip install bpython
5.2 Virtual Environments

A Virtual Environment, put simply, is an isolated working copy of Python which allows you to work on a specific
project without worry of affecting other projects.
For example, you can work on a project which requires Django 1.3 while also maintaining a project which requires
Django 1.0.
5.2.1 virtualenv
virtualenv is a tool to create isolated Python environments.
Install it via pip:
Basic Usage
1. Create a virtual environment:
$ virtualenv venv
This creates a copy of Python in whichever directory you ran the command in, placing it in a folder named venv.
You can also use a Python interpreter of your choice.
72
$ virtualenv -p /usr/bin/python2.7 venv
This will use the Python interpreter in /usr/bin/python2.7

2. To begin using the virtual environment, it needs to be activated:
$ source venv/bin/activate
You can then begin installing any new modules without affecting the system default Python or other virtual environments.
3. If you are done working in the virtual environment for the moment, you can deactivate it:
$ deactivate
This puts you back to the systems default Python interpreter with all its installed libraries.
To delete a virtual environment, just delete its folder.
After a while, though, you might end up with a lot of virtual environments littered across your system, and its possible
youll forget their names or where they were placed.
5.2.2 virtualenvwrapper
virtualenvwrapper provides a set of commands which makes working with virtual environments much more pleasant.
It also places all your virtual environments in one place.
To install (make sure virtualenv is already installed):
$ pip install virtualenvwrapper
$ export WORKON_HOME=~/Envs
$ source /usr/local/bin/virtualenvwrapper.sh
(Full virtualenvwrapper install instructions.)

For Windows, you can use the virtualenvwrapper-powershell clone.
To install (make sure virtualenv is already installed):
PS>
PS>
PS>
PS>
pip install virtualenvwrapper-powershell

$env:WORKON_HOME="~/Envs"
mkdir $env:WORKON_HOME
import-module virtualenvwrapper
Basic Usage
1. Create a virtual environment:
$ mkvirtualenv venv
This creates the venv folder inside ~/Envs.

2. Work on a virtual environment:
$ workon venv
virtualenvwrapper provides tab-completion on environment names. It really helps when you have a lot of environments and have trouble remembering their names.
workon also deactivates whatever environment you are currently in, so you can quickly switch between environments.
5.2. Virtual Environments
73
3. Deactivating is still the same:

$ deactivate
4. To delete:
$ rmvirtualenv venv
Other useful commands

lsvirtualenv List all of the environments.
cdvirtualenv Navigate into the directory of the currently activated virtual environment, so you can browse its
site-packages, for example.
cdsitepackages Like the above, but directly into site-packages directory.
lssitepackages Shows contents of site-packages directory.
Full list of virtualenvwrapper commands.
5.2.3 autoenv
When you cd into a directory containing a .env, autoenv automagically activates the environment.
Install it on Mac OS X using brew:
$ brew install autoenv
And on Linux:
$ git clone git://github.com/kennethreitz/autoenv.git ~/.autoenv
$ echo source ~/.autoenv/activate.sh >> ~/.bashrc
74
CHAPTER 6
Additional Notes
This part of the guide, which is mostly prose, begins with some background information about Python, then focuses
on next steps.
6.1 Introduction
From the official Python website:
Python is a general-purpose, high-level programming language similar to Tcl, Perl, Ruby, Scheme, or Java. Some of
its main key features include:
very clear, readable syntax
Pythons philosophy focuses on readability, from code blocks delineated with significant whitespace to intuitive
keywords in place of inscrutable punctuation.
extensive standard libraries and third party modules for virtually any task
Python is sometimes described with the words batteries included because of its extensive standard library,
which includes modules for regular expressions, file IO, fraction handling, object serialization, and much more.
Additionally, the Python Package Index is available for users to submit their packages for widespread use,
similar to Perls CPAN. There is a thriving community of very powerful Python frameworks and tools like the
Django web framework and the NumPy set of math routines.
integration with other systems
Python can integrate with Java libraries, enabling it to be used with the rich Java environment that corporate
programmers are used to. It can also be extended by C or C++ modules when speed is of the essence.
ubiquity on computers
Python is available on Windows, *nix, and Mac. It runs wherever the Java virtual machine runs, and the reference
implementation CPython can help bring Python to wherever there is a working C compiler.
friendly community
Python has a vibrant and large community which maintains wikis, conferences, countless repositories, mailing
lists, IRC channels, and so much more. Heck, the Python community is even helping to write this guide!
75
6.1.1 About This Guide

Purpose
The Hitchhikers Guide to Python exists to provide both novice and expert Python developers a best-practice handbook
to the installation, configuration, and usage of Python on a daily basis.
By the Community
This guide is architected and maintained by Kenneth Reitz in an open fashion. This is a community-driven effort that
serves one purpose: to serve the community.
For the Community
All contributions to the Guide are welcome, from Pythonistas of all levels. If you think theres a gap in what the Guide
covers, fork the Guide on GitHub and submit a pull request.
Contributions are welcome from everyone, whether theyre an old hand or a first-time Pythonista, and the authors
to the Guide will gladly help if you have any questions about the appropriateness, completeness, or accuracy of a
contribution.
To get started working on The Hitchhikers Guide, see the Contribute page.
6.2 The Community

6.2.1 BDFL
Guido van Rossum, the creator of Python, is often referred to as the BDFL the Benevolent Dictator For Life.
6.2.2 Python Software Foundation

The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language,
and to support and facilitate the growth of a diverse and international community of Python programmers.
Learn More about the PSF.
6.2.3 PEPs
PEPs are Python Enhancement Proposals. They describe changes to Python itself, or the standards around it.
There are three different types of PEPs (as defined by PEP 1):
Standards Describes a new feature or implementation.
Informational Describes a design issue, general guidelines, or information to the community.
Process Describes a process related to Python.
76
Chapter 6. Additional Notes
Notable PEPs
There are a few PEPs that could be considered required reading:
PEP 8: The Python Style Guide. Read this. All of it. Follow it.
PEP 20: The Zen of Python. A list of 19 statements that briefly explain the philosophy behind Python.
PEP 257: Docstring Conventions. Gives guidelines for semantics and conventions associated with Python
docstrings.
You can read more at The PEP Index.
Submitting a PEP
PEPs are peer-reviewed and accepted/rejected after much discussion. Anyone can write and submit a PEP
for review.
Heres an overview of the PEP acceptance workflow:
6.2.4 Python Conferences

The major events for the Python community are developer conferences. The two most notable conferences are PyCon,
which is held in the US, and its European sibling, EuroPython.
A comprehensive list of conferences is maintained at pycon.org.
6.2.5 Python User Groups

User Groups are where a bunch of Python developers meet to present or talk about Python topics of interest. A list of
local user groups is maintained at the Python Software Foundation Wiki.
6.3 Learning Python

6.3.1 Beginner
The Python Tutorial
This is the official tutorial. It covers all the basics, and offers a tour of the language and the standard library. Recommended for those who need a quickstart guide to the language.
The Python Tutorial
Learn Python Interactive Tutorial
Learnpython.org is an easy non-intimidating way to get introduced to Python. The website takes the same approach
used on the popular Try Ruby website, it has an interactive Python interpreter built into the site that allows you to go
through the lessons without having to install Python locally.
Learn Python
If you want a more traditional book, Python For You and Me is an excellent resource for learning all aspects of the
language.
6.3. Learning Python
77
Python for You and Me

Invent Your Own Computer Games with Python
This beginners book is for those with no programming experience at all. Each chapter has the source code to a
small game, using these example programs to demonstrate programming concepts to give the reader an idea of what
programs look like.
Invent Your Own Computer Games with Python
Hacking Secret Ciphers with Python
This book teaches Python programming and basic cryptography for absolute beginners. The chapters provide the
source code for various ciphers, as well as programs that can break them.
Hacking Secret Ciphers with Python
Learn Python the Hard Way
This is an excellent beginner programmers guide to Python. It covers hello world from the console to the web.
Learn Python the Hard Way
Crash into Python
Also known as Python for Programmers with 3 Hours, this guide gives experienced developers from other languages
a crash course on Python.
Crash into Python
Dive Into Python 3
Dive Into Python 3 is a good book for those ready to jump in to Python 3. Its a good read if you are moving from
Python 2 to 3 or if you already have some experience programming in another language.
Dive Into Python 3
Think Python: How to Think Like a Computer Scientist
Think Python attempts to give an introduction to basic concepts in computer science through the use of the Python
language. The focus was to create a book with plenty of exercises, minimal jargon and a section in each chapter
devoted to the subject of debugging.
While exploring the various features available in the Python language the author weaves in various design patterns and
best practices.
The book also includes several case studies which have the reader explore the topics discussed in the book in greater
detail by applying those topics to real-world examples. Case studies include assignments in GUI and Markov Analysis.
Think Python
78
Python Koans
Python Koans is a port of Edgecases Ruby Koans. It uses a test-driven approach, q.v. TEST DRIVEN DESIGN
SECTION to provide an interactive tutorial teaching basic Python concepts. By fixing assertion statements that fail in
a test script, this provides sequential steps to learning Python.
For those used to languages and figuring out puzzles on their own, this can be a fun, attractive option. For those new
to Python and programming, having an additional resource or reference will be helpful.
Python Koans
More information about test driven development can be found at these resources:
Test Driven Development
A Byte of Python
A free introductory book that teaches Python at the beginner level, it assumes no previous programming experience.
A Byte of Python for Python 2.x A Byte of Python for Python 3.x
6.3.2 Advanced
Pro Python
This book is for intermediate to advanced Python programmers who are looking to understand how and why Python
works the way it does and how they can take their code to the next level.
Pro Python
Expert Python Programming
Expert Python Programming deals with best practices in programming Python and is focused on the more advanced
crowd.
It starts with topics like decorators (with caching, proxy, and context manager case-studies), method resolution order,
using super() and meta-programming, and general PEP 8 best practices.
It has a detailed, multi-chapter case study on writing and releasing a package and eventually an application, including a
chapter on using zc.buildout. Later chapters detail best practices with writing documentation, test-driven development,
version control, and optimization/profiling.
Expert Python Programming
A Guide to Pythons Magic Methods
This is a collection of blog posts by Rafe Kettler which explain magic methods in Python. Magic methods are
surrounded by double underscores (i.e. __init__) and can make classes and objects behave in different and magical
ways.
A Guide to Pythons Magic Methods
6.3. Learning Python
79
6.3.3 For Engineers and Scientists

A Primer on Scientific Programming with Python
A Primer on Scientific Programming with Python, written by Hans Petter Langtangen, mainly covers Pythons usage
in scientific field. In the book, examples are chosen from mathematics and the natural sciences.
A Primer on Scientific Programming with Python
Numerical Methods in Engineering with Python
Numerical Methods in Engineering with Python, written by Jaan Kiusalaas, attempts to emphasis on numerical methods and how to implement them in Python.
Numerical Methods in Engineering with Python
6.3.4 Miscellaneous topics

Problem Solving with Algorithms and Data Structures
Problem Solving with Algorithms and Data Structures covers a range of data structures and algorithms. All concepts
are illustrated with Python code along with interactive samples that can be run directly in the browser.
Problem Solving with Algorithms and Data Structures
Programming Collective Intelligence
Programming Collective Intelligence introduces a wide array of basic machine learning and data mining methods. The
exposition is not very mathematically formal, but rather focuses on explaining the underlying intuition and shows how
to implement the algorithms in Python.
Programming Collective Intelligence
6.3.5 References
Python in a Nutshell
Python in a Nutshell, written by Alex Martelli, covers most cross-platform Pythons usage, from its syntax to built-in
libraries to advanced topics such as writing C extensions.
Python in a Nutshell
The Python Language Reference
This is Pythons reference manual, it covers the syntax and the core semantics of the language.
The Python Language Reference
80
Python Pocket Reference

Python Pocket Reference, written by Mark Lutz, is an easy to use reference to the core language, with descriptions of
commonly used modules and toolkits. It covers Python 3 and 2.6 versions.
Python Pocket Reference
Python Cookbook
Python Cookbook, written by David Beazley and Brian K. Jones, is packed with practical recipes. This book covers
the core python language as well as tasks common to a wide variety of application domains.
Python Cookbook
Writing Idiomatic Python
Writing Idiomatic Python, written by Jeff Knupp, contains the most common and important Python idioms in a
format that maximizes identification and understanding. Each idiom is presented as a recommendation of a way to
write some commonly used piece of code, followed by an explanation of why the idiom is important. It also contains
two code samples for each idiom: the Harmful way to write it and the Idiomatic way.
For Python 2.7.3+
For Python 3.3+
6.4 Documentation
6.4.1 Official Documentation
The official Python Language and Library documentation can be found here:
Python 2.x
Python 3.x
6.4.2 Read the Docs

Read the Docs is a popular community project that hosts documentation for open source software. It holds documentation for many Python modules, both popular and exotic.
Read the Docs
6.4.3 pydoc
pydoc is a utlity that is installed when you install Python. It allows you to quickly retrieve and search for documentation from your shell. For example, if you needed a quick refresher on the time module, pulling up documentation
would be as simple as
$ pydoc time
The above command is essentially equivalent to opening the Python REPL and running
6.4. Documentation
81
>>> help(time)
6.5 News
6.5.1 Planet Python
This is an aggregate of Python news from a growing number of developers.
Planet Python
6.5.2 /r/python
/r/python is the Reddit Python community where users contribute and vote on Python-related news.
/r/python
6.5.3 Pycoders Weekly

Pycoders Weekly is a free weekly Python newsletter for Python developers by Python developers (Projects, Articles,
News, and Jobs).
Pycoders Weekly
6.5.4 Python Weekly

Python Weekly is a free weekly newsletter featuring curated news, articles, new releases, jobs, etc. related to Python.
Python Weekly
6.5.5 Python News

News section in the official Python web site (www.python.org). It briefly highlights the news from Python community.
Python News
Contribution notes and legal information are here (for those interested).
6.6 Contribute
Python-guide is under active development, and contributors are welcome.
If you have a feature request, suggestion, or bug report, please open a new issue on GitHub. To submit patches,
please send a pull request on GitHub. Once your changes get merged back in, youll automatically be added to the
Contributors List.
6.6.1 Style Guide

For all contributions, please follow the The Guide Style Guide.
82
6.6.2 Todo List

If youd like to contribute, theres plenty to do. Heres a short todo list.
Establish use this vs alternatives are.... recommendations
Todo
Add introduction about image manipulation and its Python libraries.
(The
original
entry
is
located
guide/checkouts/latest/docs/scenarios/imaging.rst, line 5.)
in
/var/build/user_builds/python-
in
in
in
Todo
Write about Numba and the autojit compiler for NumPy
(The
original
entry
is
located
guide/checkouts/latest/docs/scenarios/speed.rst, line 227.)
Todo
Fill in Freezing Your Code stub
(The
original
entry
is
located
guide/checkouts/latest/docs/shipping/freezing.rst, line 13.)
Todo
Write steps for most basic .exe
(The
original
entry
is
located
guide/checkouts/latest/docs/shipping/freezing.rst, line 50.)
Todo
Embed
and
explain
YouTube
video
showing
python
code
reading:
http://www.youtube.com/watch?v=Jc8M9-LoEuo
This
may
require
installing
a
Sphinx
plugin.
https://bitbucket.org/birkenfeld/sphinxcontrib/src/a09f29fc16970f34350ca36ac7f229e00b1b1674/youtube?at=default
(The
original
entry
is
located
guide/checkouts/latest/docs/writing/reading.rst, line 41.)
in
Todo
Include code examples of exemplary code from each of the projects listed. Explain why it is excellent
code. Use complex examples.
(The
original
entry
is
located
in
Todo
Explain techniques to rapidly identify data structures, algorithms and determine what the code is doing.
6.6. Contribute
83
(The
original
entry
is
located
in
in
in
in
Todo
Write about Chef
Chef Documentation
(The
original
entry
is
located
guide/checkouts/latest/docs/scenarios/admin.rst, line 235.)
Todo
Write about Blueprint
(The
original
entry
is
located
Todo
Write about Buildout
Buildout Website
(The
original
entry
is
located
6.7 License
The Guide is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported license.
6.8 The Guide Style Guide

As with all documentation, having a consistent format helps make the document more understandable. In order to
make The Guide easier to digest, all contributions should fit within the rules of this style guide where appropriate.
The Guide is written as reStructuredText.
Note: Parts of The Guide may not yet match this style guide. Feel free to update those parts to be in sync with The
Guide Style Guide
Note: On any page of the rendered HTML you can click Show Source to see how authors have styled the page.
6.8.1 Relevancy
Strive to keep any contributions relevant to the purpose of The Guide.
Avoid including too much information on subjects that dont directly relate to Python development.
Prefer to link to other sources if the information is already out there. Be sure to describe what and why you are
linking.
84
Cite references where needed.

If a subject isnt directly relevant to Python, but useful in conjunction with Python (e.g., Git, GitHub, Databases),
reference by linking to useful resources, and describe why its useful to Python.
When in doubt, ask.
6.8.2 Headings
Use the following styles for headings.
Chapter title:
#########
Chapter 1
#########
Page title:
===================
Time is an Illusion
===================
Section headings:
Lunchtime Doubly So
-------------------
Sub section headings:

Very Deep
~~~~~~~~~
6.8.3 Prose
Wrap text lines at 78 characters. Where necessary, lines may exceed 78 characters, especially if wrapping would make
the source text more difficult to read.
6.8.4 Code Examples

Wrap all code examples at 70 characters to avoid horizontal scrollbars.
Command line examples:
.. code-block:: console
$ run command --help
$ ls ..
Be sure to include the $ prefix before each line.

Python interpreter examples:
Label the example::
.. code-block:: python
>>> import this
6.8. The Guide Style Guide
85
Python examples:
Descriptive title::
.. code-block:: python
def get_answer():
return 42
6.8.5 Externally Linking

Prefer labels for well known subjects (ex: proper nouns) when linking:
Sphinx_ is used to document Python.
.. _Sphinx: http://sphinx.pocoo.org
Prefer to use descriptive labels with inline links instead of leaving bare links:
Read the Sphinx Tutorial <http://sphinx.pocoo.org/tutorial.html>_
Avoid using labels such as click here, this, etc. preferring descriptive labels (SEO worthy) instead.
6.8.6 Linking to Sections in The Guide

To cross-reference other parts of this documentation, use the :ref: keyword and labels.
To make reference labels more clear and unique, always add a -ref suffix:
.. _some-section-ref:
Some Section
------------
6.8.7 Notes and Warnings

Make use of the appropriate admonitions directives when making notes.
Notes:
.. note::
The Hitchhikers Guide to the Galaxy has a few things to say
on the subject of towels. A towel, it says, is about the most
massively useful thing an interstellar hitch hiker can have.
Warnings:
.. warning:: DONT PANIC
6.8.8 TODOs
Please mark any incomplete areas of The Guide with a todo directive. To avoid cluttering the Todo List, use a single
todo for stub documents or large incomplete sections.
86
.. todo::
Learn the Ultimate Answer to the Ultimate Question
of Life, The Universe, and Everything
6.8. The Guide Style Guide
87
88
Index
E
environment variable
PATH, 5, 7
P
PATH, 5, 7
Python Enhancement Proposals
PEP 1, 76
PEP 20, 22, 77
PEP 249, 47
PEP 257, 28, 77
PEP 3101, 17
PEP 3132, 21
PEP 3333, 38
PEP 8, 23, 67, 77, 79
PEP 8#comments, 28
89

Python-Guide by Kenneth

Uploaded by

Python-Guide by Kenneth

Uploaded by

Python Guide Documentation

October 15, 2014

Writing Great Code

Shipping Great Code

Python Guide Documentation, Release 0.0.1

Welcome to The Hitchhikers Guide to Python.

Python Guide Documentation, Release 0.0.1

This part of the guide focuses on setting up your Python environment.

1.1 Picking an Interpreter

Python Guide Documentation, Release 0.0.1

Chapter 1. Getting Started

Python Guide Documentation, Release 0.0.1

Properly Install Python

1.2 Installing Python on Mac OS X

1.2. Installing Python on Mac OS X

Python Guide Documentation, Release 0.0.1

Now, we can install Python 2.7:

This will take a minute or two.

1.2.2 Setuptools & Pip

1.3 Installing Python on Windows

Chapter 1. Getting Started

Python Guide Documentation, Release 0.0.1

You can do this easily by running the following in powershell:

[Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27\;C:\Python27\Scripts\", "Us

1.3.1 Setuptools + Pip

1.3. Installing Python on Windows

Python Guide Documentation, Release 0.0.1

> virtualenv venv

1.4 Installing Python on Linux

1.4.1 Setuptools & Pip

Chapter 1. Getting Started

Python Guide Documentation, Release 0.0.1

$ pip install virtualenv

1.4. Installing Python on Linux

Python Guide Documentation, Release 0.0.1

Chapter 1. Getting Started

Writing Great Code

2.1 Structuring Your Project

2.1.1 Structure is Key

Python Guide Documentation, Release 0.0.1

Chapter 2. Writing Great Code

Python Guide Documentation, Release 0.0.1

# sqrt is visibly part of modus namespace

2.1.4 Object-oriented programming

2.1. Structuring Your Project

Python Guide Documentation, Release 0.0.1

Chapter 2. Writing Great Code

Python Guide Documentation, Release 0.0.1

2.1.6 Dynamic typing

Python Guide Documentation, Release 0.0.1

2.1.7 Mutable and immutable types

# The new x is another object

Chapter 2. Writing Great Code

Python Guide Documentation, Release 0.0.1

2.1.8 Vendorizing Dependencies

2.2 Code Style

2.2.1 General concepts

2.2. Code Style

Python Guide Documentation, Release 0.0.1

Chapter 2. Writing Great Code

Python Guide Documentation, Release 0.0.1

2.2. Code Style

Python Guide Documentation, Release 0.0.1

exception might be better

Chapter 2. Writing Great Code

Python Guide Documentation, Release 0.0.1

You can use this to swap variables as well:

Nested unpacking works too:

In Python 3, a new method of extended unpacking was introduced by PEP 3132:

Create an ignored variable

2.2. Code Style

Python Guide Documentation, Release 0.0.1

Create a length-N list of the same thing

Create a length-N list of lists

A common idiom for creating strings is to use str.join() on an empty string.

2.2.3 Zen of Python