Sold to

steven07kolesiko@gmail.com
Contents
Preface 3
What makes you good . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Appreciation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Chapter 1: Projects 5
Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Structure your project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
The reasoning behind a src directory . . . . . . . . . . . . . . . . . . . . . . . . . . 5
How to name files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Real life example when naming modules . . . . . . . . . . . . . . . . . . . . . . . . 8
Naming classes, functions, and variables . . . . . . . . . . . . . . . . . . . . . . . . 9
When to create a function or a class? . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Creating modules and entry points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Defining main for modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Structure your code after the problem it solves . . . . . . . . . . . . . . . . . . . . . . . 15
Don’t structure it after patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
When importing packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Making repetitive imports less boring . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Managing dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Stop using requirements.txt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Poetry: the red pill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
pyproject.toml vs requirements.txt . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Install your code locally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Helper scripts and tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Dealing with simple scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Dealing with complex scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Use Linters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Linter recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Recommended settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Enforce linters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Directory tree from a real company . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Templates for you . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

2
Preface
What makes you good
“ There are 10 kinds of people. Those who understand binary and
those who doesn’t. ”
What I don’t like about this phrase is that while it’s funny for those who understand that 10 is the
binary representation for 2 , it lets the other group of people (those who don’t know it) clueless.
Notice how each number can be summed up producing a total different decimal number:

When set 0 0 0 0 0 0 1 0
Then sum 128 64 32 16 8 4 +2 1

It means that 11 (2 + 1) would represent 3 kinds of people:

When set 0 0 0 0 0 0 1 1
Then sum 128 64 32 16 8 4 +2 +1

while 101 (4 + 1) would be 5 types of people:

When set 0 0 0 0 0 1 0 1
Then sum 128 64 32 16 8 +4 2 +1

Hopefully, it makes sense now and that it wouldn’t be hard for you to find out the decimal for 1010
.
Analogously I want to give you meaningful instructions so Python gets as easy as what you just read.

“ Any fool can write code that a computer can understand. Good
programmers write code that humans can understand. ” - Martin
Fowler
Again we split the reader into two possible groups: fools and good programmers.
I can’t tell you who you are, but I can guarantee you that I want to transform you into the latter
group and I’ll use this book to do that.
If we keep thinking about it we realize that all good programmers start as fools.
We all need to write some code that can be read by the computer, but what will turn you into a good
programmer is the act of evolving until humans can understand it.

Fools (Phase 1) Good Programmers (Phase 2)

Computer compiles and runs your code. Your colleagues read and maintain your code.

It took me a while to get it.

I spent months of my life thinking I should be learning more programming languages to become a
great developer, but turns out I should be sticking with as few languages as possible and just honing
the way I create software.
Let’s rephrase that and just say:

3
 “ There are 2 kinds of people. Those who write code and those
who build software ”
Code is a hobby. Software is paid.
Code runs. Software is maintained.
Code doesn’t require much planning. Software is designed.
Code is hard to change. Software scales.
There might be many books around there teaching you to write Python, but I don’t see many
teaching people how to build software with Python.
You can bet I’m going to teach you how to build great software with Python.
In Python we call it the Pythonic way of doing things.
2022, July 04,
Guilherme Magalhães Latrova (aka Gui)

Appreciation
I walked many years feeling lost in life, career, and purpose. I thank God who found me and took
care of me.
I thank my wife Thais Cabral that joined me in taking many risks and made me a better man.
Jonathan Tolentino Abila who listened to my complaints and made the past years working with
terrible code bearable.
I thank David Grandes that (out of his normal senses) hired a 26 dude to be a tech lead for the first
time.
Finally, I thank Kwan Lee that didn’t send me away when I applied for a role requiring 10+ years of
experience having less than 5. Actually I’m grateful that he had a call with me, gave me a harder
challenge, and decided to hire me for a lower position. Thank you for giving me a shot.

4
Chapter 1: Projects
Guidelines
“ For every minute spent in organizing, an hour is earned. ” -
Benjamin Franklin
Python is different from languages like C# or Java where they enforce you to have classes named
after the file they live in.
So far Python is one of the most flexible languages I had contact with and everything too flexible
enhances the odds of bad decisions.
• Do you want to keep all project classes in a single main.py file? Yes, it works.
• Do you need to read an os environment var? Just read it right there.
• Do you need to modify a function behavior? Why not a decorator!?
Many decisions that are easy to implement may backfire producing code that is extremely
hard to maintain.
This is not necessarily bad if you know what you’re doing.
During this chapter, I’m going to present to you guidelines that worked for me over the past working
in different companies and with many different people.

Structure your project

Let’s focus first on directory structure, file naming, and module organization.
I recommend you to keep all your module files inside a src dir, and all tests living side by side with
it:
Top-Level project
<project>
├── src
│ ├── <module>/*
│ │ ├── __init__.py
│ │ └── many_files.py
│ │
│ └── tests/*
│ └── many_tests.py
│
├── .gitignore
├── pyproject.toml
└── README.md

Where <module> is your main module. If in doubt, consider what people would pip install and
how you would like to import module .
Frequently it has the same name as the top project. This isn’t a rule though.

The reasoning behind a src directory

I’ve seen many projects doing differently.

Some variations include no src dir with all project modules around the tree.

5
This is quite annoying because of the lack of order, producing structures like pytorch:
pytorch
├── android
├── aten
├── benchmarks
├── binaries
├── c10
├── caffe2
├── cmake
├── docs
├── functorch
├── ios
├── modules
├── mypy_plugins
├── scripts
├── test
├── third_party
├── tools
├── torch
├── torchgen
├── aten.bzl
├── buckbuild.bzl
├── BUCK.oss
├── BUILD.bazel
├── build.bzl
├── build_variables.bzl
├── c2_defs.bzl
├── c2_test_defs.bzl
├── CITATION
├── CMakeLists.txt
├── CODE_OF_CONDUCT.md
├── CODEOWNERS
├── CONTRIBUTING.md
├── defs.bzl
├── defs_gpu.bzl
├── defs_hip.bzl
├── Dockerfile
├── docker.Makefile

6
├── GLOSSARY.md
├── LICENSE
├── Makefile
├── MANIFEST.in
├── mypy.ini
├── mypy-strict.ini
├── NOTICE
├── pt_ops.bzl
├── pt_template_srcs.bzl
├── pyproject.toml
├── pytest.ini
├── README.md
├── RELEASE.md
├── requirements-flake8.txt
├── requirements.txt
├── SECURITY.md
├── setup.py
├── ubsan.supp
├── ufunc_defs.bzl
├── version.txt
└── WORKSPACE

It’s boring to have things so apart due to the alphabetical sorting of the IDE.
The main reason behind the src dir is to keep active project code concentrated inside a single
directory while settings, CI/CD setup, and project metadata can reside outside of it. Take the lightning
AI as example.
The only drawback of doing it is that you can’t import module_a in your python code out of the
box. We need to set up the project to be installed under this repository. We’re going to discuss how
to solve this soon in this chapter.

How to name files

Rule 1: Files are seen as modules First of all, in Python “files” are not just “files” and I noticed
this is the main source of confusion for beginners.
If you’re inside a directory that contains any __init__.py it’s a directory composed of modules,
not files.
See each module as a namespace.
I mean namespace because you can’t say for sure whether they have many functions, classes, or
just constants. It can have virtually all of them or just a bunch of some.

Rule 2: Keep things together as needed It’s fine to have several classes within a single
module, and you should do so. (when classes are related to the module, obviously.)
Only break it down when your module gets too big, or when it handles different concerns.

7
 Figure 1: One does not simply create a file in Python

Often, people think it’s a bad practice due to some experience with other languages that enforce the
other way around (e.g. Java and C#).

Rule 3: By default give plural names As a rule of thumb, name your modules in the plural and
name them after a business context (more on it soon).
There’re exceptions to this rule though! Modules can be named core , main.py , and similar to
represent a single thing. Use your judgment, if in doubt stick to the plural rule.

Real life example when naming modules

I’ll share a Google Maps Crawler project that I built as an example.

This project is responsible for crawling data from Google Maps using Selenium and outputting it (Read
more here if curious).
This is the current project tree outlining exceptions to the plural rule:
gmaps_crawler
├── src
│ └── gmaps_crawler
│ ├── __init__.py
│ ├── config.py ← (Singular)
│ ├── drivers.py
│ ├── entities.py
│ ├── exceptions.py

8
│ ├── facades.py
│ ├── main.py ← (Singular)
│ └── storages.py
│
├── .gitignore
├── pyproject.toml
└── README.md

It seems very natural to import classes and functions like:

from gmaps_crawler.storages import get_storage
from gmaps_crawler.entities import Place
from gmaps_crawler.exceptions import CantEmitPlace

I can understand that I might have one or many exception classes inside exceptions and so on.
The beauty about having plural modules is that:
• They’re not too small (e.g. one per class)
• You can at any moment break it down into smaller modules if required
• They give you a strong sense of knowing what might exist inside

Naming classes, functions, and variables

Some people claim naming things is hard. It gets less hard when you define some guidelines.

Functions and Methods should be verbs Functions and methods represent an action or action-
able stuff. Something “isn’t”. Something is “happening”.
Actions are clearly stated by verbs.
A few good examples from REAL projects I worked on before:
def get_orders():
...

def acknowledge_event():
...

def get_delivery_information():
...

def publish():
...

A few bad examples:

def email_send():
...

def api_call():
...

def specific_stuff():
...

9
They’re a bit unclear whether they return an object to allow me to perform the api call or if it actually
send an email for example.
I can picture a scenario like this:
# Example of a misleading function name
email_send.title = "title"
email_send.dispatch()

Exceptions to this rule are just a few but they exist.

• Creating a main() function to be invoked in the main entry point of your application is a good
reason to skip this rule.
• Using @property to treat a class method as an attribute is also valid.

Variables and Constants should be nouns Should always be nouns, never verbs (which clarifies
the difference between functions).
Good examples:
plane = Plane()
customer_id = 5
KEY_COMPARISON = "abc"

Bad examples:
fly = Plane()
get_customer_id = 5
COMPARE_KEY = "abc"

If your variable/constant is a list or collection, make it plural!

planes: list[Plane] = [Plane()] # ← Even if it contains only one item
customer_ids: set[int] = {5, 12, 22}
KEY_MAP: dict[str, str] = {"123": "abc"} # ← Dicts are kept singular

Classes should be self explanatory, but Suffixes are fine Prefer classes with self explanatory
names. It’s fine to have suffixes like Service , Strategy , Middleware , but only when extremely
necessary to make its purpose clear.
Always name it in singular instead of plural. Plural reminds us of collections (e.g. if I read orders
I assume it’s a list or iterable), so remind yourself that once a class is instantiated it becomes a single
object.
Classes representing entities
Classes that represent things from the business context should be named as is (nouns!). Like Order
, Sale , Store , Restaurant and so on.
Example of suffixes usage
Let’s consider you want to create a class responsible for sending emails. If you name it just as “
Email ”, its purpose is not clear.
Someone might think it may represent an entity e.g.
email = Email() # inferred usage example
email.title = "Title"
email.body = create_body()
email.send_to = "guilatrova.dev"

send_email(email)

10
You should name it “ EmailSender ” or “ EmailService ”.

Casing conventions By default follow these naming conventions:

Type Public Internal

Packages (directories) lower_with_under -
Modules (files) lower_with_under.py -
Classes CapWords -
Functions and methods lower_with_under() _lower_with_under()
Constants ALL_CAPS_UNDER _ALL_CAPS_UNDER

Even though Python doesn’t support constants we can still use mypy to enforce the value won’t
change (more on type hinting later).

Disclaimer about “““private”“” methods. Some people found out that if you have
__method(self) (any method starting with two underscores) Python won’t let outside
classes/methods invoke it normally which leads them to think it’s fine.
The leading __ is built to ensure subclassers can’t override that method, so subclasses can’t even
call super().__method which may be confusing.
Take this code as example:
class ParentClass:
def call_me(self) -> str:
return self.__call_me_maybe()

def __call_me_maybe(self) -> str:

return (
"Hey, I just met you, and this is crazy\n"
"But here's my code, so call me, maybe"
)

class ChildClass(ParentClass):
def call_me(self) -> str:
return self.__call_me_maybe() + ( # ← Exception :(
"You took your time with the call"
"I took no time with the fall"
"You gave me nothing at all"
)

print("Parent call_me output:")

print(ParentClass().call_me())
print()

print("Child call_me output:")

print(ChildClass().call_me())

It gives as output:
Parent call_me output:
Hey, I just met you, and this is crazy
But here's my code, so call me, maybe

Child call_me output:

11
Traceback (most recent call last):
File "private_dunder.py", line 25, in <module>
print(ChildClass().call_me())
File "private_dunder.py", line 13, in call_me
return self.__call_me_maybe() + ( # ← Exception :(
AttributeError: 'ChildClass' object has no attribute '_ChildClass__call_me_maybe'.
Did you mean: '_ParentClass__call_me_maybe'?

By reading the error message we learn that we can do:

print(ParentClass()._ParentClass__call_me_maybe())

and it works… So tell me: Does it look “private” to you?

Figure 2: You trying to protect your class

If you came from a C# environment like myself it might sound weird that you can’t protect a method.
But Guido (Python’s creator) has a good reason behind it:
“We’re all consenting adults here”
It means that if you’re aware you shouldn’t be invoking a method, then you shouldn’t unless you
know what you’re doing.
After all if you really decided to invoke that method, you’re going to do something dirty to make it
happen (known as “Reflection” in C#).
Mark your private method/function with a single initial underscore to state it’s intended for private
use only and live with it.

When to create a function or a class?

This is a common question I received a few times.

12
If you follow the above recommendations you’re going to have clear modules and clear modules are
an effective way to organize functions:
from gmaps_crawler import storages

storages.get_storage() # ← Similar to a class, except it's not instantied and has a plural na
storages.save_to_storage() # ← Potential function inside module

Sometimes you can identify subsets of functions inside a module. When this happens a class makes
more sense:

Example on grouping different subset of functions Consider the same storages module
with 4 functions:
def format_for_debug(some_data):
...

def save_debug(some_data):
"""Prints in the screen"""
formatted_data = format_for_debug(some_data)
print(formatted_data)

def create_s3(bucket):
"""Create s3 bucket if it doesn't exists"""
...

def save_s3(some_data):
s3 = create_s3("bucket_name")
...

S3 is a cloud storage to store any sort of data provided by Amazon (AWS). It’s like Google Drive for
software.
We can say that:
• The developer can save data in DEBUG mode (that just prints on the screen) or on S3 (that
stores data on the cloud).
• save_debug uses the format_for_debug function
• save_s3 uses the create_s3 function
I can see two groups of functions and no reason to keep them in different modules as they seem
small, thus I’d enjoy having them defined as classes:
class DebugStorage:
def format_for_debug(self, some_data):
...

def save_debug(self, some_data):

"""Prints in the screen"""
formatted_data = self.format_for_debug(some_data)
print(formatted_data)

class S3Storage:
def create_s3(self, bucket):
"""Create s3 bucket if it doesn't exists"""
...

13
 def save_s3(self, some_data):
s3 = self.create_s3("bucket_name")
...

Here’s a rule of thumb:

• Always start with functions
• Grow to classes once you feel you can group different subsets of functions

Creating modules and entry points

Every application has an entry point.
It means that there’s a single module (aka file) that runs your application. It can be either a single
script or a big module.
Whenever you’re creating an entry point, make sure to add a condition to ensure it’s being exe-
cuted and not imported:
def execute_main():
...

if __name__ == "__main__": # ← Add this condition

execute_main()

By doing that you ensure that any imports won’t trigger your code by accident. Unless it’s
explicitly executed.

Defining main for modules

You might have noticed some python packages that can be invoked by passing down -m like:
python -m pytest
python -m tryceratops
python -m faust
python -m flake8
python -m black

Such packages are treated almost like regular commands since you can also run them as:
pytest
tryceratops
faust
flake8
black

To make this happen you need to specify a single __main__.py file inside your main module:
<project>
├── src
│ ├── example_module ← Main module
│ │ ├── __init__.py
│ │ ├── __main__.py ← Add it here
│ │ └── many_files.py
│ │
│ └── tests/*

14
│ └── many_tests.py
│
├── .gitignore
├── pyproject.toml
└── README.md

Don’t forget you still need to include the check __name__ == "__main__" inside your __main__.py
file.
When you install your module, you can run your project as python -m example_module .
NOTE: If you’re willing to create a CLI and distribute it, you should follow these recommendations.
We’re going to use Typer later in this book to achieve that (TODO: Link).

Structure your code after the problem it solves

We know how to name things and when to create classes or functions, but when to create different
modules?
I propose a simplification of the Domain Driven Design. In DDD we name our code and classes as
close to the domain as possible. It means that if you deal with orders and receipts you potentially
would want your code to represent the Order , the Printer , etc.
Let’s consider a domain where I have to deal with the following:

Domain Description
Orders Defines the Order class, handles Product creation and stores them in the
database
Sales Defines the Sale class, which process orders, prints them, and syncs with the IRS
system
Restaurants Manages Restaurant classes, syncs opening hours to apps like DoorDash, and
manages stock for specific products

Note the way I grouped is arbitrary based on my vision of the problem. You could have defined
that Product is relevant enough to have its own module. Alternatively, you could have defined
restaurants are too tied to every sale, thus they should be together.
Even better: Always start with a single module named after the biggest problem (for this
example either orders or sales), evolve as it gets big/handles too many concerns.
Considering our system got large that’s how I see it:
It makes wonders to name modules after their purpose or design pattern (TODO: later in
this book add links).
Given the problem above I’d like to challenge you to guess what would reside inside each module.
It’d be fine if you’re not familiar with some design patterns like the mediator, we’re going to know a
few later in this book.
sales_proj
├── src
│ └── sales_proj
│ ├── __init__.py
│ ├── __main__.py
│ │

15
 Figure 3: Problem structure

│ ├── orders
│ │ ├── models
│ │ │ ├── orders.py
│ │ │ └── products.py
│ │ │
│ │ └── exceptions.py
│ │
│ ├── restaurants
│ │ ├── services.py
│ │ ├── clients.py
│ │ └── models.py
│ │
│ └── sales
│ ├── services
│ │ ├── products.py
│ │ └── printers.py
│ │
│ ├── mediators.py
│ └── exceptions.py
│
├── .gitignore

16
├── pyproject.toml
└── README.md

Here’re some terms you might be unfamiliar with.

• Service (from DDD) represents a set of business rules (e.g. Connecting to the printer, actually
printing our order).
• Model (aka Entity) (from DDD and some frameworks) represents a business “thing” (e.g. Prod-
uct), it’s frequently tied with an ORM and saved to the database.
• Clients/Mediators/Exceptions organizes HTTP Client to connect to DoorDash, Design Pattern,
and potential exceptions respectively.
I’m not going to discuss design patterns in this chapter so we don’t lose focus.
Here is some stuff I can imply from a few specific directories:

Module What’s inside (inferred by name)

orders/models/ Many models related to Order and Product since its split into two
smaller modules (e.g. relationship between order and product)
restaurants/clients.py HTTP/API authentication, endpoints, payloads to interact with Doordash
restaurants/services.py Business logic managing our Doordash client defined in clients.py
sales/services/ Business logic for different problems, product stocks, and printer

Don’t structure it after patterns

This idea reminds me of the MVC pattern where you would split your code into Model, View, and
Controller.
Back when I worked with this pattern, many changes happening inside the View would impact the
Model, and finally the Controller. It was annoying to keep bouncing back and forth between distant
files.
Using our example it would represent:
antipattern
├── src
│ └── antipattern
│ ├── __init__.py
│ ├── __main__.py
│ │
│ ├── models
│ │ ├── orders.py
│ │ ├── restaurants.py
│ │ └── products.py
│ │
│ ├── exceptions
│ │ ├── restaurants.py
│ │ ├── sales.py
│ │ └── orders.py
│ │

17
│ ├── services
│ │ ├── sales
│ │ │ ├── products.py
│ │ │ └── printers.py
│ │ └── restaurants.py
│ │
│ └── (potentially more files or modules)
│
├── .gitignore
├── pyproject.toml
└── README.md

Some reasons why this is bad:

• You don’t have a package that resolves a single problem, you have groups of packages where
they need each other to do pretty much everything
• Names get repetitive or too verbose (e.g. order_service, order_model, etc)
• Pretty much any work you do will touch all modules

When importing packages

Python allows relative importing by prefixing paths with “ . ” (dot).
It can get complex to navigate yourself.
Let’s consider a fictitious structure:
# File located in src/managers/services/handlers/issues.py
from ...clients import MyClient
from ...models import MyModel

...

Even though the above code works it’s hard to know where these classes came from.
Here’s a rule of thumb to decide when to import packages:
• Only use relative imports when the file is in the same folder;
• For everything else, be explicit;
So considering we had the following:
# File located in src/managers/services/handlers/issues.py
from .clients import MyClient
from .models import MyModel

I can tell you right away that clients.py and models.py live side by side with issues.py , so
they’re located inside src/managers/services/handlers .

Making repetitive imports less boring

When organizing code it might feel like we’re sacrificing typing speed in favor of organization.
There’s a trick to overcome this.
Let’s imagine yet another scenario.

18
Inside our gmaps_crawler project we started having too many entities and it’s time to break it
down into three smaller modules: places.py , restaurants.py , reviews.py .
gmaps_crawler
├── src
│ └── gmaps_crawler
│ ├── __init__.py
│ ├── config.py
│ ├── drivers.py
│ ├── entities ← Module became a dir
│ │ ├── places.py
│ │ ├── restaurants.py
│ │ └── reviews.py
│ │
│ ├── exceptions.py
│ ├── facades.py
│ ├── main.py
│ └── storages.py
│
├── .gitignore
├── pyproject.toml
└── README.md
It would cause code like:
from gmaps_crawler.entities import Place, Restaurant, Review

To become:
from gmaps_crawler.entities.places import Place
from gmaps_crawler.entities.restaurants import Restaurant
from gmaps_crawler.entities.reviews import Review

That sucks, right? We just wanted to reorganize! Not rewrite the imports.

Using __init__.py to keep short imports If we create an __init__.py inside entities we

can keep the same behavior as long as we add this inside:
# File located in entities/__init__.py
from .restaurants import Restaurant
from .places import Place
from .reviews import Review

# Sometimes you need to disable linters for this file.

Expected directory structure:

gmaps_crawler
├── src
│ └── gmaps_crawler
│ ├── __init__.py

19
│ ├── config.py
│ ├── drivers.py
│ ├── entities
│ │ ├── __init__.py ← Magic
│ │ ├── places.py
│ │ ├── restaurants.py
│ │ └── reviews.py
│ │
│ ├── exceptions.py
│ ├── facades.py
│ ├── main.py
│ └── storages.py
│
├── .gitignore
├── pyproject.toml
└── README.md
The imports now can become simple again. You’re welcome!
from gmaps_crawler.entities import Place, Restaurant, Review # __init__.py magic

Managing dependencies
You’re going to end up having to install some libraries regardless of the project.
I can mention some like rich , pytest , boto3 , django , fastapi , tryceratops , mypy
, etc.
We use frameworks and tools to speed up our environment.

Stop using requirements.txt

Let’s go straight to the point. requirements.txt is not good enough for your daily needs.
Here’s the process that some people follow:
1. pip install <lib> (Responsible for actually installing)
2. pip freeze > requirements.txt (Responsible for capturing installed versions and outputting
them to a file)
3. pip install -r requirements.txt (Installing all captured packages)
Over time it becomes a nightmare because:
• You have no clarity about what’s an intended dependency and what’s added as “sub depen-
dency”
• You need to manually update the requirements.txt (and remember your team to do the
same)
• If uninstalling you must do the same
• You will end up packaging everything for production since you don’t know which packages are
intended for dev usage and which for production usage
This file grows beyond the comprehension of anyone.

20
Example of requirements.txt Here’s an example. I’ll be installing a few libs and showing how
requirements.txt starts simple and grows.
After installing Rich
❯ pip install rich
...

❯ pip freeze
commonmark==0.9.1
Pygments==2.12.0
rich==12.4.4

After installing Pytest

❯ pip install pytest
...

❯ pip freeze
attrs==21.4.0
commonmark==0.9.1 # ← From Rich
iniconfig==1.1.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
Pygments==2.12.0 # ← From Rich
pyparsing==3.0.9
pytest==7.1.2
rich==12.4.4 # ← From Rich
tomli==2.0.1

After installing boto3

This time I’ll let you try to remember. Which dependencies were added?
Do you think other team members can track it after 1 month? What about 1 year?
❯ pip install boto3
...

❯ pip freeze
attrs==21.4.0
boto3==1.24.25
botocore==1.27.25
commonmark==0.9.1
iniconfig==1.1.1
jmespath==1.0.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
Pygments==2.12.0
pyparsing==3.0.9
pytest==7.1.2
python-dateutil==2.8.2
rich==12.4.4
s3transfer==0.6.0
six==1.16.0
tomli==2.0.1
urllib3==1.26.10

21
Poetry: the red pill

Here, take it:

Figure 4: Poetry on the left, a text file on the right

Take the red pill and stay with me. We’re going to add the same libs and see how our project metadata
grows.
You still need to manually install poetry though, let’s do that: pip install poetry
And now do the following:
❯ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [poetrylibs]: python-like-pro

Version [0.1.0]:
Description []: I will never use requirements.txt again
Author [Guilherme Latrova <hello@guilatrova.dev>, n to skip]:
License []: MIT
Compatible Python versions [^3.10]:

From now on let’s define that

• boto3 is the main dependency (i.e. it will be packaged for production)
• rich , and pytest are dev dependencies (i.e. only installed locally on dev machines)
Would you like to define your main dependencies interactively? (yes/no) [yes]
You can specify a package in the following forms:
- A single name (requests)
- A name and a constraint (requests@^2.23.0)
- A git url (git+https://github.com/python-poetry/poetry.git)
- A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)

22
 - A file path (../my-package/my-package.whl)
- A directory (../my-package/)
- A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Search for package to add (or leave blank to continue): boto3

Found 20 packages matching boto3

Enter package # to add, or the complete package name if it is not listed:

[0] boto3
[1] boto4
[2] boto
[3] krux-boto
[4] boto_utils
[5] boto3_retry
[6] boto3-meiqia
[7] flake8-boto3
[8] CloeePy-Boto
[9] boto3r
> 0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^1.24.25 for boto3

Add a package:

Would you like to define your development dependencies interactively? (yes/no) [yes]
Search for package to add (or leave blank to continue): rich
Found 20 packages matching rich

Enter package # to add, or the complete package name if it is not listed:

[0] rich
[1] rich-click
[2] rich-admonitions
[3] flask-rich
[4] rich-demo
[5] rich-rst
[6] nornir-rich
[7] rich-utils
[8] ylq-rich
[9] rich-msa
> 0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^12.4.4 for rich

Add a package: pytest

Found 20 packages matching pytest

Enter package # to add, or the complete package name if it is not listed:

[0] pytest
[1] pytest123
[2] 131228_pytest_1
[3] pytest-symbols
[4] pytest-grpc
[5] pytest-level
[6] sphinx_pytest
[7] pytest-vnc
[8] exgrex-pytest

23
 [9] pytest-faker
> 0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^7.1.2 for pytest

Add a package:

Generated file

[tool.poetry]
name = "python-like-pro"
version = "0.1.0"
description = "I will never use requirements.txt again"
authors = ["Guilherme Latrova <hello@guilatrova.dev>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.10"
boto3 = "^1.24.25"

[tool.poetry.dev-dependencies]
rich = "^12.4.4"
pytest = "^7.1.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Do you confirm generation? (yes/no) [yes]

Note you just generated an initial pyproject.toml file. You didn’t install anything yet.
Let’s see what’s inside first!
[tool.poetry]
name = "python-like-pro"
version = "0.1.0"
description = "I will never use requirements.txt again"
authors = ["Guilherme Latrova <hello@guilatrova.dev>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.10"
boto3 = "^1.24.25"

[tool.poetry.dev-dependencies]
rich = "^12.4.4"
pytest = "^7.1.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

There’re two clear groups: tool.poetry.dependencies and tool.poetry.dev-dependencies

. It’s interesting that inside tool.poetry.dependencies poetry was able to determine and
enforce our Python version, so we keep the whole team consistently using the same version.

24
Run poetry install when you feel ready, and you’re going to notice a new generated file named:
poetry.lock .
This file holds all the boring and hard to understand stuff.
Let’s point out some relevant stuff from the first section:
[[package]]
name = "atomicwrites" # <-- ← Package name
version = "1.4.0" # <-- ← Version used name
description = "Atomic file writes."
category = "dev" # <-- ← Is it going to prod or be in dev?
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"

The best part is that you don’t have to deal with it ever!

Adding new dependencies Before adding anything new you must ask yourself: Do my final
user need it?
I bet your user won’t need pytest and linters. Thus these would be dev dependencies.
Anything else required for your software to be executed becomes a production dependency.
Add production dependencies
Whenever you want to add a new dependency intended to go to production, you run:
poetry add <lib> , for example: poetry add fastapi
Note how either pyproject.toml and your package.lock gets updated.
Add dev dependencies
Whenever you want to add a new dependency intended to be used locally across the team, you run:
poetry add -D <lib> , for example: poetry add -D tryceratops

pyproject.toml vs requirements.txt

You might think pyproject.toml is Poetry’s feature.

Let me stop you right there because it’s not.
PEP 518 introduced pyproject.toml as a configuration file to specify build system requirements.
Poetry just follows the standard and many other tools (like linters as we’re going to see soon) do the
same.
We can outline a few differences:

Feature requirements.txt pyproject.toml

Contents Bare textfile Project Metadata
Maintenance Needs to be manually Automatically updated when using tools like
maintained Poetry
Purpose Only holds dependency versions Can be used to setup package deployment
to Pypi, testing, linting, and more

Install your code locally

If you keep your code under src as suggested you might end up importing things like from
src.bla import Bla .

25
This is annoying. Don’t do that.
Poetry allows you to install your package locally so you can import without the src prefix.
You just need to add packages under tool.poetry :
[tool.poetry]
name = ">>> YOUR PROJECT NAME"
version = "0.1.0"
authors = ["Guilherme Latrova <your@email.com>"]
description = ">>> YOUR PROJECT DESCRIPTION"
license = ">>> LICENSE"
packages = [
{ include = ">>> YOUR PACKAGE NAME", from = "src" }, # ← This is the trick
]

So whenever you or your team runs poetry install it will automatically handle it for you.

Helper scripts and tasks

What are your most used commands in the terminal?
Maybe it’s git . Or maybe you like to run tests a lot with pytest .
Beyond these commands, you might need to run longer commands when working with complex
microservices or apps for either the initial setup or to execute different aspects of your system.
Sharing these commands with your team also speeds up the development and onboarding of new
team members.

Dealing with simple scripts

For simpler scripts I recommend either: bin or a scripts directory at the root.
Be my guest to come up with other names. As long as it works for you and your team that’s enough.
It would introduce the following directory:
gmaps_crawler
├── bin ← New directory
│ ├── setup.sh ← Potential command to setup project
│ └── linters.sh ← Command to run all linters in the project
│
├── src
│ └── gmaps_crawler
│ ├── __init__.py
│ ├── config.py
│ ├── drivers.py
│ ├── entities.py
│ ├── exceptions.py
│ ├── facades.py
│ ├── main.py
│ └── storages.py
│

26
├── .gitignore
├── pyproject.toml
└── README.md

These scripts should be easy to be executed by running by simply typing: ./bin/linters.sh .

I already created scripts to generate migrations, send messages to Kafka for specific workflows, run
test coverage, and even install some stuff required to bootstrap the application.
I can’t tell you what scripts you need because it changes for every project.

Dealing with complex scripts

The above recommendation won’t work well in case you need to write longer/complex scripts or even
reuse Python code to generate whatever you need.
Like when you want to reuse your ORM to store data into the database, or even reuse some
Enum/Constant you already defined in code.
If that’s your case you should ditch your simple scripts dir and use something more robust like
pyinvoke .
When using Pyinvoke you need to create a tasks.py file at the root:
gmaps_crawler
├── src
│ └── gmaps_crawler
│ ├── __init__.py
│ ├── config.py
│ ├── drivers.py
│ ├── entities.py
│ ├── exceptions.py
│ ├── facades.py
│ ├── main.py
│ └── storages.py
│
├── tasks.py ← Required for Pyinvoke
├── .gitignore
├── pyproject.toml
└── README.md

A simple tasks.py example would be:

@task
def clean(c, docs=False, bytecode=False, extra=''):
patterns = ['build']
if docs:
patterns.append('docs/_build')
if bytecode:
patterns.append('**/*.pyc')
if extra:
patterns.append(extra)

27
 for pattern in patterns:
c.run("rm -rf {}".format(pattern))

Running inv clean would trigger the task above.

The best part is that your team can run inv -l (shorthand for invoke --list ) to see all available
tasks.
If you want to expand it into many directories, you can:
1. Make tasks a directory
2. Add modules inside tasks
3. Load collections inside tasks/__init__.py
The structure would be like this:
gmaps_crawler
├── src
│ └── gmaps_crawler
│ ├── __init__.py
│ ├── config.py
│ ├── drivers.py
│ ├── entities.py
│ ├── exceptions.py
│ ├── facades.py
│ ├── main.py
│ └── storages.py
│
├── tasks ← Became a dir
│ ├── __init__.py ← Requires configuration
│ ├── stuff.py
│ └── more_stuff.py
│
├── .gitignore
├── pyproject.toml
└── README.md

Inside tasks/__init__.py you would have:

from invoke import Collection

from . import stuff, more_stuff

namespace = Collection(stuff, more_stuff)

And now, these tasks can be invoked by running it as:

inv stuff.command

Note: Pyinvoke is not a replacement for Typer or click Some people end up thinking Pyinvoke
is a CLI lib. It’s not.

28
That’s not its purpose! (Even though it looks like a regular CLI).
Think of Pyinvoke as a development’s command cheat sheet that is kept local and Typer/click as the
user interface that is deployed to the end user.

Use Linters
Have you ever tried asking your teammates to don’t write long lines and to clear up imports when
they’re not being used?
That’s easier said than done.
We can’t blame them. When focused on working, it’s hard to remember of amenities that keep
your code clean. That’s where linters help to warn the team and ensure we’re following the same
standards across the company.
Linters are tools that analyze your work, complain about inconsistencies, and suggest changes. Al-
most like our ex-girlfriends but focused on code.
The sooner you apply this to any project the better and the more consistent it will be.

Linter recommendations

You’re about to see a few recommendations. I’ve used them successfully in many past companies
and they worked fine to get us in sync allowing us to focus on what matters: producing code.

Linter Reasoning
black Formatting
flake8 General linting (e.g. unused variables)
isort Sort of imports
mypy Typing validation
tryceratops Avoid try/except anti-patterns
end-of-file-fixer / trailing-whitespace General clean up

We’re going to discuss the importance of type hinting (mypy) and good practices with try/except
(tryceratops) over the next chapters.

Recommended settings

The importance of cleaning your code Let’s face it. The more code you add the harder it is to
maintain.
We should strive for less code and more value. That’s efficiency. Doing more with less.
Linters like flake8 keep your code clear by removing unused code (imports, vars, etc).

The importance of formatting You might not agree with everything Black suggests (check out
Blue for a different take), but it’s undeniable that getting your team following the same set of rules
increases productivity.
I’ve seen many people adding blank lines and different spacing, only to later, someone else removes
them .
The goal of using a formatting linter is to agree on something without thinking.
To focus on the code and ignore the whitespaces and commas.
To not have to measure whether your line is too long.
If not black, then try blue, or anything else, just have your team agree on something.

29
Formatting: Line length PEP 8 suggests 79 characters line length while Black enforces 88 char-
acters by default.
Chances are your current monitor is bigger, so why restrain ourselves?
It gets even harder to respect <=88 length when we start adding type hints. See how easy it is to
go over the recommendation:
# ••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• (Black limit) ↓
# •••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• (PEP 8 limit) ↓
class OrderService:
def handle_client_order(self, client: Client, purchased_items: list[Purchases]) -> Order:
...

It led me and my colleagues to play with other numbers. We tried both 100 and 120. 120 is GREAT!
Use it.
Oh, and don’t be afraid of testing other numbers for your project. Don’t restrain yourself because of
PEP 8, Black, or myself. We’re all humans with opinions based on our experiences and so are you.
Start with a recommendation and formulate your own opinion!

Imports: Why sorting It looks good to the eyes to see imports grouped as standard lib, third-party,
and local packages, and sorted alphabetically.
You can find yourself faster as you navigate code and import stuff.

Set up pyproject.toml We can also use pyproject.toml to set up our settings (line length, and
etc).
Here’s what I recommend:
[tool.black]
line-length = 120 # ← Override default line length

[tool.isort]
profile = "black" # ← Matches internal configs with Black, so they don't fight each other
line_length = 120 # ← Required to sync with Black
# ↓ I always feel like pytest and pydantic are core libs (they aren't),
# ↓ so it feels natural to me to organize them together with os, unittest, etc
extra_standard_library = ["pytest", "pydantic"]

[tool.tryceratops]
ignore = ["TC002", "TC003"]

[tool.mypy]
python_version = "3.10"
warn_unused_configs = true
warn_unused_ignores = true
namespace_packages = true
explicit_package_bases = true
ignore_missing_imports = true
plugins = [
"pydantic.mypy"
]

Unfortunately, at the time of this writing flake8 doesn’t support pyproject.toml and their main-
tainers have no intentions of supporting it (yes, they’re opinionated as well!). So you need to create
a .flake8 file:

30
[flake8]
max-line-length=120 # ← Required to sync with Black/isort
extend-ignore=E203 # ← Required to sync with Black
exclude=__init__.py # ← Frequently I found myself willing to ignore __init__.py files

Enforce linters

(This section implies the reader is minimally familiar with git).

Now that your team agreed on something (finally!) you have to ensure they follow it.
pre-commit is a great tool to ensure that every code committed (before pushed) to the repository
follows the rules mentioned above.

Install and set up pre-commit Installing pre-commit is easy!

poetry add -D pre-commit
pre-commit install
Done. pre-commit is installed.
Create a .pre-commit-config.yaml file and add the following:
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.4.0
hooks:
# ↓ Nice extra file formatters
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/psf/black
rev: 22.3.0
hooks:
- id: black
- repo: https://gitlab.com/pycqa/flake8
rev: 4.0.1
hooks:
- id: flake8
exclude: samples/
- repo: https://github.com/timothycrosley/isort
rev: 5.10.1
hooks:
- id: isort
- repo: https://github.com/guilatrova/tryceratops
rev: v1.1.0
hooks:
- id: tryceratops
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.950
hooks:
- id: mypy
# ↓ Example of additional deps, only use those if you need
additional_dependencies: [
mypy-extensions==0.4.3,
types-requests==2.27.25,
]

Ensure the versions set in pre-commit are in sync with the versions set in pyproject.toml .

31
This needs to be done manually.
Once done you can execute it manually by:
pre-commit run -a
or commit any file and it will automatically run.

Guarantee linters are being followed No, you can’t trust your beautiful linters will be respected.
For a variety of reasons people disable them, forget to set them up, or simply don’t care.
You must use your Continuous Integration (aka CI) system to ensure it’s respected.
If you use GitHub, you can enforce your pre-commit rules are respected by using an action:
jobs:
lint:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3

# Set up python + install here

- name: Lint
uses: pre-commit/action@v2.0.3 # ← Here

Yes. That simple.

Directory tree from a real company

This is the full directory tree (with minimal redacts/renames) from a company I worked with over the
past.
It’s an abandoned project, so hopefully, I won’t receive any lawyers anytime soon. (Please, don’t tell
my boss anyway ).
This project is a full microservice responsible for integrating with a third-party named Onfleet.
Their integration would pick one of our couriers and assign them to deliver the order.
This project had to deal with:
• Receiving customer events (Customer ordered X at restaurant Y);
• Requesting Onfleet to pick the best courier for us based on location;
∘ It involves 2 steps to tell Onfleet:
∘ Pick up the order at the restaurant,
∘ Drop off the order at the customer’s home;
• Notifying the kitchen and customer the courier is on the way;
• Handling customer cancelations, by canceling Onfleet requests;
project
├── bandit.yaml
├── docker-compose.yml
├── Dockerfile
├── img
│ └── onfleet.jpg
│

32
├── pyproject.toml
├── README.md
├── scripts
│ ├── bootstrap.sh
│ ├── code-coverage.sh
│ ├── fixlinters.sh
│ ├── integration-test.sh
│ ├── linters.sh
│ ├── makemigrations.sh
│ ├── manage.sh
│ ├── migrate.sh
│ ├── payloads
│ │ └── task_sample.json
│ ├── produce_task.sh
│ ├── setup-env.sh
│ ├── shell.sh
│ ├── unit-test.sh
│ └── watch-test.sh
│
├── sonar-project.properties
└── src
├── consumer.py
├── __init__.py
├── manage.py
├── onfleet_adapter
│ ├── asgi.py
│ ├── __init__.py
│ ├── models.py
│ ├── settings
│ │ ├── base.py
│ │ ├── development.py
│ │ ├── __init__.py
│ │ ├── local.py
│ │ ├── production.py
│ │ └── staging.py
│ │
│ ├── urls.py
│ ├── views.py

33
│ └── wsgi.py
│
├── onfleet_tasks
│ ├── apps.py
│ ├── dtos.py
│ ├── exceptions
│ │ ├── base.py
│ │ ├── db.py
│ │ ├── general.py
│ │ ├── __init__.py
│ │ ├── interruption.py
│ │ └── state.py
│ ├── facades.py
│ ├── factories
│ │ ├── events.py
│ │ ├── __init__.py
│ │ └── payloads.py
│ ├── __init__.py
│ ├── listeners
│ │ ├── agents.py
│ │ ├── __init__.py
│ │ └── topics.py
│ ├── managers.py
│ ├── migrations
│ │ ├── 0001_initial.py
│ │ ├── 0002_featureflag.py
│ │ ├── 0003_auto_20201123_1925.py
│ │ ├── enums
│ │ │ ├── task_status
│ │ │ │ └── 0001.sql
│ │ │ └── task_type
│ │ │ └── 0001.sql
│ │ └── __init__.py
│ ├── models
│ │ ├── enums.py
│ │ ├── feature_flags.py
│ │ ├── __init__.py
│ │ └── tasks.py

34
│ ├── serializers.py
│ ├── services
│ │ ├── brokers.py
│ │ ├── __init__.py
│ │ ├── onfleet.py
│ │ └── webhooks.py
│ ├── urls.py
│ ├── utils.py
│ └── views.py
│
└── tests
├── conftest.py
├── fixtures
│ ├── address_incidents.py
│ ├── __init__.py
│ ├── onfleet.py
│ └── tasks.py
├── __init__.py
├── integration
│ ├── conftest.py
│ ├── __init__.py
│ ├── onfleet_tasks
│ │ ├── __init__.py
│ │ ├── services
│ │ │ ├── __init__.py
│ │ │ └── test_onfleet.py
│ │ └── test_views.py
│ └── test_onfleet_setup.py
├── unit
│ ├── __init__.py
│ └── onfleet_tasks
│ ├── exceptions
│ │ ├── __init__.py
│ │ ├── test_db.py
│ │ ├── test_general.py
│ │ ├── test_interruption.py
│ │ └── test_state.py
│ ├── factories

35
 │ │ ├── __init__.py
│ │ ├── test_events.py
│ │ └── test_payloads.py
│ ├── __init__.py
│ ├── listeners
│ │ ├── __init__.py
│ │ └── test_agents.py
│ ├── services
│ │ ├── __init__.py
│ │ ├── test_onfleet.py
│ │ └── test_webhooks.py
│ ├── test_dtos.py
│ ├── test_facades.py
│ ├── test_serializers.py
│ └── test_urls.py
│
└── utils
├── dates.py
├── factories
│ └── onfleet_tasks.py
├── __init__.py
├── onfleet.py
├── random.py
├── serializers.py
└── urls.py

Templates for you

You can save a lot of time as you create a new project by reusing a template.
Make it easy to follow good practices, and hard to do not.
Take it, use it, customize it, and make it yours.

Template Link
Regular https://github.com/guilatrova/python-template
project
Lambda https://github.com/guilatrova/sam-lambda-python-template
with SAM

36