Fr

Web scraping is becoming increasingly useful as a way

to easily gather and make sense of the plethora of
information available online. Using a simple language
such as Python, you can scrape complex websites with
little programming.

What you will learn from this book

Follow links to crawl a website
Extract data from web pages with lxml

This book is the ultimate guide to using Python to scrape

data from websites. It covers how to extract data from static
web pages and how to use caching to manage the load on
servers. Learn how to use AJAX URLs, employ the Firebug
extension. Discover more scraping nitty-gritties such as
using a browser renderer, managing cookies, submitting
forms to extract data from complex websites protected by
CAPTCHA, and so on. Finally, create high-level scrapers
with Scrapy and implement what has been learned on
real websites.

Web Scraping with Python

Web Scraping
with Python

Learn how to parse JavaScript-dependent

websites

C o m m u n i t y

Solve CAPTCHAs on protected web pages

Reverse engineer AJAX calls

Richard Lawson

Create high level scrapers with Scrapy

$ 34.99 US
22.99 UK

P U B L I S H I N G

pl

Cache downloads to reduce bandwidth

Who this book is written for

community experience distilled

Sa
m

Build a threaded crawler to process web

pages in parallel

Interact with forms and sessions

This book is aimed at developers who want to build reliable

solutions to scrape data from websites. It is assumed that
the reader has prior programming experience with Python.
Anyone with general knowledge of programming languages
should be able to pick up the book and understand the
principles involved.

ee

D i s t i l l e d

Web Scraping
with Python
Scrape data from any website with the power of Python

Prices do not include

local sales tax or VAT
where applicable

Visit www.PacktPub.com for books, eBooks,

code, downloads, and PacktLib.

E x p e r i e n c e

Richard Lawson

In this package, you will find:

The author biography

A preview chapter from the book, Chapter 1 'Introduction to Web Scraping'
A synopsis of the books content
More information on Python Web Scraping

About the Author

Richard Lawson is from Australia and studied Computer Science at the University

of Melbourne. Since graduating, he built a business specializing at web scraping

while traveling the world, working remotely from over 50 countries. He is a
fluent Esperanto speaker, conversational at Mandarin and Korean, and active in
contributing to and translating open source software. He is currently undertaking
postgraduate studies at Oxford University and in his spare time enjoys developing
autonomous drones.

Preface
The Internet contains the most useful set of data ever assembled, which is largely
publicly accessible for free. However, this data is not easily reusable. It is embedded
within the structure and style of websites and needs to be extracted to be useful.
This process of extracting data from web pages is known as web scraping and is
becoming increasingly useful as ever more information is available online.

What this book covers

Chapter 1, Introduction to Web Scraping, introduces web scraping and explains ways to
crawl a website.
Chapter 2, Scraping the Data, shows you how to extract data from web pages.
Chapter 3, Caching Downloads, teaches you how to avoid redownloading by
caching results.
Chapter 4, Concurrent Downloading, helps you to scrape data faster by downloading
in parallel.
Chapter 5, Dynamic Content, shows you how to extract data from dynamic websites.
Chapter 6, Interacting with Forms, shows you how to work with forms to access the
data you are after.
Chapter 7, Solving CAPTCHA, elaborates how to access data that is protected by
CAPTCHA images.

Preface

Chapter 8, Scrapy, teaches you how to use the popular high-level Scrapy framework.
Chapter 9, Overview, is an overview of web scraping techniques that have been covered.

Introduction to Web Scraping

In this chapter, we will cover the following topics:

Introduce the field of web scraping

Explain the legal challenges

Perform background research on our target website

Progressively building our own advanced web crawler

When is web scraping useful?

Suppose I have a shop selling shoes and want to keep track of my competitor's
prices. I could go to my competitor's website each day to compare each shoe's price
with my own, however this would take a lot of time and would not scale if I sold
thousands of shoes or needed to check price changes more frequently. Or maybe
I just want to buy a shoe when it is on sale. I could come back and check the shoe
website each day until I get lucky, but the shoe I want might not be on sale for
months. Both of these repetitive manual processes could instead be replaced with
an automated solution using the web scraping techniques covered in this book.
In an ideal world, web scraping would not be necessary and each website would
provide an API to share their data in a structured format. Indeed, some websites do
provide APIs, but they are typically restricted by what data is available and how
frequently it can be accessed. Additionally, the main priority for a website developer
will always be to maintain the frontend interface over the backend API. In short, we
cannot rely on APIs to access the online data we may want and therefore, need to learn
about web scraping techniques.

[1]

Introduction to Web Scraping

Is web scraping legal?

Web scraping is in the early Wild West stage, where what is permissible is still being
established. If the scraped data is being used for personal use, in practice, there is
no problem. However, if the data is going to be republished, then the type of data
scraped is important.
Several court cases around the world have helped establish what is permissible when
scraping a website. In Feist Publications, Inc. v. Rural Telephone Service Co., the United
States Supreme Court decided that scraping and republishing facts, such as telephone
listings, is allowed. Then, a similar case in Australia, Telstra Corporation Limited v. Phone
Directories Company Pty Ltd, demonstrated that only data with an identifiable author
can be copyrighted. Also, the European Union case, ofir.dk vs home.dk, concluded that
regular crawling and deep linking is permissible.
These cases suggest that when the scraped data constitutes facts (such as business
locations and telephone listings), it can be republished. However, if the data is
original (such as opinions and reviews), it most likely cannot be republished for
copyright reasons.
In any case, when you are scraping data from a website, remember that you are their
guest and need to behave politely or they may ban your IP address or proceed with
legal action. This means that you should make download requests at a reasonable
rate and define a user agent to identify you. The next section on crawling will cover
these practices in detail.
You can read more about these legal cases at http://
caselaw.lp.findlaw.com/scripts/getcase.
pl?court=US&vol=499&invol=340, http://www.
austlii.edu.au/au/cases/cth/FCA/2010/44.html, and
http://www.bvhd.dk/uploads/tx_mocarticles/S_-_
og_Handelsrettens_afg_relse_i_Ofir-sagen.pdf.

Background research
Before diving into crawling a website, we should develop an understanding about
the scale and structure of our target website. The website itself can help us through
their robots.txt and Sitemap files, and there are also external tools available to
provide further details such as Google Search and WHOIS.

[2]

Chapter 1

Checking robots.txt
Most websites define a robots.txt file to let crawlers know of any restrictions about
crawling their website. These restrictions are just a suggestion but good web citizens
will follow them. The robots.txt file is a valuable resource to check before crawling
to minimize the chance of being blocked, and also to discover hints about a website's
structure. More information about the robots.txt protocol is available at http://
www.robotstxt.org. The following code is the content of our example robots.txt,
which is available at http://example.webscraping.com/robots.txt:
# section 1
User-agent: BadCrawler
Disallow: /
# section 2
User-agent: *
Crawl-delay: 5
Disallow: /trap
# section 3
Sitemap: http://example.webscraping.com/sitemap.xml

In section 1, the robots.txt file asks a crawler with user agent BadCrawler not to
crawl their website, but this is unlikely to help because a malicious crawler would
not respect robots.txt anyway. A later example in this chapter will show you how
to make your crawler follow robots.txt automatically.
Section 2 specifies a crawl delay of 5 seconds between download requests for all
User-Agents, which should be respected to avoid overloading their server. There is
also a /trap link to try to block malicious crawlers who follow disallowed links. If you
visit this link, the server will block your IP for one minute! A real website would block
your IP for much longer, perhaps permanently, but then we could not continue with
this example.
Section 3 defines a Sitemap file, which will be examined in the next section.

[3]

Introduction to Web Scraping

Examining the Sitemap

Sitemap files are provided by websites to help crawlers locate their updated content

without needing to crawl every web page. For further details, the sitemap standard
is defined at http://www.sitemaps.org/protocol.html. Here is the content of the
Sitemap file discovered in the robots.txt file:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>http://example.webscraping.com/view/Afghanistan-1
</loc></url>
<url><loc>http://example.webscraping.com/view/Aland-Islands-2
</loc></url>
<url><loc>http://example.webscraping.com/view/Albania-3</loc>
</url>
...
</urlset>

This sitemap provides links to all the web pages, which will be used in the next
section to build our first crawler. Sitemap files provide an efficient way to crawl a
website, but need to be treated carefully because they are often missing, out of date,
or incomplete.

Estimating the size of a website

The size of the target website will affect how we crawl it. If the website is just a few
hundred URLs, such as our example website, efficiency is not important. However,
if the website has over a million web pages, downloading each sequentially would
take months. This problem is addressed later in Chapter 4, Concurrent Downloading,
on distributed downloading.
A quick way to estimate the size of a website is to check the results of Google's
crawler, which has quite likely already crawled the website we are interested in. We
can access this information through a Google search with the site keyword to filter
the results to our domain. An interface to this and other advanced search parameters
are available at http://www.google.com/advanced_search.

[4]

Chapter 1

Here are the site search results for our example website when searching Google for
site:example.webscraping.com:

As we can see, Google currently estimates 202 web pages, which is about as
expected. For larger websites, I have found Google's estimates to be less accurate.
We can filter these results to certain parts of the website by adding a URL path to
the domain. Here are the results for site:example.webscraping.com/view, which
restricts the site search to the country web pages:

[5]

Introduction to Web Scraping

This additional filter is useful because ideally you will only want to crawl the part of
a website containing useful data rather than every page of it.

Identifying the technology used by a website

The type of technology used to build a website will effect how we crawl it. A useful
tool to check the kind of technologies a website is built with is the builtwith
module, which can be installed with:
pip install builtwith

This module will take a URL, download and analyze it, and then return the
technologies used by the website. Here is an example:
>>> import builtwith
>>> builtwith.parse('http://example.webscraping.com')
{u'javascript-frameworks': [u'jQuery', u'Modernizr', u'jQuery UI'],
u'programming-languages': [u'Python'],
u'web-frameworks': [u'Web2py', u'Twitter Bootstrap'],
u'web-servers': [u'Nginx']}

We can see here that the example website uses the Web2py Python web framework
alongside with some common JavaScript libraries, so its content is likely embedded
in the HTML and be relatively straightforward to scrape. If the website was instead
built with AngularJS, then its content would likely be loaded dynamically. Or, if
the website used ASP.NET, then it would be necessary to use sessions and form
submissions to crawl web pages. Working with these more difficult cases will be
covered later in Chapter 5, Dynamic Content and Chapter 6, Interacting with Forms.

Finding the owner of a website

For some websites it may matter to us who is the owner. For example, if the owner
is known to block web crawlers then it would be wise to be more conservative in our
download rate. To find who owns a website we can use the WHOIS protocol to see
who is the registered owner of the domain name. There is a Python wrapper to this
protocol, documented at https://pypi.python.org/pypi/python-whois, which
can be installed via pip:
pip install python-whois

Here is the key part of the WHOIS response when querying the appspot.com domain
with this module:
>>> import whois
>>> print whois.whois('appspot.com')
[6]

Chapter 1
{
...
"name_servers": [
"NS1.GOOGLE.COM",
"NS2.GOOGLE.COM",
"NS3.GOOGLE.COM",
"NS4.GOOGLE.COM",
"ns4.google.com",
"ns2.google.com",
"ns1.google.com",
"ns3.google.com"
],
"org": "Google Inc.",
"emails": [
"abusecomplaints@markmonitor.com",
"dns-admin@google.com"
]
}

We can see here that this domain is owned by Google, which is correctthis domain
is for the Google App Engine service. Google often blocks web crawlers despite
being fundamentally a web crawling business themselves. We would need to be
careful when crawling this domain because Google often blocks web crawlers,
despite being fundamentally a web crawling business themselves.

Crawling your first website

In order to scrape a website, we first need to download its web pages containing the
data of interesta process known as crawling. There are a number of approaches
that can be used to crawl a website, and the appropriate choice will depend on the
structure of the target website. This chapter will explore how to download web
pages safely, and then introduce the following three common approaches to crawling
a website:

Crawling a sitemap

Iterating the database IDs of each web page

Following web page links

[7]

Introduction to Web Scraping

Downloading a web page

To crawl web pages, we first need to download them. Here is a simple Python script
that uses Python's urllib2 module to download a URL:
import urllib2
def download(url):
return urllib2.urlopen(url).read()

When a URL is passed, this function will download the web page and return the
HTML. The problem with this snippet is that when downloading the web page, we
might encounter errors that are beyond our control; for example, the requested page
may no longer exist. In these cases, urllib2 will raise an exception and exit the
script. To be safer, here is a more robust version to catch these exceptions:
import urllib2
def download(url):
print 'Downloading:', url
try:
html = urllib2.urlopen(url).read()
except urllib2.URLError as e:
print 'Download error:', e.reason
html = None
return html

Now, when a download error is encountered, the exception is caught and the
function returns None.

Retrying downloads
Often, the errors encountered when downloading are temporary; for example, the
web server is overloaded and returns a 503 Service Unavailable error. For these
errors, we can retry the download as the server problem may now be resolved.
However, we do not want to retry downloading for all errors. If the server returns
404 Not Found, then the web page does not currently exist and the same request is
unlikely to produce a different result.

[8]

Chapter 1

The full list of possible HTTP errors is defined by the Internet Engineering
Task Force, and is available for viewing at https://tools.ietf.org/html/
rfc7231#section-6. In this document, we can see that the 4xx errors occur when
there is something wrong with our request and the 5xx errors occur when there is
something wrong with the server. So, we will ensure our download function only
retries the 5xx errors. Here is the updated version to support this:
def download(url, num_retries=2):
print 'Downloading:', url
try:
html = urllib2.urlopen(url).read()
except urllib2.URLError as e:
print 'Download error:', e.reason
html = None
if num_retries > 0:
if hasattr(e, 'code') and 500 <= e.code < 600:
# recursively retry 5xx HTTP errors
return download(url, num_retries-1)
return html

Now, when a download error is encountered with a 5xx code, the download is
retried by recursively calling itself. The function now also takes an additional
argument for the number of times the download can be retried, which is set to two
times by default. We limit the number of times we attempt to download a web page
because the server error may not be resolvable. To test this functionality we can try
downloading http://httpstat.us/500, which returns the 500 error code:
>>> download('http://httpstat.us/500')
Downloading: http://httpstat.us/500
Download error: Internal Server Error
Downloading: http://httpstat.us/500
Download error: Internal Server Error
Downloading: http://httpstat.us/500
Download error: Internal Server Error

As expected, the download function now tries downloading the web page, and then
on receiving the 500 error, it retries the download twice before giving up.

[9]

Introduction to Web Scraping

Setting a user agent

By default, urllib2 will download content with the Python-urllib/2.7 user agent,
where 2.7 is the version of Python. It would be preferable to use an identifiable
user agent in case problems occur with our web crawler. Also, some websites block
this default user agent, perhaps after they experienced a poorly made Python web
crawler overloading their server. For example, this is what http://www.meetup.
com/ currently returns for Python's default user agent:

So, to download reliably, we will need to have control over setting the user agent.
Here is an updated version of our download function with the default user agent
set to 'wswp' (which stands for Web Scraping with Python):
def download(url, user_agent='wswp', num_retries=2):
print 'Downloading:', url
headers = {'User-agent': user_agent}
request = urllib2.Request(url, headers=headers)
try:
html = urllib2.urlopen(request).read()
except urllib2.URLError as e:
print 'Download error:', e.reason
html = None
if num_retries > 0:
if hasattr(e, 'code') and 500 <= e.code < 600:
# retry 5XX HTTP errors
return download(url, user_agent, num_retries-1)
return html

Now we have a flexible download function that can be reused in later examples to
catch errors, retry the download when possible, and set the user agent.

[ 10 ]

Chapter 1

Sitemap crawler
For our first simple crawler, we will use the sitemap discovered in the example
website's robots.txt to download all the web pages. To parse the sitemap, we will
use a simple regular expression to extract URLs within the <loc> tags. Note that a
more robust parsing approach called CSS selectors will be introduced in the next
chapter. Here is our first example crawler:
def crawl_sitemap(url):
# download the sitemap file
sitemap = download(url)
# extract the sitemap links
links = re.findall('<loc>(.*?)</loc>', sitemap)
# download each link
for link in links:
html = download(link)
# scrape html here
# ...

Now, we can run the sitemap crawler to download all countries from the
example website:
>>> crawl_sitemap('http://example.webscraping.com/sitemap.xml')
Downloading: http://example.webscraping.com/sitemap.xml
Downloading: http://example.webscraping.com/view/Afghanistan-1
Downloading: http://example.webscraping.com/view/Aland-Islands-2
Downloading: http://example.webscraping.com/view/Albania-3
...

This works as expected, but as discussed earlier, Sitemap files often cannot be relied
on to provide links to every web page. In the next section, another simple crawler
will be introduced that does not depend on the Sitemap file.

ID iteration crawler
In this section, we will take advantage of weakness in the website structure to easily
access all the content. Here are the URLs of some sample countries:

http://example.webscraping.com/view/Afghanistan-1

http://example.webscraping.com/view/Australia-2

http://example.webscraping.com/view/Brazil-3

[ 11 ]

Introduction to Web Scraping

We can see that the URLs only differ at the end, with the country name (known
as a slug) and ID. It is a common practice to include a slug in the URL to help
with search engine optimization. Quite often, the web server will ignore the slug
and only use the ID to match with relevant records in the database. Let us check
whether this works with our example website by removing the slug and loading
http://example.webscraping.com/view/1:

The web page still loads! This is useful to know because now we can ignore the slug
and simply iterate database IDs to download all the countries. Here is an example
code snippet that takes advantage of this trick:
import itertools
for page in itertools.count(1):
url = 'http://example.webscraping.com/view/-%d' % page
html = download(url)
if html is None:
break
else:
# success - can scrape the result
pass

[ 12 ]

Chapter 1

Here, we iterate the ID until we encounter a download error, which we assume

means that the last country has been reached. A weakness in this implementation is
that some records may have been deleted, leaving gaps in the database IDs. Then,
when one of these gaps is reached, the crawler will immediately exit. Here is an
improved version of the code that allows a number of consecutive download errors
before exiting:
# maximum number of consecutive download errors allowed
max_errors = 5
# current number of consecutive download errors
num_errors = 0
for page in itertools.count(1):
url = 'http://example.webscraping.com/view/-%d' % page
html = download(url)
if html is None:
# received an error trying to download this webpage
num_errors += 1
if num_errors == max_errors:
# reached maximum number of
# consecutive errors so exit
break
else:
# success - can scrape the result
# ...
num_errors = 0

The crawler in the preceding code now needs to encounter five consecutive
download errors to stop iterating, which decreases the risk of stopping the iteration
prematurely when some records have been deleted.
Iterating the IDs is a convenient approach to crawl a website, but is similar to the
sitemap approach in that it will not always be available. For example, some websites
will check whether the slug is as expected and if not return a 404 Not Found error.
Also, other websites use large nonsequential or nonnumeric IDs, so iterating is not
practical. For example, Amazon uses ISBNs as the ID for their books, which have at
least ten digits. Using an ID iteration with Amazon would require testing billions of
IDs, which is certainly not the most efficient approach to scraping their content.

[ 13 ]

Introduction to Web Scraping

Link crawler
So far, we have implemented two simple crawlers that take advantage of the
structure of our sample website to download all the countries. These techniques
should be used when available, because they minimize the required amount of web
pages to download. However, for other websites, we need to make our crawler act
more like a typical user and follow links to reach the content of interest.
We could simply download the entire website by following all links. However, this
would download a lot of web pages that we do not need. For example, to scrape user
account details from an online forum, only account pages need to be downloaded and
not discussion threads. The link crawler developed here will use a regular expression
to decide which web pages to download. Here is an initial version of the code:
import re
def link_crawler(seed_url, link_regex):
"""Crawl from the given seed URL following links matched by link_regex
"""
crawl_queue = [seed_url]
while crawl_queue:
url = crawl_queue.pop()
html = download(url)
# filter for links matching our regular expression
for link in get_links(html):
if re.match(link_regex, link):
crawl_queue.append(link)
def get_links(html):
"""Return a list of links from html
"""
# a regular expression to extract all links from the webpage
webpage_regex = re.compile('<a[^>]+href=["\'](.*?)["\']',
re.IGNORECASE)
# list of all links from the webpage
return webpage_regex.findall(html)

To run this code, simply call the link_crawler function with the URL of the website
you want to crawl and a regular expression of the links that you need to follow. For
the example website, we want to crawl the index with the list of countries and the
countries themselves. The index links follow this format:

http://example.webscraping.com/index/1

http://example.webscraping.com/index/2

[ 14 ]

Chapter 1

The country web pages will follow this format:

http://example.webscraping.com/view/Afghanistan-1

http://example.webscraping.com/view/Aland-Islands-2

So a simple regular expression to match both types of web pages is /(index|view)/.

What happens when the crawler is run with these inputs? You would find that we
get the following download error:
>>> link_crawler('http://example.webscraping.com',
'example.webscraping.com/(index|view)/')
Downloading: http://example.webscraping.com
Downloading: /index/1
Traceback (most recent call last):
...
ValueError: unknown url type: /index/1

The problem with downloading /index/1 is that it only includes the path of the
web page and leaves out the protocol and server, which is known as a relative link.
Relative links work when browsing because the web browser knows which web
page you are currently viewing. However, urllib2 is not aware of this context.
To help urllib2 locate the web page, we need to convert this link into an absolute
link, which includes all the details to locate the web page. As might be expected,
Python includes a module to do just this, called urlparse. Here is an improved
version of link_crawler that uses the urlparse module to create the absolute links:
import urlparse
def link_crawler(seed_url, link_regex):
"""Crawl from the given seed URL following links matched by link_regex
"""
crawl_queue = [seed_url]
while crawl_queue:
url = crawl_queue.pop()
html = download(url)
for link in get_links(html):
if re.match(link_regex, link):
link = urlparse.urljoin(seed_url, link)
crawl_queue.append(link)

[ 15 ]

Introduction to Web Scraping

When this example is run, you will find that it downloads the web pages without
errors; however, it keeps downloading the same locations over and over. The reason
for this is that these locations have links to each other. For example, Australia links to
Antarctica and Antarctica links right back, and the crawler will cycle between these
forever. To prevent re-crawling the same links, we need to keep track of what has
already been crawled. Here is the updated version of link_crawler that stores the
URLs seen before, to avoid redownloading duplicates:
def link_crawler(seed_url, link_regex):
crawl_queue = [seed_url]
# keep track which URL's have seen before
seen = set(crawl_queue)
while crawl_queue:
url = crawl_queue.pop()
html = download(url)
for link in get_links(html):
# check if link matches expected regex
if re.match(link_regex, link):
# form absolute link
link = urlparse.urljoin(seed_url, link)
# check if have already seen this link
if link not in seen:
seen.add(link)
crawl_queue.append(link)

When this script is run, it will crawl the locations and then stop as expected. We
finally have a working crawler!

Advanced features
Now, let's add some features to make our link crawler more useful for crawling
other websites.

Parsing robots.txt
Firstly, we need to interpret robots.txt to avoid downloading blocked URLs.
Python comes with the robotparser module, which makes this straightforward,
as follows:
>>> import robotparser
>>> rp = robotparser.RobotFileParser()
>>> rp.set_url('http://example.webscraping.com/robots.txt')
>>> rp.read()
>>> url = 'http://example.webscraping.com'
[ 16 ]

Chapter 1
>>> user_agent = 'BadCrawler'
>>> rp.can_fetch(user_agent, url)
False
>>> user_agent = 'GoodCrawler'
>>> rp.can_fetch(user_agent, url)
True

The robotparser module loads a robots.txt file and then provides a can_fetch()
function, which tells you whether a particular user agent is allowed to access a web
page or not. Here, when the user agent is set to 'BadCrawler', the robotparser
module says that this web page can not be fetched, as was defined in robots.txt
of the example website.
To integrate this into the crawler, we add this check in the crawl loop:
...
while crawl_queue:
url = crawl_queue.pop()
# check url passes robots.txt restrictions
if rp.can_fetch(user_agent, url):
...
else:
print 'Blocked by robots.txt:', url

Supporting proxies
Sometimes it is necessary to access a website through a proxy. For example, Netflix
is blocked in most countries outside the United States. Supporting proxies with
urllib2 is not as easy as it could be (for a more user-friendly Python HTTP module,
try requests, documented at http://docs.python-requests.org/). Here is how
to support a proxy with urllib2:
proxy = ...
opener = urllib2.build_opener()
proxy_params = {urlparse.urlparse(url).scheme: proxy}
opener.add_handler(urllib2.ProxyHandler(proxy_params))
response = opener.open(request)

Here is an updated version of the download function to integrate this:

def download(url, user_agent='wswp', proxy=None, num_retries=2):
print 'Downloading:', url
headers = {'User-agent': user_agent}
request = urllib2.Request(url, headers=headers)

[ 17 ]

Introduction to Web Scraping

opener = urllib2.build_opener()
if proxy:
proxy_params = {urlparse.urlparse(url).scheme: proxy}
opener.add_handler(urllib2.ProxyHandler(proxy_params))
try:
html = opener.open(request).read()
except urllib2.URLError as e:
print 'Download error:', e.reason
html = None
if num_retries > 0:
if hasattr(e, 'code') and 500 <= e.code < 600:
# retry 5XX HTTP errors
html = download(url, user_agent, proxy,
num_retries-1)
return html

Throttling downloads
If we crawl a website too fast, we risk being blocked or overloading the server.
To minimize these risks, we can throttle our crawl by waiting for a delay between
downloads. Here is a class to implement this:
class Throttle:
"""Add a delay between downloads to the same domain
"""
def __init__(self, delay):
# amount of delay between downloads for each domain
self.delay = delay
# timestamp of when a domain was last accessed
self.domains = {}
def wait(self, url):
domain = urlparse.urlparse(url).netloc
last_accessed = self.domains.get(domain)
if self.delay > 0 and last_accessed is not None:
sleep_secs = self.delay - (datetime.datetime.now() last_accessed).seconds
if sleep_secs > 0:
# domain has been accessed recently
# so need to sleep
time.sleep(sleep_secs)
# update the last accessed time
self.domains[domain] = datetime.datetime.now()

[ 18 ]

Chapter 1

This Throttle class keeps track of when each domain was last accessed and will
sleep if the time since the last access is shorter than the specified delay. We can add
throttling to the crawler by calling throttle before every download:
throttle = Throttle(delay)
...
throttle.wait(url)
result = download(url, headers, proxy=proxy,
num_retries=num_retries)

Avoiding spider traps

Currently, our crawler will follow any link that it has not seen before. However,
some websites dynamically generate their content and can have an infinite number
of web pages. For example, if the website has an online calendar with links provided
for the next month and year, then the next month will also have links to the next
month, and so on for eternity. This situation is known as a spider trap.
A simple way to avoid getting stuck in a spider trap is to track how many links
have been followed to reach the current web page, which we will refer to as depth.
Then, when a maximum depth is reached, the crawler does not add links from this
web page to the queue. To implement this, we will change the seen variable, which
currently tracks the visited web pages, into a dictionary to also record the depth they
were found at:
def link_crawler(..., max_depth=2):
max_depth = 2
seen = {}
...
depth = seen[url]
if depth != max_depth:
for link in links:
if link not in seen:
seen[link] = depth + 1
crawl_queue.append(link)

Now, with this feature, we can be confident that the crawl will always complete
eventually. To disable this feature, max_depth can be set to a negative number so
that the current depth is never equal to it.

[ 19 ]

Introduction to Web Scraping

Final version
The full source code for this advanced link crawler can be downloaded at

https://bitbucket.org/wswp/code/src/tip/chapter01/link_crawler3.py.
To test this, let us try setting the user agent to BadCrawler, which we saw earlier
in this chapter was blocked by robots.txt. As expected, the crawl is blocked and

finishes immediately:

>>> seed_url = 'http://example.webscraping.com/index'

>>> link_regex = '/(index|view)'
>>> link_crawler(seed_url, link_regex, user_agent='BadCrawler')
Blocked by robots.txt: http://example.webscraping.com/

Now, let's try using the default user agent and setting the maximum depth to 1 so
that only the links from the home page are downloaded:
>>> link_crawler(seed_url, link_regex, max_depth=1)
Downloading: http://example.webscraping.com//index
Downloading: http://example.webscraping.com/index/1
Downloading: http://example.webscraping.com/view/Antigua-and-Barbuda-10
Downloading: http://example.webscraping.com/view/Antarctica-9
Downloading: http://example.webscraping.com/view/Anguilla-8
Downloading: http://example.webscraping.com/view/Angola-7
Downloading: http://example.webscraping.com/view/Andorra-6
Downloading: http://example.webscraping.com/view/American-Samoa-5
Downloading: http://example.webscraping.com/view/Algeria-4
Downloading: http://example.webscraping.com/view/Albania-3
Downloading: http://example.webscraping.com/view/Aland-Islands-2
Downloading: http://example.webscraping.com/view/Afghanistan-1

As expected, the crawl stopped after downloading the first page of countries.

Summary
This chapter introduced web scraping and developed a sophisticated crawler that
will be reused in the following chapters. We covered the usage of external tools and
modules to get an understanding of a website, user agents, sitemaps, crawl delays,
and various crawling strategies.
In the next chapter, we will explore how to scrape data from the crawled web pages.

[ 20 ]

Get more information Python Web Scraping

Where to buy this book

You can buy Python Web Scraping from the Packt Publishing website.
Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet
book retailers.
Click here for ordering and shipping details.

www.PacktPub.com

Stay Connected: