Download the Full Version of ebook for Fast Typing at ebookmass.

com

Data Ingestion with Python Cookbook: A practical

guide to ingesting, monitoring, and identifying
errors in the data ingestion process 1st Edition
Esppenchutz
https://ebookmass.com/product/data-ingestion-with-python-
cookbook-a-practical-guide-to-ingesting-monitoring-and-
identifying-errors-in-the-data-ingestion-process-1st-
edition-esppenchutz/

OR CLICK BUTTON

DOWNLOAD NOW

Download More ebook Instantly Today - Get Yours Now at ebookmass.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Python Data Cleaning Cookbook - Second Edition Michael

Walker

https://ebookmass.com/product/python-data-cleaning-cookbook-second-
edition-michael-walker/

ebookmass.com

Data Engineering with dbt: A practical guide to building a

cloud-based, pragmatic, and dependable data platform with
SQL Zagni
https://ebookmass.com/product/data-engineering-with-dbt-a-practical-
guide-to-building-a-cloud-based-pragmatic-and-dependable-data-
platform-with-sql-zagni/
ebookmass.com

Exploratory Data Analysis with Python Cookbook: Over 50

recipes to analyze, visualize, and extract insights from
structured and unstructured data Oluleye
https://ebookmass.com/product/exploratory-data-analysis-with-python-
cookbook-over-50-recipes-to-analyze-visualize-and-extract-insights-
from-structured-and-unstructured-data-oluleye/
ebookmass.com

Data-Driven SEO with Python: Solve SEO Challenges with

Data Science Using Python 1st Edition Andreas Voniatis

https://ebookmass.com/product/data-driven-seo-with-python-solve-seo-
challenges-with-data-science-using-python-1st-edition-andreas-
voniatis/
ebookmass.com
Statistical Process Monitoring Using Advanced Data-Driven
and Deep Learning Approaches: Theory and Practical
Applications 1st Edition Fouzi Harrou
https://ebookmass.com/product/statistical-process-monitoring-using-
advanced-data-driven-and-deep-learning-approaches-theory-and-
practical-applications-1st-edition-fouzi-harrou/
ebookmass.com

Beginner's Guide to Streamlit with Python: Build Web-Based

Data and Machine Learning Applications 1st Edition Sujay
Raghavendra
https://ebookmass.com/product/beginners-guide-to-streamlit-with-
python-build-web-based-data-and-machine-learning-applications-1st-
edition-sujay-raghavendra/
ebookmass.com

Data Structure and Algorithms With Python: The Ultimate

Guide Towards Coding John Thomas

https://ebookmass.com/product/data-structure-and-algorithms-with-
python-the-ultimate-guide-towards-coding-john-thomas/

ebookmass.com

(eBook PDF) Intro to Python for Computer Science and Data

Science: Learning to Program with AI, Big Data and The
Cloud
https://ebookmass.com/product/ebook-pdf-intro-to-python-for-computer-
science-and-data-science-learning-to-program-with-ai-big-data-and-the-
cloud/
ebookmass.com

Nonclinical Study Contracting and Monitoring: A Practical

Guide 1st Edition

https://ebookmass.com/product/nonclinical-study-contracting-and-
monitoring-a-practical-guide-1st-edition/

ebookmass.com
Data Ingestion with Python
Cookbook

A practical guide to ingesting, monitoring, and identifying

errors in the data ingestion process

Gláucia Esppenchutz

BIRMINGHAM—MUMBAI
Data Ingestion with Python Cookbook
Copyright © 2023 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted
in any form or by any means, without the prior written permission of the publisher, except in the case
of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information
presented. However, the information contained in this book is sold without warranty, either express
or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable
for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and
products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot
guarantee the accuracy of this information.

Group Product Manager: Reshma Raman

Publishing Product Manager: Arindam Majumdar
Senior Editor: Tiksha Lad
Technical Editor: Devanshi Ayare
Copy Editor: Safis Editing
Project Coordinator: Farheen Fathima
Proofreader: Safis Editing
Indexer: Sejal Dsilva
Production Designer: Jyoti Chauhan
Marketing Coordinator: Nivedita Singh

First published: May 2023

Production reference: 1300523

Published by Packt Publishing Ltd.

Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.

ISBN 978-1-83763-260-2

www.packtpub.com
 This book represents a lot and wouldn’t be possible without my loving husband, Lincoln, and his
support and understanding during this challenging endeavor. I want to thank all my friends that
didn’t let me give up and always boosted my spirits, along with my grandmother, who always believed,
helped, and said I would do big things one day. Finally, I want to thank my beloved and four-pawed
best friend, who is at peace, Minduim, for “helping” me to write this book.

– Gláucia Esppenchutz
 Contributors

About the author

Gláucia Esppenchutz is a data engineer with expertise in managing data pipelines and vast amounts
of data using cloud and on-premises technologies. She worked in companies such as Globo.com,
BMW Group, and Cloudera. Currently, she works at AiFi, specializing in the field of data operations
for autonomous systems.
She comes from the biomedical field and shifted her career ten years ago to chase the dream of
working closely with technology and data. She is in constant contact with the open source community,
mentoring people and helping to manage projects, and has collaborated with the Apache, PyLadies
group, FreeCodeCamp, Udacity, and MentorColor communities.

I want to thank my patient and beloved husband and my friends. Thanks also to my mentors in the
Python open source community and the DataBootCamp founders, who guided me at the beginning
of my journey.
Thanks to the Packt team, who helped me through some hard times; you were terrific!
About the reviewers
Bitthal Khaitan is currently working as a big data and cloud engineer with CVS Health, a Fortune
4 organization. He has a demonstrated history of working in the cloud, data and analytics industry
for 12+ years. His primary certified skills are Google Cloud Platform (GCP), the big data ecosystem
(Hadoop, Spark, etc.), and data warehousing on Teradata. He has worked in all phases of the SDLC
of DW/BI and big data projects with strong expertise in the USA healthcare, insurance and retail
domains. He actively helps new graduates with mentoring, resume reviews, and job hunting tips in
the data engineering domain. Over 20,000 people follow Bitthal on LinkedIn. He is currently based
out of Dallas, Texas, USA.
Jagjeet Makhija is a highly accomplished technology leader with over 20 years of experience. They are
skilled not only in various domains including AI, data warehouse architecture, and business analytics,
but also have a strong passion for staying ahead of technology trends such as AI and ChatGPT.
Jagjeet is recognized for their significant contributions to the industry, particularly in complex proof
of concepts and integrating Microsoft products with ChatGPT. They are also an avid book reviewer
and have actively shared their extensive knowledge and expertise through presentations, blog articles,
and online forums.
Krishnan Raghavan is an IT professional with over 20 years of experience in the area of software
development and delivery excellence across multiple domains and technology, ranging from C++ to
Java, Python, data warehousing, and big data tools and technologies. Krishnan tries to give back to the
community by being part of GDG – Pune Volunteer Group, helping the team in organizing events.
When not working, Krishnan likes to spend time with his wife and daughter, as well as reading fiction,
non-fiction, and technical books. Currently, he is unsuccessfully trying to learn how to play the guitar.
You can connect with Krishnan at mail to: k r i s h n a n @ g m a i l . c o m or via
LinkedIn: www.linkedin.com/in/krishnan-raghavan

I would like to thank my wife, Anita, and daughter, Ananya, for giving me the time and space to
review this book.
 Table of Contents

Prefacexv

Part 1: Fundamentals of Data Ingestion

1
Introduction to Data Ingestion 3
Technical requirements 4 How to do it… 17
Setting up Python and its environment 4 How it works… 21
See also 22
Getting ready 4
How to do it… 5 Creating schemas 22
How it works… 7 Getting ready 22
There’s more… 7 How to do it… 22
See also 7 How it works… 24
Installing PySpark 8 See also 25

Getting ready 8 Applying data governance in ingestion 25

How to do it… 10 Getting ready 25
How it works… 12 How to do it… 26
There’s more… 12 How it works… 28
See also 12 See also 28
Configuring Docker for MongoDB 13 Implementing data replication 29
Getting ready 13 Getting ready 29
How to do it… 13 How to do it… 29
How it works… 14 How it works… 29
There’s more… 16 There’s more… 30
See also 16
Further reading 30
Configuring Docker for Airflow 16
Getting ready 17
viii Table of Contents

2
Principals of Data Access – Accessing Your Data 31
Technical requirements 31 How to do it… 47
Implementing governance in a data How it works… 48
access workflow 32 There’s more… 49
Getting ready 32 See also 52
How to do it… 33 Managing encrypted files 52
How it works… 34 Getting ready 52
See also 34 How to do it… 53
Accessing databases and data How it works… 54
warehouses34 There’s more… 55
Getting ready 35 See also 56
How to do it… 35 Accessing data from AWS using S3 56
How it works… 37 Getting ready 56
There’s more… 38 How to do it… 59
See also 39 How it works… 62
Accessing SSH File Transfer Protocol There’s more… 63
(SFTP) files 39 See also 63
Getting ready 39 Accessing data from GCP using
How to do it… 41 Cloud Storage 64
How it works… 43 Getting ready 64
There’s more… 43 How to do it… 66
See also 44 How it works… 68
Retrieving data using API There’s more… 70
authentication44 Further reading 70
Getting ready 45

3
Data Discovery – Understanding Our Data before Ingesting It 71
Technical requirements 71 How to do it… 73
Documenting the data discovery How it works… 77
process71 Configuring OpenMetadata 77
Getting ready 72 Getting ready 77
 Table of Contents ix

How to do it… 79 Getting ready 86

How it works… 84 How to do it… 88
There’s more… 86 How it works… 92
See also 86
Further reading 93
Connecting OpenMetadata Other tools 94
to our database 86

4
Reading CSV and JSON Files and Solving Problems 95
Technical requirements 95 How it works… 105
Reading a CSV file 96 There’s more… 106
See also 107
Getting ready 96
How to do it… 96 Using PySpark to read CSV files 108
How it works… 98 Getting ready 108
There’s more… 98 How to do it… 108
See also 99 How it works… 109
Reading a JSON file 99 There’s more… 110
See also 114
Getting ready 100
How to do it… 100 Using PySpark to read JSON files 114
How it works… 100 Getting ready 114
There’s more… 101 How to do it… 115
See also 103 How it works… 116
Creating a SparkSession for PySpark 103 There’s more… 117
See also 117
Getting ready 103
How to do it… 104 Further reading 117

5
Ingesting Data from Structured and Unstructured Databases 119
Technical requirements 119 There’s more… 127
Configuring a JDBC connection 120 See also 129

Getting ready 120 Ingesting data from a JDBC database

How to do it… 124 using SQL 129
How it works… 126 Getting ready 129
x Table of Contents

How to do it… 130 Getting ready 146

How it works… 132 How to do it… 146
There’s more… 133 How it works… 148
See also 133 There’s more… 149
See also 150
Connecting to a NoSQL
database (MongoDB) 134 Ingesting data from MongoDB
Getting ready 134 using PySpark 150
How to do it… 136 Getting ready 150
How it works… 142 How to do it… 151
There’s more… 144 How it works… 153
See also 145 There’s more… 155
See also 156
Creating our NoSQL table in
MongoDB145 Further reading 157

6
Using PySpark with Defined and Non-Defined Schemas 159
Technical requirements 159 How to do it… 169
Applying schemas to data ingestion 160 How it works… 170

Getting ready 160 Ingesting unstructured data with a

How to do it… 162 well-defined schema and format 172
How it works… 163 Getting ready 172
There’s more… 165 How to do it… 172
See also 165 How it works… 174
Importing structured data using a There’s more… 176
well-defined schema 165 See also 176

Getting ready 165 Inserting formatted SparkSession

How to do it… 165 logs to facilitate your work 176
How it works… 167 Getting ready 176
There’s more… 168 How to do it… 176
See also 169 How it works… 178
Importing unstructured data without There’s more… 179
a schema 169 See also 179

Getting ready… 169 Further reading 179

 Table of Contents xi

7
Ingesting Analytical Data 181
Technical requirements 181 How it works… 197
Ingesting Parquet files 182 There’s more… 198
See also 200
Getting ready 182
How to do it… 183 Ingesting partitioned data 200
How it works… 184 Getting ready 200
There’s more… 185 How to do it… 201
See also 185 How it works… 201
Ingesting Avro files 185 There’s more… 203
See also 204
Getting ready 186
How to do it… 186 Applying reverse ETL 204
How it works… 188 Getting ready 204
There’s more… 190 How to do it… 205
See also 190 How it works… 206
Applying schemas to analytical data 191 There’s more… 207
See also 207
Getting ready 191
How to do it… 191 Selecting analytical data for reverse
How it works… 194 ETL207
There’s more… 194 Getting ready 207
See also 195 How to do it… 208
Filtering data and handling common How it works… 209
issues195 See also 210

Getting ready 195 Further reading 210

How to do it… 196

Part 2: Structuring the Ingestion Pipeline

8
Designing Monitored Data Workflows 213
Technical requirements 213 Getting ready 214
Inserting logs 214 How to do it… 214
How it works… 216
xii Table of Contents

See also 217 How it works… 229

There’s more… 229
Using log-level types 217
See also 230
Getting ready 217
How to do it… 217 Logging based on data 231
How it works… 219 Getting ready 231
There’s more… 220 How to do it… 231
See also 221 How it works… 232
There’s more… 233
Creating standardized logs 221
See also 233
Getting ready 222
How to do it… 222 Retrieving SparkSession metrics 234
How it works… 224 Getting ready 234
There’s more… 227 How to do it… 234
See also 227 How it works… 237
There’s more… 241
Monitoring our data ingest file size 227
See also 242
Getting ready 228
How to do it… 228 Further reading 242

9
Putting Everything Together with Airflow 243
Technical requirements 244 How to do it… 257
Installing Airflow 244 How it works… 260
There's more… 262
Configuring Airflow 244 See also 262
Getting ready 244
How to do it… 245 Configuring sensors 262
How it works… 247 Getting ready 262
See also 248 How to do it… 263
How it works… 264
Creating DAGs 248 See also 265
Getting ready 248
How to do it… 249 Creating connectors in Airflow 265
How it works… 253 Getting ready 266
There's more… 254 How to do it… 266
See also 255 How it works… 269
There's more… 270
Creating custom operators 255 See also 270
Getting ready 255
 Table of Contents xiii

Creating parallel ingest tasks 270 Getting ready 274

Getting ready 270 How to do it… 275
How to do it… 271 How it works… 277
How it works… 272 There's more… 278
There's more… 273 See also 279
See also 274
Further reading 279
Defining ingest-dependent DAGs 274

10
Logging and Monitoring Your Data Ingest in Airflow 281
Technical requirements 281 Designing advanced monitoring 304
Installing and running Airflow 282 Getting ready 304
How to do it… 306
Creating basic logs in Airflow 283
How it works… 308
Getting ready 284
There’s more… 309
How to do it… 284
See also 309
How it works… 287
See also 289 Using notification operators 309
Getting ready 310
Storing log files in a remote location 289
How to do it… 312
Getting ready 289
How it works… 315
How to do it… 290
There’s more… 318
How it works… 298
See also 299 Using SQL operators for data quality 318
Getting ready 318
Configuring logs in airflow.cfg 299
How to do it… 320
Getting ready 299
How it works… 321
How to do it… 299
There’s more… 323
How it works… 301
See also 323
There’s more… 303
See also 304 Further reading 324

11
Automating Your Data Ingestion Pipelines 325
Technical requirements 325 Scheduling daily ingestions 326
Installing and running Airflow 326 Getting ready 327
xiv Table of Contents

How to do it… 327 There's more… 340

How it works… 330
Setting up the schedule_interval
There's more… 330
parameter340
See also 331
Getting ready 340
Scheduling historical data ingestion 331 How to do it… 341
Getting ready 331 How it works… 342
How to do it… 332 See also 342
How it works… 335
Solving scheduling errors 343
There's more… 336
Getting ready 343
Scheduling data replication 337 How to do it… 343
Getting ready 337 How it works… 346
How to do it… 338 There’s more… 347
How it works… 339
Further reading 347

12
Using Data Observability for Debugging, Error Handling,
and Preventing Downtime 349
Technical requirements 349 Getting ready 358
Docker images 350 How to do it… 358
How it works… 361
Setting up StatsD for monitoring 351 There’s more… 363
Getting ready 351
How to do it… 351 Creating an observability dashboard 363
How it works… 353 Getting ready 363
See also 354 How to do it… 363
How it works… 369
Setting up Prometheus for storing There’s more… 370
metrics354
Getting ready 354 Setting custom alerts or notifications 370
How to do it… 354 Getting ready 371
How it works… 356 How to do it… 371
There’s more… 357 How it works… 377

Setting up Grafana for monitoring 358 Further reading 378

Index379

Other Books You May Enjoy 388

Discovering Diverse Content Through
Random Scribd Documents
 most unjustly and unfairly paid Mr. Griffith. Those cases are
continually arising, and of course they are very inconvenient
for a department: they not only take up a great deal of time,
but they very often prevent some very desirable process
being gone on with.”

Admiral Robinson said—

“There have been twelve upon the construction of ships

since 1861.

Mr. Bush Construction of ships.

Mr. J. Clare Construction of ships.
Mr. P. Drake Construction of ships.
Mr. A. Lamb Construction of ships.
Keels, stern posts,
Mr. W. Rae
&c.
Mr. Thomas and Col. De Bathe Mr. G. Clarke’s target.
Animal fibre. Armour
Mr. Truss
plates.
Mr. Beslay Preservation of iron.
Position of guns in
Capt. Wheatley
ships.
M. De Lapparent Carbonising timber.
Commander Warren Bow rudder.
Mr. Feathers Construction of ships.
Messrs. Woodcraft, Smith, Ericsson, Purchase of Patents
Lowe, Blaxland, and Mr. Currie. for screw propellers.
Capt. Carpenter Screw propeller.
Disconnecting
Capt. Trewhitt
apparatus.
Mr. Griffith Screw propeller.
Mr. J. O. Taylor Screw propeller.
W. Ireland Cupola.
Messrs. Laird and Cowper Trimming coals in
 ships.
Distilling apparatus in
——
‘Defence.’

“In those cases the patentees claimed compensation for

infringement?—Yes; and it was necessary for the Admiralty to
have recourse to their solicitor, and to enter into a very long
correspondence.
“It is very possible that you may infringe upon these Patents
without knowing it?—Constantly. The inconvenience which
the Duke of Somerset has mentioned resulting from Patents
applied to shipbuilding is so very great that it is scarcely
possible to build a ship, being a combination of wood and iron
(and you always have some of each in a ship), without
treading upon somebody’s Patent; and I am entirely of opinion
that the Patents are drawn up for that especial purpose,
without any idea of their being practically applied for the
benefit of the public, but only that the patentee may lie in wait
for a colourable evasion of his Patent taking place.”

Now I present the evidence of General Lefroy, deputed by the

War-office:—

“The expectations of patentees are very extravagant,

generally speaking, and prior to trial it is very difficult to
determine at all what is the value of an invention. As an
example, a gentleman some time ago made a great
improvement in cooking apparatus, and he assessed his own
reward at a large portion of the whole saving in fuel which
might be effected by the application of this improvement to an
enormous extent upon the whole military consumption of the
Crown, which would have come to many thousands of
pounds. Such an improvement should not be assessed by the
value to the Crown, but by what it cost the originator in
intellectual labour or previous experiment, and its importance
in a large sense.”
Let me next cite Mr. Clode, Solicitor to the War-office:—

“If he has not the power either of keeping those

improvements perfectly secret, or of securing them to himself
by Patent, then the War-office authorities are placed in the
position of having in all probability to pay private individuals
for inventions or improvements actually made by their own
officers.”

Next Mr. Abel, F.R.S., Head Chemist to the War Department:—

“In your experimental inquiries, when you have happened

to fall upon any discovery, you have not been much annoyed
by claimants saying that they have had precedence of you?—
Not at all, and it is to that that I referred in my first answer. We
do not meet practically with those embarrassments during
experiments, but we may meet with them in applying the
details of improvements. For instance, I am at present
engaged upon the working out of the application of gun
cotton, the whole details of which application were
communicated as a great secret to this Government by the
Austrian Government.... While every care was taken by this
Government to keep them secret, a Patent was taken out in
this country for the whole improved process of the
manufacture.”

Mr. Clode again:—

“Some time after I commenced these experiments, while

they remained a perfect secret, and while every care was
taken by this Government to keep them secret, a Patent was
taken out in this country for the whole improved process of
the manufacture.... One of them who is present is
experimenting upon gun cotton, but it is with him a matter of
extreme embarrassment to know how to deal with the subject;
if he discloses by way of specification all that he knows, he
 sends the invention or discovery he has made away to the
winds—the very night that it is put upon the file it goes to
Paris, Dresden, Berlin, and elsewhere. If he does not do that,
he is afraid that some man will find out precisely what he has
in view, and put a Patent on the file, and tax the Government
in that way. So that we are upon the horns of a dilemma.”

If I were now to stop, and say not a word more, I might trust to the
candour of the House for an admission that the case against Patents
is proved, on the ground that the conditions of the Statute of
Monopolies have been systematically violated, these violations being
of the very texture and vitals of the institution.
But I proceed. If the House permit, I will now advert to the new
phases the question has assumed since the inauguration of free
trade, understanding by that term le libre échange, and not la liberte
du travail.
The pernicious effect of home Patents on trade with our Indian
empire, is stated thus by Mr. Rendel, in 1851:—

“As engineer to the East India Railway, we had a little

inconvenience the other day; we wanted to manufacture
articles patented in this country, and we would have had to
pay Patent-rights; it was a question whether we had not better
buy the iron in India, and avoid the Patent-rights. Those
cases, I think, are constantly occurring. The Patent-Laws not
being applicable to India, people will not unfrequently order
things to be manufactured in India to avoid the licence dues in
this country; and the consequence was that I made an
arrangement with the patentees at about one-half of the
ordinary charge for the Patent in this country.”

In 1851 it was proposed, and in 1852 an Act was passed, to limit

British Patents to the United Kingdom, with exclusion of the
Colonies. This change was desired by an influential and intelligent
portion of the West India Association. Their conduct contradicted,
and their experience proves the fallacy of, the allegation so
confidently made and repeated in spite of its futility, by some
interested or else ignorant parties, that inventions thrive most where
Patents exist—i.e., where trade is trammelled with prohibitions or
burdened with royalties. The home sugar refiners exclaimed against
an exemption which, being partial, operated against their trade. The
following is an extract from one of the petitions presented by that
body:—

“That, so far as regards home manufacturers and

producers, such a change of the immemorial usages of the
kingdom is virtually a bestowal on parties carrying on the
same businesses in the colonies of a right to use patented
inventions fourteen years sooner than they.
“That if, at any time, the British Parliament might have put
home manufacturers on such an unfavourable footing, surely
this cannot be supposed under free-trade and equalised
duties, when they must task their utmost energies, and adopt
every improvement in mechanism and processes, in order to
maintain their ground.
“That the use of future Patents, at the rates that have been
freely paid by sugar refiners for Patents granted before now
would subject each sugar house, of average size, to a
payment of about £3,000 a year.
“That to exempt their competitors in the colonies from such
a tax (for tax it is, payable by order of, though not to, the
State) is really to give them a bounty of that very large
amount.
“That, in so far as patent fees may be considered a
premium for stimulating improvements, an equal share of the
benefit is enjoyed by the colonists, who, therefore, should
bear a due share of the burden.”

Soon after that time, protection having ceased, the unfairness of

burdening British manufacturers came more vividly into sight. How
can they compete with Prussia and Switzerland? Here is evidence
regarding those countries. From a Prussian witness:—

“I am a member of the Board of Trade and Commerce, and

at the same time a member of the Patent Commission.
“Will you be good enough to state what is the system
adopted in Prussia with regard to protection to inventions?—
We have the principle in our country to give as much liberty
as possible to every branch of industry and art, and,
considering every sort of Patent as an hindrance to their free
development, we are not very liberal in granting them. We
merely grant a Patent for a discovery of a completely novel
invention, or real improvement in existing inventions.”

From an important Swiss witness:—

“There is no want of persons to import them into

Switzerland, although those persons thus importing them
obtain no monopoly?—When a Patent is taken out in France
or England, the process is published; therefore it becomes the
property of the public in Switzerland; the Swiss have access
to the French or English Patents.
“In that way the Swiss have the benefit of the invention
without the charge of the licence?—Yes.
“And so far they have an advantage?—Certainly.
“When inventions in the watchmaking trade are made in
France, are they immediately introduced into Switzerland?—I
should think so, if they are useful.”

How, I ask, can British manufacturers compete with Prussia, which

prudently grants less than 100 Patents in a-year; or with Saxony,
which grants only about 134; or the Netherlands, which grant only
about 42? Rather, I may ask, how can they compete with other
countries in general, even those that grant Patents freely, seeing that
it is not incumbent on the British patentee to take a Patent in any
other country whatsoever; seeing also that, unlike some countries
which grant Patents, we in most cases do not terminate the currency
of those we grant at the time when the Patents taken elsewhere
expire? Honourable members will understand how serious is the
disadvantage under which our manufacturers, and with them, of
course, the labourers and artisans who co-operate in manufactures,
are placed if they are precluded from using inventions which their
continental rivals may use. When licences are given by patentees,
the disadvantage is lessened, but not very greatly. The House will
agree when it hears how enormous are the royalties sometimes
exacted. For a set of inventions in the iron trade, which is not the
subject of Patents in Prussia, a single firm is said to be paying at the
rate of £16,000 every quarter. Let me quote from a leading article in
the Engineer:—

“Owing to the invalidation of his Austrian Patents, Mr.

Bessemer derives no pecuniary benefit from the working of
his inventions in that country. This is also the state of things in
Prussia, whose really iniquitously-managed Patent
Commission have refused to give Mr. Bessemer any Patent at
all. The great Prussian steel works there manufacture
Bessemer steel unweighted by any royalty. We regret this, not
merely for Mr. Bessemer’s sake, but also on public grounds.
Our steel makers are thus heavily handicapped in the
industrial race with royalties of from one to even three pounds
per ton.”

See a confirmation of this in the following piece of a private letter:

—

“The very heavy royalty payable under Bessemer’s Patent

does, to a very great extent, prevent English manufacturers
competing on the Continent for steel rails; but, from the
accidental circumstance of continental manufacturers being
obliged to buy a considerable portion of their raw material
 from this country, we have not been exposed to competition in
England, as the cost of carriage backwards and forwards
about equalled the benefit which the Germans enjoyed of
paying no royalty.”

The sugar-refiners, in a printed document before me, put the case,

convincingly no doubt to all who will consider how small is the
percentage margin of profit in great businesses:—

“If, for any invention, French producers of refined sugar

should have only royalties of one per cent. ad valorem, while
the British should have to pay royalties of five per cent., it is
obvious the Patent-Law may in effect impose on the latter a
most onerous differential duty.”

In that trade I myself, shortly before my retiring from commerce,

paid £3,000 for a year’s right to use a new process, which proved
unworkable, and had to pay a solatium of £1,000 for leave to
discontinue it.
The agricultural interest should not remain indifferent. Mark what
was told the Commission by Mr. Reeve, Registrar to the Privy
Council. In Mr. Bovill’s Patent there was charged a royalty of 6d. a
quarter on all the corn ground in Great Britain by millers who thought
it desirable to adopt his plan. Obviously the royalty in that case had
the effect of a protective duty leviable for individual benefit, and
enabling foreigners to undersell in the British markets. And what title
to this power had Mr. Bovill? He was not the inventor. Another case
is exhibited in the following extract from a private letter with which I
am favoured, from a highly respectable quarter:—

“Patents have become so numerous and so various, that it

is not safe to use any piece of machinery, or make any
variation without first making a careful search to ascertain
whether it is not protected by a Patent. The Patent-Law has
also been the cause of much litigation, there being very few
Patents of any real worth but have had to go through the
ordeal of the Law Courts, and there can be little doubt that
injustice has frequently been done both to patentees and to
the public. A case of considerable hardship connected with
our own trade occurred regarding the application of the
exhaust to grinding purposes. It was clearly proved at the trial
that the machine for which the patentee claimed protection
had been in public use in Denmark, where it had been seen
by a Glasgow miller, who erected a similar machine on his
premises in Glasgow, but hastily threw it aside without putting
it to a proper test prior to the date of the Patent, but it was
held that no profitable use having been made of the machine
by the Glasgow miller, the Patent was good and perfectly
protected. In our opinion a Patent obtained in such
circumstances should never be allowed to stand, and if some
means could be devised for ascertaining the circumstances
beforehand, it should never be granted. The trade suffered
very considerably in consequence of this Patent being
sustained, and the consequence was, that although the
patentee was not the original inventor, he pocketed a very
large sum of money.
“A more recent instance has occurred, however, of a large
sum being pocketed by parties not the inventors of the article
patented. We can, however, only give you the figures as
popularly reported, without vouching for their accuracy, and in
relating the story we shall endeavour to reply to your queries
seriatim. 1st, The patented article is a machine for dressing
millstones by means of a black diamond, or piece of ‘bort,’
instead of by the hand with picks. It was originally patented in
France by the party said to be the inventor, and shortly
afterwards was patented by him in this country. 2nd and 3rd,
A Leith commission agent (a German) and an Edinburgh
miller saw the machine in the Paris Exhibition of 1867, and
induced the patentee to bring it over to Scotland for trial, and
ultimately they, in conjunction with a third party, purchased the
patentee’s right for the whole kingdom for £4,000. 4th, These
parties immediately put the machine in the market, and it was
 at once seized hold of by speculators, who readily gave most
extraordinary sums for it. One party is said to have paid
£40,000 for the right for a dozen counties in England; another
£15,000 for three counties; and another £20,000 for some
counties in Ireland: the whole sum realised by the original
purchasers amounting, it is said, to upwards of £150,000. 5th,
The consequence is, that such enormous sums having been
paid by the speculators, the trade can only get the use of the
machine by paying a most exorbitant price, and hitherto it has
remained all but a dead letter. We cannot give you in round
numbers the amount expected to be realised by the
speculators, but the price originally charged by them would
have yielded four or five times the amount they paid if the
whole trade had become purchasers. This machine has not
yet been the subject of litigation, but there is every probability
that it soon will be.”

But I can reproduce a case where the effect was far, far worse,
communicated to me in a private letter:—

“The patentee of the Howard series of improvements in

sugar-refining granted licences to houses in Liverpool and
Hull, with a condition in each case that he would not grant a
licence to any party carrying on business within seventy miles
of either town. A sugar refiner of long standing, established in
Sheffield, applied for a licence, and was refused for the
reason above stated, Sheffield being just within the
prescribed distance. The consequence was, he had to carry
on his manufacture for nearly fourteen years on the old
system; and during this period sustained great losses by
working, which he, as well as parties cognisant with the facts,
attributed to the disadvantage he was compelled to carry on
under. His fortune disappeared, and he became insolvent.—I
am, &c.
“Sheffield, December 17, 1863.”
 This distressing result will, I trust, drive home the conviction that,
great as is the evil of multiplying Patents, it would be but a mitigation
not worthy of being looked to as a cure, to get the number lessened.
If in an earlier part of this address I have shown that the condition
not to produce “general inconvenience” has been preposterously set
at nought, surely these passages prove no less conclusively that
there has been equal disregard of the condition not to “hurt trade.” I
will satisfy myself, and I hope the House, with one extract only to
prove what I apprehend is the rule rather than the exception, that
Patents offend against the other condition, not to “raise prices.” It is
from a paper read by Mr. Lowry Whittle before the Statistical Society
of Dublin:—

“I was informed lately of a case in the North of England

where a successful patentee produced a machine at the cost
of £200 for working in the linen trade. On this machine his
royalty is £1,000.”

I may give one instance from my own experience, where the

pretensions of the applicant for a Patent were equal to about a
farthing a pound on all the sugar that the process perfected. The
House may understand the hardship this would inflict on the
population when told that it was for the use of a single process only,
or rather of a machine invented by another, an engineer firm, who
had overlooked, and not included in their Patent, its applicability to
sugar. My experience in that case was very instructive. Pardon my
introducing a few particulars. I have no reason to think the idea of
applying the machine to the refining of sugar was original; on the
contrary, it had been already made practical on the Continent. Nor
was the idea patented by my friend alone; on the contrary, to several
persons it had occurred, by some (I forget how many) it had been
patented. One of my partners and I had a good deal of travelling in
England and Scotland, when we discovered the first patentee of the
application at length. We traced the indubitable priority home to a
good neighbour, whose office was within a bow-shot of a sugar-
house of which I myself was managing partner. He told me, when I
called about his Patent, that he had not attended to it for years. I
regret to be able to add that he was afterwards led, by
representations which I will not characterise, to part with his privilege
—it was really a very valuable one—for a most inadequate
consideration, to a person who had applied for a parasitical Patent
for something, the value of which could not be substantiated.
Perhaps the worst of all is, that the really most meritorious person,
the patentee of the machine, got comparatively little advantage from
its new but natural application. A coalition was formed whose terms
violated one of the conditions to which I have called attention, by
charging an exorbitant price for the machines, and, what is the
greatest mischief of Patents as now administered, by further
charging high royalties proportioned to the quantity of work they did.
Now will the House consider why it subjects the nation to all this
inconvenience, loss, and expense? It is not because without it we
would miss many important inventions. The groundlessness of such
a fear has already been indicated with sufficient plainness.
The House can hardly doubt, from its individual acquaintance with
what goes on in the world, and from the extracts I have troubled it
with, that whatever argument in favour of maintaining a Patent
system may be founded on the claims of inventors, the material
interests of the nation would suffer little from the cessation of Patents
as a stimulus. Unquestionably, if the system induces some
inventions to be made and published, it deters others. What we gain
is a matter of doubt. That much inconvenience is inflicted by it, and
much disadvantage and very heavy burdens, is no matter of doubt. It
is a case in which we have to balance the positive disadvantages
against the supposed advantages. To enable the House to weigh
these, by seeing how few inventions we would lose by total abolition,
a few more quotations may be permitted.
Very significantly Mr. Richard Roberts answers:—

“Would the absence of Patents for inventions, in your

judgment, have any effect in producing secret trades; or have
you had any opportunity of judging whether non-patented
 inventions are used much in secret trade?—I do not think
there is much secret trade, but I know this, that no trade can
be kept secret long; a quart of ale will do wonders in that
way.”

Let me adduce Mr. Woodcroft:—

“Do you think there is any natural tendency or propensity in

inventors to keep to themselves their inventions, or have they
a natural tendency to make them known?—The natural
tendency of an inventive mind is to make the invention
known.”

I now adduce the late able Mr. Fairrie:—

“You believe that the same energy of mind would be

displayed, and the same anxiety to make new discoveries felt,
whether there were this hope of protection or not?—I think so;
in the case of manufacturers certainly. I think the great bulk of
improvements proceed from the manufacturers themselves,
and not from mere inventors.”

Hear Colonel Reid, so well entitled to speak:—

“Supposing the law were so modified as to make the

acquisition of a Patent easy and simple, and to provide for the
publication at the earliest possible period, do not you think
there would be more inducement to the disclosure of the
secret under such a system than if all privileges of the kind
were abolished?—I am inclined to think that the advance in
improvement in all our arts would be greater by leaving them
entirely unshackled.”

Sir W. Cubitt was asked—

 “Have you ever been an inventor yourself?—Yes, of many
things; but a patented inventor of but one.
“You have taken out a Patent?—I took out a Patent in the
year 1807.
“Has your attention been at all directed to the advantages
or disadvantages of the present system?—Yes, it has been
drawn to the subject very frequently indeed; but the more it
was drawn to it, and the more I saw of it, the less I approved
of it; but with that disapproval I could not satisfy myself how to
devise anything much better; whether to make alterations, or
whether to do away with Patents altogether would be best, I
can hardly determine.
“Will you state, generally, your objections to the present
system?—The objections to the present system are the very
advanced state of scientific and practical knowledge, which
renders it difficult to secure anything. The principles of
mechanism being very well known and very well understood,
inventions involving exactly the same principle and to effect
the same object may be practically and apparently so
different, that Patents may be taken out for what is only a
difference in form, intended to produce the same effect,
without there being any difference in principle.”

So Sir W. Armstrong:—

“My firm conviction is, that if there was no artificial reward

for invention you would have just as much as at present.”

Mr. Grove perhaps goes at least part of the way:—

“The Patent is to encourage invention; if, therefore, you

would get the same inventions as we now get without Letters
Patent, I would have no Letters Patent at all. I believe that,
with respect to the minor class of inventions, you would get
them.”
Mr. Platt also has his doubts:—

“Is not almost every Patent which is now granted a Patent

for an improvement?—A great many Patents are granted for
things which are no improvement at all.
“I would simply limit the Patent-Law to that extent. I think
there are so many Patents granted that it is a great question
with me, I confess, if Patents for these combinations are to be
granted, whether it would not be better to abolish the Patent-
Laws altogether, as it becomes such a nuisance in conducting
a large business.”

How emphatic was Mr. I. Kingdom Brunel:—

“Do you think that there would be an equal inducement for

a man to turn his attention to improvements if there were no
Patent-Laws, as compared with the present state of things,
which lead him to the expectation and hope that he will obtain
some exclusive advantage from the discovery of some new
improvement?
“I feel certain of it; I have felt it very strongly, and it always
struck me as surprising that it was not seen by everybody
else; but we have so long been in the habit of considering that
the granting of an exclusive privilege to a man who invents a
thing is just and fair, that I do not think the public have ever
considered whether it was, after all, advantageous to him. My
feeling is, that it is very injurious to him.
“My impression is, that in every class of inventions you
would practically in the end have a more rapid supply and
increase of inventions than you have now; I believe that men
of science, and all those who do it for pleasure as well as for
profit, would produce more, they would be less interfered with
by existing Patents, and they would really produce more; I
believe that the working class, the smaller class of inventors,
would introduce very much more. With respect to that class of
 inventions, which I believe to be very few in number, though
they are talked of very much, which really involve long-
continued expenses, I believe they would probably be brought
about in a different manner. I wish, however, to have it
understood that I limit my observations to the present state of
things. I do not wish to express any opinion as to what might
have been formerly the effects of Patents, or whether they did
originally encourage inventions or not. I believe that in the first
place they are very prejudicial, on the whole, to a large class
supposed to exist of inventors, and principally from these
circumstances: the present state of things is this, that in all
branches, whether in manufactures or arts of any sort, we are
in such an advanced state, and every process in every
production consists of such a combination of the results of the
improvements which have been effected within the last twenty
or thirty years, that a good invention now is rarely a new
idea.”

So likewise Mr. James Spence:—

“The evils of the present system are serious. There is a

charm in the name of a Patent which entices large numbers of
men to neglect their own affairs in pursuit of some phantom.
Where intellectual power exists of an inventive character, it
will develop itself without any spur; it is, indeed, irrepressible
in its nature. To such minds the stimulus of a Patent is
superfluous.
“Besides the progress of the arts, another change has
occurred which affects this question. Formerly improvements
made slow progress, and unless an inventor were protected
for many years he had little chance of recompense. Now the
power of advertising is so great and intelligence is so diffused,
that any really useful invention can be brought immediately
into operation and profit. Were Patents abolished, any one
with an invention of value could find a manufacturer to take it
up. It is true it would be open to the rest of the world as soon
 as found out, but the manufacturer would obtain the first start
of all others, in itself a profit. Under the present system the
legal protection breaks down in practice. The moment a
specification is published, competing manufacturers strain
their wits to contrive how to reach the same result through
other means or modifications; in other words, how to infringe.
Against this the patentee has no remedy, except proceedings
at law of the most costly nature.
“No change can be proposed in Patent-Law that will not be
open to objections based on individual cases of hardship; but,
on a comprehensive view of the subject in all its bearings, I
hold that it would benefit the country to abolish the system in
toto. Manufacturers would be relieved from present perplexity,
delusions would no longer be kept up by excitement, an
enormous waste of money would be stayed; and whilst the
mass of worthless Patents would disappear, any of real value
would be taken up on its merits and produce sufficient
remuneration to the inventor.”

The Report of the Commission, founded on the evidence of which

I have shown the general character, contains the following just
observations:—

“The majority of witnesses, however, decidedly affirm the

existence of practical inconvenience from the multiplicity of
Patents. It is clear that Patents are granted for matters which
can hardly be considered as coming within the definition, in
the Statute of Monopolies, of ‘a new manufacture.’ It is in
evidence that the existence of these monopolies embarrasses
the trade of a considerable class of persons, artisans, small
tradesmen, and others, who cannot afford to face the
expense of litigation, however weak the case against them
may seem to be; and a still stronger case is made out as to
the existence of what may be called obstructive Patents, and
as to the inconvenience caused thereby to manufacturers
directly, and through them to the public.
 “Other instances will be found in the evidence of particular
manufactures and branches of invention which are so blocked
up by Patents, that not only are inventors deterred from taking
them up with a view to improvement, but the manufacturer, in
carrying on his regular course of trade, is hampered by
owners of worthless Patents, whom it is generally more
convenient to buy off than to resist. The evil also results in
another practice, having the same obstructive tendency—
namely, that of combination amongst a number of persons of
the same trade to buy up all the Patents relating to it, and to
pay the expense of attacking subsequent improvers out of a
common fund. From a comparison of evidence, it cannot be
doubted that this practice prevails to a considerable extent.
We must also conclude that when the obstruction is not to be
got rid of without the expense and annoyance of litigation, in a
large majority of cases the manufacturer submits to an
exaction, rather than incur the alternative.
“We desire to call special attention to the evidence given by
the First Lord of the Admiralty, and by various witnesses on
behalf of the War Department, showing the embarrassment
which has been caused to the naval and military services by
the multitude of Patents taken out for inventions in use in
those departments.
“It has long been the practice, founded on judicial decision,
to consider that the use or publication of an invention abroad
did not deprive that invention of the character of ‘a new
manufacture within this realm.’ It appears to us, and is
generally admitted in the evidence, that the present facilities
of communication subsisting between all parts of the world
have done away with the only valid reason for such a
construction of the words of the Statute of Monopolies. The
object of allowing such Patents might fairly be, in an age of
slow international communication, to encourage enterprising
persons to go in search of, and to introduce to this country,
useful processes employed abroad, but not otherwise likely to
be adopted here, for the want of which we should long have
 been behind other nations. It does not, however, seem worth
while to continue the same facilities now, when foreign
inventions are most frequently patented in this country and in
their native land simultaneously; especially, as we are well
informed, that one result of the practice is to encourage
unscrupulous persons to steal the inventions of foreigners
and to run a race with the legitimate owner to get them
patented here.”

The extracts which I have culled sufficiently prove that, in the

opinion of men selected because they were competent to speak with
authority on account of their character, ability, and experience, our
Patent system is “generally inconvenient” and is “hurtful to trade.”
Being so, it is inconsistent with the conditions on faith of which, while
other monopolies were prohibited by the Act, it was spared. But I rest
my case on absolute evils, without regard to that inconsistency. I am
sure nobody can go over the evidence as a whole, or even those
scraps of evidence which I have presented—I am well aware in a
very promiscuous and ineffective manner—without becoming
convinced that the trade and manufactures of this country are
seriously obstructed, fettered, retarded, harassed, and burdened,
sometimes demoralised, often wronged, or even robbed, by the
multitude and vexatious character of Patents, and by the claims and
conduct of patentees;—that these Patents, though very numerous, in
general possess little merit, yet often produce large revenues, the
result of exactions from persons who use them, to the assignees,
rather than to the original grantees,—that the uncertainty of receiving
a good return (in place of which experience shows there is, in most
cases, disappointment or even positive loss), and the utter
incongruity existing between the earnings, where there are any, and
the merits of inventions, render the system of Patents an
exceedingly unsatisfactory way of stimulating invention or rewarding
inventors;—and that there is wide-spread dissatisfaction with things
as they are, yet despair of amendment, among the most intelligent of
those portions of the community for whose benefit the system is
plausibly represented to exist.
 The evidence goes to show that the poor man and the working
man suffer in two ways. Such cannot bring their inventions into play
for want of capital, and they could not, even if it were in that respect
different, make head against rich infringers who are able by the
costliness of law proceedings to set them at defiance. I might allege,
also, that while the expenses of patenting are clearly too heavy to
suit the circumstances of the poor, there is little or no favour shown
by any influential witnesses to propositions for reducing them,
because of the tendency that a suitable reduction would have to still
further multiply Patents. Surely this indicates sufficiently that there is
something radically wrong in the principle on which we proceed.
Allow me, while adverting to the case of the poor, to express my
belief that the Patent system has an effect on wages which demands
the serious consideration of the friends of working men. I believe it
helps to keep wages low. The abolition would work in this manner:
whenever, in any establishment, an improvement is introduced, the
fact of its use becomes, of course, speedily known throughout the
establishment and in other establishments. The employés who in
their ordinary occupations must come to know what the improvement
is and how to work according to it—for this is a matter of necessity,
especially now that operations are conducted on a large scale, with
the indispensable aid of men intelligent and independent—very soon
find they are in request. To prevent their leaving, they are offered an
advance, which itself in its turn may be outbid. The rise which
indisputably would result in the case of individuals will, in my opinion,
tend towards a general rise. If I am correct in my anticipations,
operatives and artisans are much injured by Patent-Laws. But
independently of this hypothetical advantage, a good system of
dealing with inventors will be beneficial directly to operatives, by
removing from trade the present hindrances.
Having seen how little store there is set on Patents by eminent
engineers, by manufacturers, and by the public services, let me
appeal to eminent statesmen. Among these I name foremost the
apostle of free-trade. Mr. Cobden told me, many years ago, that he
was opposed to Patents; and at a later period, Oct., 1862, he wrote:
—
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookmass.com