Do you have a dirty, messy data problem?
Whether you work as a software developer or as a data scientist,
you've surely run across data that is malformed, incomplete, or maybe even wrong.
Don't let messy data wreck your apps or generate wrong results.
What should you do?
Listen to this episode of Talk Python To Me with Catherine Jarmul
about the book she co-authored called Data Wrangling in Python
and her PyCon UK presentation entitled
How to Automate Your Data Cleanup with Python.
This is Talk Python To Me, recorded November 28, 2016.
Welcome to Talk Python To Me, a weekly podcast on Python,
the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm
and follow the show on Twitter via at Talk Python.
This episode has been sponsored by Rollbar and GoCD.
Thank them both for supporting the podcast
by checking out what they're offering during their segments.
Catherine, welcome to Talk Python.
Thanks.
I'm really excited to wrangle some data together.
It's going to be fun.
Yeah, it sounds good.
I've seen some of your talks and, you know, look through your book
and you're doing really cool stuff with data, data cleansing, data processing, data pipelines.
And so I'm really looking forward to getting into those topics and sharing them with everybody.
But before we do, let's start at the beginning.
What's your story?
How do you get into programming in Python?
Yeah, so I kind of have a varied history with programming.
I started doing programming in high school where I learned C++ as part of my AP computer science.
And then I went to school and I was attempting to get my degree in computer science.
And although I really loved my maths courses, I was pretty secluded in my computer programming courses.
And I attribute this to the fact that I wasn't that into gaming.
And I was one of about three women out of an incoming freshman class of over 300.
I totally know where you're coming from.
It seems like young guys that are into programming, so many of them come from the gaming world.
That's why they got into programming is because they're so excited about games.
Do you feel like that you felt like that kind of left you excluded from the social circles a little bit
just because you didn't want to go hang out and play games and eat pizza at two in the morning?
Yeah, it was definitely.
I mean, this was a bit of a different era.
I'm happy to hear it's a bit different now, but this was definitely like LAN parties and EverQuest.
And I just didn't connect with that very well.
So I didn't find a lot of people that wanted to work on projects with me or, you know, socialize
with me outside of class, maybe about math things rather than about computer things.
And for that reason, I kind of gravitated into economics and statistics and political science.
And I kind of ended up doing statistics with soft sciences, if you will.
Sure.
And that's kind of the nudge that sent you down this data science path a little bit, huh?
Yeah, yeah.
I ended up eventually getting into journalism from that.
And I ended up at the Washington Post and was that's where I met Jackie Kazel, my co-author
for the book.
And I learned Python there.
And that was really fun.
Oh, nice.
What kind of work were you doing there?
So the Washington Post was initially, and I don't know if it still is, one of the largest
Django installs in the world.
And we had built up a big app stack where we did the elections and numerous, all of the
data pieces were built on top of Django with quite a lot of Python in the background doing
the data wrangling.
So that was kind of my first exposure to data wrangling, if you will, with Python.
Yeah, that sounds really cool.
And there's actually a lot of data science and interesting things going on in this data
journalism space, right?
That's a pretty big growing area, isn't it?
Yeah, it's really neat to...
I'm still in touch with some folks that are still in journalism.
And I meet new people all the time because I'm involved in PyData Berlin.
And so for those reasons, I'm constantly impressed and seeing new things that journalists are doing
with data.
And if any time, now is the time to do really good data journalism, in my opinion.
Yeah, it's definitely a good time.
And it's also a time where a lot of...
It's very challenging, right?
I just did a show with Jonathan Morgan from Partially Derivative about the top 10 data science
stories of 2016 that's coming out at the end of the year.
And I'm saving it for the end of the year because that seems like that's the time you do a look
back episode.
But a lot of the themes were about how data science and sort of the pollsters and so on
didn't really get things right this year.
And I think it's both an interesting time and a challenging time.
It's pretty difficult with statistics to predict something like human behavior sometimes.
So math can only do so much for you when you have some unpredictability, if you will.
Yeah, absolutely.
Absolutely.
All right.
Well, we have a bunch of stuff to talk about.
So we're going to kind of skip around a little bit.
But let's start with your book.
Tell us, what's the title?
Where did you get the idea to write it in the first place?
Yeah.
So the title is Data Wrangling with Python.
And it was a kind of project of love between Jackie Kazel and I.
Again, we had worked together initially at The Post back in 2008.
And she actually first pitched the book to O'Reilly.
And she was working on it.
And she decided, you know what, this would be a lot better if I had a co-author.
And so she called me up.
I was in Berlin.
And I said, sure, that sounds great.
I would love to help.
That's really cool.
So the idea is, it's really for people who are not super seasoned Python programmers, but they're just getting into this whole data wrangling world.
And it sort of introduces them to Python a little bit and hits all the major problems or types of things you want to do getting started, right?
Yeah.
So our initial idea was, this is for beginners.
This is for someone that may or may not even know where their command line prompt is.
And we're going to take them through the steps.
And this is to make data wrangling more accessible to people that might not have a computer science background.
And that kind of is a passion of mine and also of Jackie's, being that we both were involved in PyLadies chapters.
And for that, yeah, we want an easy, accessible way to get involved working with Python, working with data.
And this was our idea, our product of that.
Yeah, I feel like, depending on your background, you sympathize more with people who you know are going to struggle, right?
Like, I learned programming in college, not when I was really young.
And so I still remember not getting my C libraries to link correctly and all the pain of what it is to be a new programmer.
So I feel like, you know, it sounds like you have some of those experiences as well.
And that probably comes through in the book.
Yeah, I think, you know, there's a lot of ways that even just as we get more advanced as developers and programmers or engineers,
whatever you think of yourself as, that you start using a lot of jargon and you start, you know, thinking in these bigger pictures.
And sometimes it's good to remind yourself, you know, there's probably an easier way to explain this.
There's probably a way to make this slightly more accessible.
And I feel like that's a really big, important step towards making beginners feel like they can be a part of the community.
Yeah, absolutely.
I mean, there's certainly times when having high level conversations that maybe depend on terminology out of design patterns or some common library that you all know or something like that.
It's important because you want to be efficient and get stuff done, right, as experts.
But when you're presenting or when you're writing books, sometimes this perspective is really, really cool.
So some of the things you talked about in there were basically just, you know, how do you get data loaded up?
And there were three major areas where this data comes from.
One of them was CSV and JSON, basically text files, right?
Yeah.
So this is particularly from a journalism perspective and also from just data that I see at companies.
This is a big way that people handle data, even just, yeah, plain text files, if you will.
Yeah, absolutely.
And so how often did you hear from people that were like trying to parse CSV files directly or JSON files directly rather than using the built-in modules?
I'm not quite sure.
I do think that there's a lot of people that just try to do that directly in the files or directly in a program like Excel.
And so this is kind of taking them out of the pre-programmed programs to actually get started writing code and using Python to do it.
Yeah, that's cool.
Speaking of Excel, like that's got to be the world's biggest database, right?
You know, there's so much data and data processing actually runs on Excel.
Until you hit about.
Yeah, yeah.
Yeah.
Until it just like, it just doesn't work anymore.
Like, I don't think people leave Excel very willingly.
It's like, we have to find a different answer.
Yeah.
Yeah, that and I've found, you know, quite a lot of resistance to learning programming for people that are really adept at SQL.
They kind of stay in this databasing world where everything is a SQL query.
And every report can be made with a really, really complex 40 line SQL query.
And I have a lot of respect for that.
But maybe it's a little bit easier to let a programming language do some of the heavy lifting for you.
Sure.
I mean, SQL is great for declarative stuff.
But sometimes you want an imperative type of problem solving and with a little declarative mixed in or who knows, right?
Another area that you said you really liked were PDFs, right?
Oh, yeah.
Yeah, PDFs are amazing.
Amazingly painful, right?
Yeah, exactly.
Indeed, a very large pain point.
And I still find that there's so many different NGOs and other governmental organizations that release all of their data in a yearly report in PDF form.
Yeah.
And, you know, to some degree, that's fine if it's meant for like reading.
But there should be a, and here's the actual JSON version or something, right?
And that probably is missing.
Yeah.
And, you know, the biggest thing that we encourage folks to do in the book is actually just pick up the phone and call the agency or the NGO or whomever it is and ask, hey, do you happen to have this in any other format?
Because PDFs are really painful.
There's not a lot of fun.
And you end up doing quite a lot of post-processing.
Can you do it, though, like from Python?
If you've got like tables and a PDF, can you get in there and get it out with Python?
Yeah.
So we ran into this issue with the book because it was actually quite difficult to get at some of the tabular data.
And I came across an old kind of forgotten library called PDF tables.
At the time, it wasn't Python 3 compliant.
So we needed to do it all in Python 2.
But now I went to EuroPython and I was talking about this problem.
And there was a guy there.
I forget his name right now.
But he converted it to Python 3.
So now it's, I believe, compatible with 3.4.
And you can just simply use a few commands.
The documentation is not very well made.
But you can use a few commands and you can parse out actual tabular data.
And it does it quite well.
That's really excellent.
I think that's an interesting story and that happens all the time, right?
Like there are so many packages and this functionality is spread across so many different places, right?
I mean, Python is great because you can pip install anything, right?
Pip install, you know, import anti-gravity sort of thing, right?
Yeah.
Which is great.
But when you're newer, how do you know where to find these things?
So I think when you get a chance to share these cool libraries that you found that solve problems, I think that's great.
And we'll do some more of that later, actually, from your talk.
Cool.
Yeah, for sure.
So that's sort of the data acquisition stuff.
And then you talk a little bit about storing and presenting data.
You want to talk about that?
Yeah.
So I gave people basically an introduction to databasing and kind of went over with them.
Okay.
Here's how you might use relational or non-relational databases.
And here's a little bit of the pros and cons of each.
It's kind of difficult to make that extremely accessible to beginners.
But I feel like it's a good problem to start to introduce them to, that there's quite a lot of different ways to store data.
And you should start thinking now about what makes the most sense for your project or for your team.
And that's a problem that we face every day, you know, as data people, when we're deciding how to construct something new or we're deciding, okay, how should we build this workflow?
We constantly have to think, okay, what's going to be, you know, do we need it in high availability storage, so to speak?
Or do we need it maybe stored away somewhere in a file on somebody's computer?
Right.
Are you going to do aggregation or MapReduce type stuff to it?
Are you going to do joins?
Like, there's just a lot to consider.
And when you're new, of course, that's extra hard, right?
Yeah.
You don't even know what a join is.
Like, how do you, like, evaluate whether you need one?
It's even something that we, as more seasoned people, forget sometimes.
I mean, sometimes I'm doing some speed comparisons of something, and I just change the format to maybe one that's more preferable to that particular library or tool.
And, oh, yeah, that speeded it up five times.
I don't even have to optimize code because I chose a different format to read the data from.
Yeah, it's really amazing.
So then another area that you talk about is web scraping.
And I think web scraping is pretty cool.
Web scraping and APIs kind of go together, right?
Like, there's so much data out there.
You just got to go get it.
Yeah, and I feel like that is really empowering first step when you're beginning is, oh, yeah, I scraped, you know, my favorite website or whatever it is.
I feel like that can be a really empowering first step.
And it doesn't take that much Python code.
I used in the book, I used some scrapey, which is about my favorite library for doing web scraping and web crawling.
Yeah, scraping is cool.
Oh, I had Pablo on the show a while ago.
And his story with scrapey is really cool.
Oh, that's awesome.
I'll have to check it out.
They're a really great team.
I had a chance to meet them out of PyCon a while ago.
And they're just really awesome, inspiring, you know, fully remote team.
So how much do you have to be careful about usage rights or restrictions on websites if you're looking at data for, like, internal use?
You know, this is just something that's coming to mind as I'm thinking about web scraping.
Like, if you want to answer a question for your company, how open is the web for you?
Do you know?
I feel like I go over quite a lot in the book.
And also, you know, whenever I give talks on web scraping, that you need to look at the terms of service of the website.
You need to look at the robots.txt file.
You should be a conscientious scraper.
These are important things to think about.
Clearly, there's been times where I've found APIs that are undocumented.
And I've reached out and sometimes they're like, no, don't use that.
We didn't know it was out there.
And other times they're like, oh, yeah, fine.
We don't have documentation for it, but you can go ahead.
So I think that's, you know, that's the best practices.
I know quite a few people that don't follow those.
But I do think it's really important to be a conscientious scraper.
And there are, of course, media laws around that.
And so you don't want to find yourself on the wrong end of a lawsuit because you were a little bit too lazy to read the terms of service.
Yeah, absolutely.
I feel like those kinds of challenges are really hard because it's not just knowing the laws of your country.
Right.
I mean, it's all the countries.
You might be interacting with many of them.
Right.
And so it's extra hard.
And, you know, Europe has different privacy rules than the U.S.
And it's a bit of a bit of a Wild West.
Yeah, it's definitely, you know, there's obviously no international regulation of this at any point in time.
And I don't even think international regulation would work properly for this.
But I do think it's a matter of being kind of an ethical person and saying, okay, am I doing this in an unethical way?
Probably try and avoid it.
It seems like a good rule to live your life by, really.
I mean, especially if you're going to go write an article, right, if you're doing this for like data journalism sort of thing.
Yeah, clearly.
And a lot of times, honestly, if you pick up the phone and you call people or you write an email, people are more than willing to help share the data with you.
And I feel like we're kind of in an era of open data everywhere, open APIs everywhere.
And so I think people are very responsive to that.
Yeah, totally agree.
So the last thing that I want to talk about that you cover in your book before we get to your cleaning data story is about automation and scaling.
What do you guys do with regard to that?
So I think that we just give a little bit of a taste of that.
And this is this idea, automate the...
Automate the boring stuff with elsewhere.
Yes, yes.
Sorry.
That's a great one.
Yeah.
And I think that this is a really important skill as a beginner that is just like, wow, that's magical.
I can just have something run and it runs either via Cron or it runs as a salary task or however it runs.
And wow, it just did it on its own.
I think as a beginner, that's a really exciting moment of just like, oh, my program's alive.
So...
Yeah, if you can take something that...
Yeah, yeah, that's awesome.
And if you could take something that's like four hours you had to do at the end of the week that's super painful and repetitive and make it a button press and remove all the human error.
Like, that's magical.
Yeah, and I think it's really, you know, it's something that people can give back quickly to their team.
So if you're just on the side like, okay, I'm going to learn Python.
I want to do this one report I do every week in Excel.
I want to do it on my own.
And then, yeah, you're right.
Like, they can use Flask or Django or something and turn it into a one-click button.
That's, you know, they're the hero for the next year or so.
Your book seems really interesting.
And if you're just getting into Python from this data angle, it's definitely worth checking out.
Yeah, thanks.
Yeah, you bet.
So let's move on to something that you did.
Was that this summer?
Fall?
It was this fall, right?
Like September, right?
The PyCon UK?
Yeah, yeah.
So this is a bit, my PyCon UK talk was based a little bit off of my initial talk on this topic, which was at PyData Berlin last year.
All right.
So at PyCon UK 2016, you gave a talk called Cleaning Data, something that, what was the title exactly?
I don't think I have it written down here.
I believe it was Automate Your Data Cleanup with Python.
I think it was really interesting.
Like, you really laid out some of the places that we get data from.
And then you gave a bunch of libraries and techniques to fix it.
And that's what I found really interesting is I feel like if any of these problems that we're going to talk about line up with a problem you have, it's like, and here's the solution, which is super cool.
Yeah.
And even if it doesn't quite fix your problem, looking through sometimes the source code and how people are approaching it can give you new ideas about how to fix it for your own particular problem set.
Yeah.
So you kind of set the stage.
Let me read a little quote from one of your slides.
There was a paper or an article or something called Towards Reliable Interactive Data Cleaning.
And you said something to the fact, or the quote said something like, such approaches require a clear definition of dirty data that is independent of downstream analysis.
Even worse, one consultant noted that errors might be found after some result is reported.
That's a really big problem, right?
Like understanding, even knowing whether you have good data or not, right?
Yeah, I feel like this is a massive problem and a problem that we see often.
One thing I'm pretty passionate about that a lot of people don't do is even just data unit testing.
So we have these unit tests for nearly everything.
And then we don't test, you know, what happens if we get a really odd piece of data or something that doesn't fit.
And if it just goes to the pipeline and then it's reported, this can take, you know, a few analysts down the line before somebody's like, wait, we had negative sales in January.
What does that even mean?
That's right.
That's the pit in your stomach that forms when you realize what has happened, because it's not just like a bug in some code that like, well, this website renders a little weird or doesn't scale so well.
It's like decisions could have been made, right?
Yeah, or maybe already have been.
Yeah, you have been made based on this data.
We decided to buy this company or not as an acquisition because of your report.
We decided to pursue this product line and not that one on your report or whatever, right?
You know, killed a product based on your report that was not quite accurate.
So, yeah.
Or, you know, yeah, like kind of this goes back to what we were initially talking about, about statistics being off about the election.
You know, it's a little bit of knowing your inputs and your outputs and being cognizant that you need to actually test for some of those things or at least have safety checks along the way.
Yeah, absolutely.
And so you brought up data unit testing, which was one of your recommendations, let's say, during the talk.
And you talked about hypothesis, right?
Oh, I'm a big fan of hypothesis.
It's really amazing work by David McEver.
And, yeah, I highly recommend if you haven't looked at it to take a look at it.
It's based off of Haskell's QuickCheck.
And it's this idea of deterministic testing.
It's really, really fun.
Yeah, I'm really impressed with it as well.
I didn't know about it until six months ago or something like this.
And then I learned, wow, this is really nice.
So, yeah, I actually had David on episode 67 if people want to learn more.
But the idea is you're going to do some kind of test.
And I guess the terminology we were coming up with was the type of test that you think of when you write unit tests are example-based unit tests or something like this, where you say, if I put in a five here and this customer that has registered equals true on them and I call this function, then this thing happens.
I'm going to do some kind of test.
And that's not really how Python hypothesis works.
It's more like there's an input that's the number.
There's an input that's a customer.
And you can let the framework change all the values.
And then you just make assertions.
Inputs like this should lead to outputs like that.
And it's really cool, I think.
Yeah, it's the idea of property-based testing.
So you're essentially saying like, okay, this accepts a list of floats.
And it should return a valid float that's greater than two.
And when that fails, then you know you have these edge cases.
And the fun thing is sometimes you just come up on the edge cases of whatever tool you're using or even the edge cases if you're using Python 2 of floats.
And that's fine.
But it's good to first understand, okay, what are the possible inputs?
And what do I expect the output to be?
And this helps determine in your workflow or in your data science that you're doing that you actually know what you're seeing coming in and that you actually know what you should be producing.
Yeah, absolutely.
And it both has the happy path and the edge cases where so many of the bugs and errors live.
Yeah.
So whether that's bugs in data or bugs in just pure algorithms, right, it's really important.
So I definitely second your recommendation hypothesis.
That's cool.
So we talked about having this bad data and not even really knowing.
I mean, having your report or your pipeline or whatever it is you're working on to process this data run to completion is not really enough, is it?
The problem is, is that just running something to completion, like you said, or just assuming that because there's new data in my database, that that means that everything has processed correctly is always a false assumption.
And sometimes you're correct in that.
And other times you didn't have the right checks.
And it's impossible to write perfect code, right?
We know this as engineers, as data people, that there's going to be bugs in our programming.
And because of that, we need to have, you know, real eyes on the problem.
And this is interesting.
This is interesting.
This is some of the things that I was looking at when I was doing research for this talk, is that within the academic field, they're actually determining ways to have the data cleanup process report.
Hey, I'm not quite sure about this one because it seems either like an outlier or the probability that it's correct based on the algorithm I used is very low.
And then actually taking that data and presenting it to the user again, you know, the next day or once a week and having the user actually confirm yes or no, whether the cleanup operated correctly.
And I think that this is, you know, an important lesson to learn from where academics are coming from it.
And also a way that we can help automation actually be a solution where we have buy-in from everyone, right?
Yeah, that's really cool.
And basically you take the least trusted data and you say, okay, based on your algorithm, like this, this, I'm really not so sure about the other ones that were cool.
And you can just show that to the user, right?
So you also had some, some other interesting stuff from academics as well.
This portion of TalkPyPyPundMe has been brought to you by Rollbar.
One of the frustrating things about being a developer is dealing with errors, relying on users to report errors, digging through log files, trying to debug issues, or a million alerts just flooding your inbox and ruining your day.
With Rollbar's full stack error monitoring, you'll get the context, insights, and control that you need to find and fix bugs faster.
It's easy to install.
You can start tracking production errors and deployments in eight minutes or even less.
Rollbar works with all the major languages and frameworks, including the Python ones, such as Django, Flask, Pyramid, as well as Ruby, JavaScript, Node, iOS, and Android.
You could integrate Rollbar into your existing workflow, send error alerts to Slack or HipChat, or even automatically create issues in Jira, Pivotal Tracker, and a whole bunch more.
Rollbar has put together a special offer for Talk Python To Me listeners.
Visit rollbar.com slash Talk Python To Me, sign up, and get the bootstrap plan free for 90 days.
That's 300,000 errors tracked, all for free.
But hey, just between you and me, I really hope you don't encounter that many errors.
Love to buy developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch, and more.
Give Rollbar a try today.
Go to rollbar.com slash Talk Python To Me.
One of the parts of that talk, or the second half of that talk, I guess, that I thought was really interesting
was the sort of catalog of libraries or Python packages that you could use for solving various problems
that were reworking your data into a way that's going to be much more useful to you.
So let's take a moment and kind of go through those and tell people about them because I think they're really useful.
And the first one that you mentioned was about if you have some kind of duplication.
It's called D-dupe.
Yeah, and that along with Probable People and U.S. Address,
they're all from this company called DataMade that works with journalists really to tell good data stories.
I believe they're based in Chicago.
I'm not quite sure.
But they have a few good talks on this.
And so they have D-dupe, which essentially, you know, kind of can go through your tabular data and say,
hey, these two rows look quite similar.
Are they actually the same data?
And they use a mixture of fuzzy matching and I believe stringetic distance to determine,
okay, this Bob Dole and Robert Dole are the same person.
And that's a really great one.
They also have used some of their same techniques to determine probable people,
which is their other library, which tries to essentially parse people names and determine this is the first name,
this is the last name, this is the title,
and U.S. Address, which they use to parse, you know, this is the street number, this is the city.
And it's all based on a very, very simple neural network.
And you can therefore train it on your own data.
That's really great.
Like probable people, you can give it like a first name, a middle name, like a nickname,
last name, you know, junior, senior, whatever.
And it can pull those out and say, well, here's actually the last name.
Things like that, right?
Yeah.
So it's really great if you have, you know, survey data or other things that haven't been processed,
which is most of the data that you have to do when you're dealing with just randomly generated data
or randomly input data, then yeah, these are really great ways to kind of cut down on your time.
You also mentioned some for string matching.
One was called Jellyfish to do approximate and phonetic matching of strings.
Yeah.
So in this, you know, speech to text world, we all know that sometimes things go wrong.
So if you have, you know, strings that you potentially need phonetic matches for,
then Jellyfish is a great tool for that.
And it can help you find kind of maybe some of the errors in this, you know, auto transcription
or speech to text.
Oh yeah.
Very cool.
What about Fuzzy Wuzzy?
Fuzzy Wuzzy is one of my favorite libraries.
You got to love it just for the name, right?
Yeah.
Right.
And it allows you basically to, you know,
take strings and do Levenstein distance between them.
And this can help you either in like a token sort way.
So if it doesn't matter the order of the words, just that the words are the same.
And they use this a lot because they actually sell game tickets.
So whether I say it's the Steelers versus the Patriots or the Patriots versus the Steelers,
it's the same game that I'm likely talking about.
And so they can do, you know, token sort matches and they can also just do partial ratio matches.
And it's really useful if you just want a quick installation to work on string distance.
Yeah.
That sounds great.
The other one, the next one, rather, I really liked the name.
There's a lot of great names in here.
The next one is Scrub-A-Dub.
Yeah.
So Scrub-A-Dub is great if you've ever had to work with medical data or maybe even potentially
customer data and you essentially need to privatize it before you do reporting.
Scrub-A-Dub is going to try to go through, find these personally identifiable pieces of information
and remove them or replace them with like UIDs or something like that.
Yeah.
That's really cool.
You know, another area that comes to mind that that might be useful is developer data
for like web applications and stuff.
You want to take the data that drives your website, say, and you want to put it on the
dev machines and staging machines and whatnot.
But maybe it's got like e-commerce information in there and you want that gone.
Right?
Yeah.
Like maybe you, like if the developer loses their laptop or it gets broken into, it's like,
well, it's not really that important.
That would be great to say.
It doesn't happen very often, I think.
Yeah.
I mean, and there's quite a lot of tools too that you can use to generate, you know, fake
data with that.
I believe, I think one of them is called Faker.
Yeah.
Faker.
You definitely mentioned Faker.
Faker is cool.
And it'll generate all sorts of interesting data.
It'll do like addresses, right?
Yeah.
It can do addresses.
It can do people.
And then you can write your own methods and it will generate those.
So, so you need to, I think it even actually has credit cards built in and other things
like that.
So.
Oh, that's excellent.
Yeah.
Because one of the things that's challenging is it's harder than it sounds, I think, to
generate real looking fake data for a unit test for web design.
Like if I'm going to look at like a profile page, if it doesn't like have realistic looking
data, well, maybe my design isn't really going to capture what it should.
Yeah.
And kind of goes back to what we were talking about with testing your pipeline.
If you don't have realistic enough data, then everything can look fine until you throw actual
data into it.
Yeah.
That's for sure.
Yeah.
And it's cool because it'll do addresses in different locales.
You can say, give me a US address.
Give me a German address.
And of course, like the zip postal code comes in different orders and stuff like that.
Yeah.
I definitely was impressed with that one.
Another one that made me happy that you talked about was pint.
Because who wouldn't want a pint?
Actually, how big is a pint?
I mean, that's like how, like, well, to say a liter.
Like, I have no idea.
My wife is German.
She asks me these questions all the time.
How many ounces in this thing?
I have no idea.
Like, ounces don't make any sense.
Like, I just have to compute it.
Right?
And so pint kind of addresses this problem of conversions, right?
Yeah.
So, and it does it in really, really simple terminology.
And for that reason, you can do things like multiply meters by centimeters or convert easily
from feet.
Yeah.
That's a lifesaver when it comes to maybe you have an international application.
We don't really have a good way of internationalizing units built into Python.
And so for that reason, yeah, pint is a great tool.
Yeah.
Pint is really cool.
And you can do, like you said, you can compute with the measurements within a particular scheme
of measurement or convert between them.
So, for example, you can say like three times unit registry dot meter plus four times unit
registry dot centimeters.
And it will compute that.
Or you can say five times foot plus nine times inches.
Now, tell me that in meters or something like this, right?
Yeah.
And that's just a lot of math that you don't have to do yourself and that you don't have
to check yourself.
Sure.
And it sounds simple, right?
Like, okay, well, there's, you know, like 3.31 feet per meter.
So you divide like this.
But we've had like billion dollar spacecraft like crash straight into the ground because
somebody inverted that multiplication or something, right?
Like they needed pint.
Yeah.
Not quite sure I'm going to challenge NASA or SpaceX, but sure.
No, of course.
But definitely these sorts of problems have vexed like really important projects.
Yeah.
And clearly, you know, again, if you have, you know, an international site or if you're
dealing with data from numerous places and you want to make sure that it's correct, it's
better to rely on another library that has its own set of unit tests, that has its own
set of supporters and contributors and, you know, rely on the smarts of other people.
Yeah, absolutely.
Another thing that's really cool that I've talked about a little bit before on the show,
I think it was on episode 77, was Arrow.
And Arrow lets you deal with time in a much cleaner way, right?
Arrow is super useful if you're, you don't want to sit there and determine, okay, this
date is written in exactly this way.
It has an auto-recognition, which can really help you if you're seeing data, date times in
a few different syntaxes.
And this is especially great for if you have like a distributed system and maybe half of
your machines are set to report dates in one way and half of your machines are set in another
way.
You can still distribute the same code and it will actually function properly most of the
time.
Yeah, that's really cool.
It definitely adjusts for the time time in a much nicer way than the built-in one, built-in
date times.
And also it has a nice humanized bit.
So you can say like, given an arrow time, humanize this and say, well, that was about an hour
ago.
Yeah.
Which is great.
So you don't have to write, you know, all of the math around those and use like five JavaScript
libraries to have it written out to your users.
Yeah.
I've definitely, definitely written that code more than I need to write it in my life.
So another one that you mentioned, just a shout out to it again, is PDF tables, PDF tables.6.
That's really cool to go get the tabular data out of PDFs because, you know, who thinks it
belongs in there?
And nobody wants to look at the code underneath the hood there.
So again, hat tip.
Yeah, yeah, absolutely.
And by the way, all of these packages we're talking about, there's links to every one of them
in the show notes.
So just go back and find them that way.
Another one was called Data Cleaner.
What's the story of Data Cleaner?
The idea behind Data Cleaner is that it can automatically clean data for you.
Now, that sounds a bit too good to be true.
And it definitely depends on what you need to have cleaned and how you might need it cleaned.
But I feel like this is starting to get on the edge of what my biggest excitement when
researching this talk was, which is kind of the ways that academics are using machine learning
to automatically clean and automatically edit dirty data, if you will.
And I feel like we're definitely on the edge of this.
I mean, machine learning is very sexy and has had quite, quite a lot of advances.
And it's exciting to see it applied to something that is definitely unsexy, like data cleanup.
Well, that's what we should use AI for, right?
Yeah, right?
I mean, nobody wants to sit around and do string matching on their own.
Really, this is not a thing that any of us have a deep passion for.
So the more that we can automate this away, the better our lives will be.
The more time we can spend doing the fun stuff, like actual data analysis.
Yeah, absolutely.
Another one that you talked about was really cool, I thought, was about creating simple
domain-specific languages, basically, or parsers that parse simple domain-specific languages
called Parserator.
Yeah, so this is, again, from the DataMade team.
And this is kind of some of the back end that powers their U.S. addresses and their probable
people.
And you can essentially use it to find some different data structures.
And depending on if you have data available that's labeled, you can train it.
And I think that this is just a really great way that we can stop doing a lot of the difficult
work.
So this is the last time I want to see, you know, a 40-if-else statement line going through
and testing, is it this, that, or the other, to, you know, parse a name or to parse an address.
We can use machines to help us do this in far less code.
This portion of Talk Python To Me is brought to you by GoCD from ThoughtWorks.
GoCD is the on-premise, open-source, continuous delivery server.
With GoCD's comprehensive pipeline and model, you can model complex workflows for multiple
teams with ease.
And GoCD's value stream map lets you track changes from commit to deployment at a glance.
GoCD's real power is in the visibility it provides over your end-to-end workflow.
You get complete control of and visibility into your deployments across multiple teams.
Say goodbye to release day panic and hello to consistent, predictable deliveries.
Commercial support and enterprise add-ons, including disaster recovery, are available.
To learn more about GoCD, visit talkpython.fm/gocd for a free download.
That's talkpython.fm/gocd.
Check them out.
It helps support the show.
Another thing that you mentioned that I thought was interesting was DBpedia.
What's DBpedia?
DBpedia is a knowledge base based on Wikipedia.
There's also Yago that's based off of Wikipedia.
And it essentially is these, you know, knowledge base APIs.
And you have to write Sparkle, which, yeah, takes a bit of getting used to.
And for that reason, there's several ways that you can write this Sparkle using some Python helpers,
including RDF Alchemy is a really popular one and Surf RDF.
But basically, you can essentially get all of this data from this Wikipedia-based database.
And this can be really essential in terms of cleaning your data, particularly when you get into natural language processing.
And you want to say, okay, you know, I have William Clinton.
Like, what does this mean?
Who is this person?
And you can essentially go query Wikipedia via DBpedia or via Yago.
And it can tell you all of these different, okay, it's in topics, human people.
It's in topics, U.S. presidents.
And you can kind of get a lot of information from this that you wouldn't know just from saving the string and then having to have humans go through and annotate it.
Okay.
Yeah, that sounds really cool.
I mean, to be able to harness Wikipedia in like a structured way is definitely more useful than trying to scrape it.
Or I don't know how big the download of the data is.
I know it used to be huge many years ago, so it's probably even huger.
Yeah, it's pretty massive.
And this is a great way that you can just use it like an API.
Okay.
Yeah, very cool.
You know, another thing that you talked about that looks really useful and it's quite Pythonic in its style is this thing called OnGuard.
And you can use it to come up with decorators to apply to your functions to verify stuff, right?
Yeah.
So OnGuard is specifically for pandas.
And you can essentially decorate functions that, you know, essentially take in data frames and ensure that it has particular types or ensure that it's not empty.
Things like that.
And so this is essentially static typing for pandas, if you will.
And I think that that's really great.
I mean, I know that there's numerous opinions within the Python community about the need for something like static typing or type checking.
But when you work with data, this is really important steps.
And yeah, maybe you only want it to throw a warning, not an exception.
But these are things that are important to building scripts, building pipelines that actually work.
Yeah, absolutely.
Because it's super easy for something that you thought had a number to have, quote, that number in it.
And then math goes crazy, right?
Yeah.
And, you know, in pandas, there are certain things that are only available from particular d types.
So, you know, given that everything is NumPy based in pandas, if you have an incorrect d type, it could just go through and, like, add all your strings together and give you, you know, a really awesome large number that has absolutely nothing to do with the math that you expected.
Absolutely.
I expected a number.
I got a huge string that's full of numbers back.
I don't know why.
Exactly.
Nice.
All right.
Well, I think that kind of wraps it up for the solutions, stating the problem, these little tools that you found to solve it.
And I thought they were really cool.
So hopefully people will find them interesting to know they exist as well.
I hope so.
And if people want to check out the talk, it's on the PyCon UK YouTube channel.
Yeah, absolutely.
And I'm definitely going to link to it in the show notes as well.
So you can find it there on the page as well.
And then speaking of videos, you actually took this idea of data wrangling and data pipelines and did a couple of cool O'Reilly videos as well, right?
Yeah.
So I have one O'Reilly video that's very focused on pandas.
It's an introduction video to pandas.
So it's meant for somebody that's trying to learn about pandas that hasn't done a lot in it yet.
And it basically gives you an overview of these are the things that you can do with pandas.
And we play around with some data in Jupyter notebooks and such.
So it's a really great introduction if you've been meaning to check out pandas and you haven't.
And then I also have a new one coming out on data pipelines or automation workflows, if you will.
And this is going to cover things like Luigi and Airflow as well as Celery and kind of talk about how do you do distributed task processing and DAGs in Python.
That sounds really cool, especially the second one.
You know, I hadn't really heard of Airflow or Luigi.
So I suspect many people listening probably haven't either.
Maybe tell us what those are.
Yeah.
So Luigi is from Spotify.
It's been out, I believe, for four or five years now.
And Airflow is from Airbnb.
And it's actually incubating as an Apache incubator project right now.
And they're both really neat.
They have a little bit of different approaches.
But essentially, they're a way to build DAGs, which are directed acyclic graphs in Python, and to push all of your data through those graphs.
So essentially, when you think about, let's think about a problem like MapReduce.
We have a beginning, right, where we have all the data collected somewhere.
Then we map the task.
Then we shuffle and sort.
And then we reduce.
And finally, we have our output files.
And when you think about that flow, you can see that it goes through a directed acyclic graph, right?
It only moves one way.
And it has only a particular set of nodes or edges that it will go through.
And for that reason, you know, you can take this idea and apply it to quite a lot of workflow or data pipeline problems.
And in the video series, I cover a few of those and how to parallelize them and how to work with them so that you get kind of an introduction to how to build these things and whether they're right for your tool set.
Yeah, it sounds really cool.
I feel like a lot of projects have these data pipelines and they're just hiding.
They're not really brought out in a real clear way, you know?
Yeah.
I mean, there's definitely like quite a lot of scripting done in this instead of actually building a pipeline, right?
So you have a script and it's calling, okay, when this finishes, then call this other function.
What happens if it fails then?
You know, you get an exception in your logs and you have to go through and say like, okay, do I start it again?
Or where did it actually error?
And having these instead as graphs, you can see exactly where they failed.
And both Luigi and Airflow, as well as we know, Celery has ways to go and retry those based on the failure status.
Yeah, absolutely.
And once you understand them in this data pipeline way, distributing them so they run in parallel might be a really cool thing to do.
There's a lot of stuff, right?
Yeah.
And I mean, this is just kind of, again, an introduction, but it's there, you know, it's to give you an idea of what's available and what people are using from a distributed scale.
And then you can determine, okay, is this right for my project?
Another thing that I covered in that, which is one of my favorite libraries to play around with, is Dask.
And Dask is amazing.
Basically, you can have parallelized DAGs directly on your local computer that is going through sometimes terabytes of data, just because it has the ability to parallelize it and to do out of core memory processes.
That sounds really cool.
I haven't played with Dask, but maybe I need to now.
Yeah, yeah.
And Matthew Rocklin is super smart.
So, we'd love to hear him on the show sometime.
Oh, that'd be great.
Absolutely.
Those videos sound really interesting.
And people can definitely check them out if they're interested in it.
Let's talk about a few other things you're up to because you're juggling many things, right?
I like to keep busy.
That's good.
So, you were involved in the first Pi Ladies chapter in LA, right?
I was lucky enough to be part of the original seven ladies that formed Pi Ladies in Los Angeles.
And that was a really, really neat experience and very near and dear to my heart.
Yeah, that's really great.
You must be really proud to see how far it's coming.
Yeah, it's kind of amazing to be at a conference somewhere in Europe and people will be like,
Oh, yeah, I run Pi Ladies.
I run Pi Ladies Prague.
I run Pi Ladies Moscow.
I run Pi Ladies wherever.
That's just so inspiring.
And I feel really, really great that the community has kind of just embraced it and ran with it.
And I feel like that's really inspiring for young women that are getting into programming.
Yeah, I think it's really great to have that support and just that whole structure.
I think it's really important.
And I think it makes a big difference in the Python community.
As I look to other programming communities that don't have these things, they're definitely less well off for it.
Yeah, I feel like there's a lot of communities that are now looking at Python as a really great example of how do you have a diverse community?
And how do you have a supportive and open community?
And that just feels really amazing to be part of a community that other languages are trying to replicate, if you will.
Yeah, I'm sure.
I'm sure.
It's definitely, it feels like a better place to be, in my opinion.
I love the Python community.
And that's one of the reasons.
So another thing, speaking of community, that you do is you help organize Pi Data in Berlin, right?
Yeah.
So I got asked to be a part of that at the conference last year.
And it's a great group of folks.
And we're constantly organizing the monthly meetups.
Sometimes we have hackathons.
And then we've already started organizing the conference for next year, which will probably be the first weekend of July.
So if you want to come to Europe, you should do so the first weekend of July and come visit Pi Data Berlin.
Yeah, I definitely recommend that.
And maybe I'll make it.
Who knows?
It would be wonderful.
Yeah, yeah.
We're going to have a call for speakers pretty soon here.
So.
Okay, excellent.
So I hear this a lot.
And I know it means that you're really busy.
But what does it mean to organize a conference, a Python conference?
Yeah.
So, I mean, right now it means quite a lot of emailing and telephoning with folks to find a good venue.
We have a few keynote speakers that we're speaking with.
And then we work, of course, with NumFocus, which is the overarching nonprofit organization based in the States.
And they're the ones that are really kind of running the Pi Data behind the scenes and really helping organizers like us with the tools that we need to set up a good conference.
Yeah, that's really cool.
One of the things I think is great about Python is there's not just one big Pi Data conference somewhere in the world and you have to go to it.
But they're in many places, right?
And I think that makes it more accessible as well.
Yeah, and it's really the support is there kind of with the same idea of, you know, having Pi Ladies chapters all over.
The support is there within NumFocus to really say, okay, you know, we have your back and we're going to help you figure out how to get good speakers.
We're going to help you figure out a good venue.
And having support like that is just, I think, tremendous for being able to make Python accessible and also make it amazing and great.
And I think this is why it has been able to grow so much within the data and scientific communities.
That's an interesting question.
And maybe we're talking about a little bit is just a few weeks ago, just this year, somebody measured and realized that Python became the second most popular language on GitHub.
And it's second only to JavaScript.
It displays Java, which cheer for that, right?
Current most popular one is JavaScript.
And I feel like that's probably highly overcounted because JavaScript appears in Python web apps, Ruby web apps, ASP.net.
Like, you know, everything that is up there has JavaScript, right?
So I think it's overcounting it.
People were asking, like, why is Python so much popular all of a sudden when it's been around for 25 years?
What are your thoughts on that?
I have my own, but.
I definitely come at it from a data perspective.
So I know I'm looking at it through my own Python colored glasses.
But I do think that the embrace of the data and scientific communities around Python has been massive.
And I feel like it definitely came from definitely second place to R, possibly second place to Java.
And now it has really been embraced within machine learning, within artificial intelligence, within the chatbot movement, and within data analysis.
And I think that that's a really strong thing because those fields are obviously growing right now.
And it's really powerful to have Python be kind of the up and coming language within those communities.
And even for something like, okay, I do Apache Spark and I only write Scala, the second in line is Python, always the PySpark library.
And so for these reasons, I feel like we are no longer the weird kid on the block.
Yeah, I tend to agree with that analysis.
I think that's a really large part of it.
I think there's more to it, but I think that maybe is the single, if you're going to pick one thing, I think that might be the one thing that's making the biggest difference.
You know, for example, like cloud computing, right, Google App Engine, and some of the other things that made running Python apps at scale much easier than it had been before.
I think that has something to do with it.
Yeah.
But the data is probably number one and sort of, because it's a new area where it's really growing fast.
Well, and it's just really wonderful when you hear frameworks released, particularly around new machine learning or deep learning, that almost always Python is supported from the start.
And that's just really powerful that even if the backend is C++ or Java, that they understand and know that the Python community will kind of pick it up and run with it.
And so they want to make sure that from the very beginning, you know, TensorFlow runs with Python and things like that.
Yeah, absolutely.
That's very cool.
So another thing, as if you're not busy enough with all this stuff, another thing that you're up to is you're running a data consulting company in Berlin.
What's the name of it?
And what do you guys do?
Yeah.
So I guess off Deutsch, it's Kjamistan.
But it's Kjamistan.
And that's from a fond nickname several lifetimes ago in job perspective.
And basically, we do data consulting.
It's primarily me.
Sometimes I get a chance to hire some folks to help out some friends and other people I respect within the data community.
And to kind of specialize in doing a mixture of natural language processing work and data analysis work.
And so I usually take on clients and get to work alongside teams a lot of the time or build proof of concepts.
And that's really, really fun work.
I bet it is.
Yeah.
Are there a lot of companies that maybe are not software companies per se?
There's some other specialty, but they've got a software team in-house, but not a data science team in-house.
Also, is it pretty common that they maybe bring somebody like you or your company in to mix in a little data science or natural language processing for a particular project?
Yeah.
So sometimes it's things like that.
Or other times, yeah, it's just a new startup and they need some data analyzed so that they can take it to their investors, things like that.
Other times, it's that maybe they have a data science team and the data science team is very busy doing one thing.
They want somebody that knows Python that they can trust their code to build out kind of a new proof of concept or a new idea.
And then they decide, okay, is this useful?
Are people in the company happy with it?
If so, then they usually will take it on their own or potentially hire people to help manage it.
Yeah, sure.
That makes sense.
All right.
Very cool.
That sounds like a fun job.
Yeah, it's really fun.
Very, very cool.
All right, Catherine, I think we're running low on time, so we probably need to wrap up our topics there.
So let me ask you two questions I always ask my guests on at the end.
And the first is we already enumerated quite the list, but there's over 90,000 packages on PyPI.
What one would you recommend to people that maybe they don't know about or haven't heard about?
Well, we didn't get to talk much about NLP, but I really, really love Gensum and spaCy.
Both of them are really changing and pushing the space of kind of where academia is with natural language processing and giving it and making it available for us mere mortals.
So I really recommend if you want to take a look at how to use neural networks with natural language processing, that you check out both Gensum and spaCy.
Okay.
Those sound really, really excellent.
And I have not tried either, so it sounds fun.
And when you write some Python code, what editor do you open up?
I use Vim, so I'm kind of old school.
You know, I think that might be the single most popular editor of all the guests.
So there's a wide variety of topics of editors, but that one might have the highest histogram bar.
Go Vim.
Right on.
Okay.
Well, that's really cool.
And final call to action before we say goodbye to everyone.
Keep in touch.
I'm at KJM on Twitter.
And you can find my company and blog posts on KJMastan.com.
And if you're interested in speaking at PyData Berlin, feel free to reach out.
Yeah, that sounds great.
That sounds like such a lovely opportunity to go to Berlin, to meet all the people, and to present there.
So definitely want to second that.
All right.
Well, I've learned a lot about data cleaning and picked up a bunch of cool tools along the way.
So thank you for your time, Catherine.
That was great.
Thanks so much, Michael.
Thanks for having me.
You bet.
Talk to you later.
Ciao.
This has been another episode of Talk Python To Me.
Today's guest has been Catherine Jarmul.
And this episode has been sponsored by Rollbar and GoCD.
Thank you both for supporting the show.
Rollbar takes the pain out of errors.
They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course.
As Talk Python To Me listeners, track a ridiculous number of errors for free at rollbar.com slash Talk Python To Me.
GoCD is the on-premise, open-source, continuous delivery server.
Want to improve your deployment workflow but keep your code and builds in-house?
Check out GoCD at talkpython.fm/GoCD and take control over your process.
Are you or a colleague trying to learn Python?
Have you tried books and videos that just left you bored by covering topics point by point?
Well, check out my online course, Python Jumpstart, by building 10 apps at talkpython.fm/course to experience a more engaging way to learn Python.
And if you're looking for something a little more advanced, try my Write Pythonic Code course at talkpython.fm/Pythonic.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.
Corey just recently started selling his tracks on iTunes, so I recommend you check it out at talkpython.fm/music.
You can browse his tracks he has for sale on iTunes and listen to the full-length version of the theme song.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Smix, let's get out of here.
Stay tuned.
Don't believe.
Thank you.