Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
This is episode 92, recorded August 23rd, 2018. I'm Michael Kennedy.
And I'm Brian Okken.
Hey, Brian. How are you doing?
I'm doing great.
Yeah, same here. I'm excited to be back together. And I can't wait to talk about some of these things.
We have some like amazing drop the mic sort of stuff coming up. So pretty excited to get into that.
And also pretty excited that DigitalOcean is sponsoring this episode. So thank you, DigitalOcean.
Check them out at pythonbytes.fm/digitalocean. Get $100 credit for new users.
Right now, have we covered the fact that Python is popular?
Yeah, but I wanted to cover it again.
Why not?
So Python is pretty popular. And for the second year in a row, it came out as the number one popular programming language in the IEEE survey.
So there's a every year IEEE puts out a popular programming survey and that it's at the top isn't really news.
What's really kind of cool about it, though, and we'll have a link to the results. It's an interactive results thing. I think we covered it last year.
But this time, Python is number one, even in embedded. And that's the thing that stands out this year.
Yeah. And it's worth pointing out that this survey is for electrical engineers. It's not like the same as, say, the Stack Overflow survey, which is maybe more developer focused.
So it's like you saw another area where Python is sort of taking over, which is awesome. Embedded, like you say.
The article that we're going to link to actually talks about whether or not this is real.
And some people are doubting that Python really is that popular in embedded because, I mean, there's a lot of day-to-day job are embedded.
The embedded work I do is in C++.
There's a big range of what it means to be embedded. So they don't really define that. You have to kind of define that yourself.
I define it mostly as you're developing software on a platform that's not the target platform.
And the target platform doesn't really look like a computer.
Like a Raspberry Pi or maybe like something embedded in your car or something like that?
Yeah, definitely. But even it's a broad definition.
So the embedded software I do actually runs on a Windows machine, but it doesn't look like a Windows machine.
It looks like a, it's a testing equipment.
It looks like a stereo component with dials and stuff.
We don't run Python. We run C++.
However, like you said, like Raspberry Pi, I think there's been a big push lately.
I think that's where this is coming from.
There's been a big push lately of trying to get people involved in more circuits and more real interactive programming with electronics through the use of MicroPython and CircuitPython on like Raspberry Pis and Arduinos.
MicroBits and all that sort of stuff.
And including the Mew editor that really tries to teach people right out of the get-go.
So there is some doubt of whether or not really Python's the top for embedded, but I don't doubt it at all.
I think it's going to stay.
Yeah, I think it's cool.
I mean, you could have the debate of, well, it's on professional shipping embedded devices that are for sale versus embedded in general, which like you say, could include $5 Adafruit type things that people play with or they're using in school, but it's not for sale per se.
Right.
There's a lot of people taking those Adafruit things and turning them into kits for people or products for people to like wear.
And then the people that are learning on that are going to go into, it might go into other embedded jobs.
And I think since Python is faster to develop in, there's a big incentive, I think, for people to use a higher level language and embedded if they can, even in a production level.
Yeah, and if they are, it's got to be Python, right?
Like what else would it be?
Right.
Yeah.
Maybe JavaScript, but that's a little shaky.
Very interesting.
All right.
So are you ready for something incredible?
Yeah.
Is this drop the mic thing you're talking about?
Yeah.
Unfortunately, it's the second one, so I'll have to pick it back up.
But let me work my way into this the way that it develops.
So we have the mailing list, Python dev, okay?
And that's where a lot of the core developers talk.
Okay.
On this list, someone was talking about finding a better way to test the C APIs instead of writing some C tests, some Python tests, and those kinds of things, right?
You can be sympathetic with this, I'm sure.
And they were saying, look, we need to test the C code inside CPython.
And one way to do this would maybe we could use something like Cython.
And we talked previously about how Cython lets you pretty easily integrate with C and Python at the same time.
So there was some debate saying, you know, it might be a good idea.
It might not be a good idea.
Like on one hand, Ccython is changing a lot.
It's a pretty big dependency to take as part of the core development component of Python itself.
Things like that.
But then something pretty incredible came along here.
And I don't know how far along this is or whatever.
But this guy named Yuri steps in and drops a bit of a bombshell.
And he says, hey, you guys, this Cython thing is kind of cool.
And I don't know if you want to do it.
But over at Dropbox, we're working on a new compiler for Python called mypyC.
And it just says, well, this mypyC will compile type annotated Python to an optimized C code, which ultimately compiles to machine instructions.
That's pretty cool.
How about that?
So it's like Cython, right?
And we've seen that where Cython can make things like 20 or 50 times faster in certain circumstances because it's compiled.
Really interesting stuff there.
But the problem with Cython is it's a superset of Python.
You have like the C def.
Like you basically write in a separate language that if you squint, you're like, wait a minute, is that Python?
No, no, no.
There's a lot of weird other stuff going on around here.
This is not Python, but it's Python-like.
And it doesn't even use the same type annotations because it sort of predated that.
So Cython is good, but it has like this, you got to write in Cython, not Python.
This mypyC thing they're talking about, this is straight Python 3.6 code.
It runs on CPython, but you can compile it to machine instructions like Cython.
It's not all of Python.
It's a subset, right?
It's a subset, exactly.
And so I think, you know, the main thing is you have to type annotate everything so it can actually compile it.
Like that's more or less the difference with Cython is you type annotate everything.
But it's a different type.
It's using Cython types.
Right.
But this is using just normal type annotations.
It's pretty incredible.
It says, look, this is standard Python.
It's a subset of Python.
And you can easily integrate with C libraries because of CFFI.
You can even run it on PyPy because it's a subset of Python.
So it's pretty cool.
And if Dropbox releases a full-on compiler for Python, that would shock the world, I think.
Yeah, definitely.
Like even with this, with a subset, to be able to develop it in Python and then do some little extra work to go to the, to make sure you're able to compile it within this, this would be great.
And the extra work is not much.
It's like put type annotations on it.
So maybe you have like three or four functions that you call that are like really called frequently or really a major portion of your time is spent there.
And if you wrote it in like C, it would be way faster.
Well, if you just put type annotations on it, you know, you could probably hit it with this thing.
So pretty interesting little thread that got picked up there.
We'll probably have to watch this and make, update people if we see any more progress.
Yeah, what I did not see is here's a link to the Git repo or here's the announcement or anything like that.
So we'll have to see if anything comes from it.
But I think it's really interesting, you know, that, that we've sort of gotten to this place.
Because from my understanding is Gito is not a huge fan of compiling Python code.
It didn't see a need for this, but he must be involved in this because it's from Dropbox.
And the main work he's doing at Dropbox is on my Pi.
And it's all about this.
So I suspect he's involved somehow, but I have no confirmation of that.
Yeah.
Okay, cool.
Boom.
Mic drop.
Okay, I'll piggyback on that.
Let's, let's talk about Netflix.
They are doing such cool stuff with Python.
Yeah.
So it's Python and more.
There's a kind of a really cool article called Beyond Interactive Notebook Innovation at Netflix.
And as we've all known, Jupyter Notebooks have, and using the Jupyter system has kind of revolutionized a lot of data analysis and machine learning and, and quite a bit of that interactive, interacting with data environment.
And that's true at Netflix as well.
And that's what kind of this article is talking about is a lot of the cool things that they've done.
And it's, it's over the last few years, they've seen a larger and larger growth.
And they've also said, basically, this is our supplied data chain.
So there's, I never really thought about all the different types of people that interact with data, but they, they listed, they have a little like diagram.
They listed like eight, nine or 10 different job roles that interact with data a little bit differently, but they're all interacting with it.
Like, you know, data experts or data analysts or algorithm engineers or data scientists.
There's all these different things that I'm not quite sure what they all do different, but they're all interacting with these huge data sets.
And in their effort to use notebooks and Jupyter to kind of streamline and make things easier for people in the process, they've came up with some really cool extra projects that they've open sourced or developing in the open.
And one of them is, I'm going to run through a few of them.
One of them is called Interact.
And it's an alternative user interface that has some cool things like inline cell toolbars and drag and drop cells.
I didn't know that you couldn't do that in Jupyter, but that's neat.
And a built-in data explorer.
And then if you've got a Jupyter notebook all set up with some data, well, it wouldn't be cool if you could have some of your states parameterized so you can run the whole notebook with different parameters.
This is almost like making a notebook like a function you can call and pass it inputs and get outputs.
Exactly.
And that's what Papermill is partly.
So there's a project called Papermill that parameterizes notebooks.
And then it also does things like if you've parameterized a bunch, you can analyze the different set of parameterized runs and do some analysis on the set.
And then there's, yeah, it's kind of making a notebook kind of, like you said, like a function that you can call.
And then if you want to, if you're using that and you want to share it with other people with the data and with the results, they've put together a thing called commuter that lets people share them together, share them with each other.
And then also scheduling.
So there's a separate article that's attached at the end about how if you've got this function with different parameters and stuff, maybe you want to, it's, you want to test different live data.
So you want to schedule it regularly or different times.
They've got a notebook scheduling system in place.
So all of this is, it's not like a super easy thing to jump into, but they're trying to make it easier for people.
And this is actually some pretty neat things that they've shared with everybody.
This is incredible.
And if you look at this big infrastructure diagram they have, it's like, it's just this entire architecture and cloud platform for running their style of executable notebooks.
Using this new UI on containers you can configure and schedule.
It's really crazy.
And you can even come down and say, I'd like to run on some container with four CPUs and 30 gigs of RAM.
Go.
Right?
That's crazy.
It's kind of amazing.
And I'm trying to start exploring a little bit more of what people are doing with Jupyter notebooks and other parts of the Jupyter system because it's, it is sort of opening up a lot of different data analysis into the open, which is neat.
Yeah.
That is really cool.
Something else I just see in here, and I don't know quite where this shows up, but in that same organization that released all of these projects on GitHub, you know, they're grouped into a bunch of projects under an org and the GitHub org style.
There's a thing called VDOM for Python.
What does it say?
It says a virtual HTML DOM for Python, which is pretty interesting for creating basically HTML and whatnot, like divs and paragraphs and stuff.
So that's also in here somewhere for whatever people want to do with that.
Interesting.
Yeah.
So I feel like this one is massive too.
Like this is a really major thing.
And, you know, organizations that depend on this notebook style, like this could really be a big deal.
And I like that they split it out into different projects.
Like the parameterization idea is, is powerful.
And maybe you can use that by itself.
And if you don't quite like the user interface, and so they've got the interact there that you can use that instead.
These are cool that they've split them out a bit.
I don't know how much they're tied together.
I expect things like Titus and commuter might depend on other pieces more than others.
But it's neat.
It's really neat.
So another thing that is pretty cool that I want to talk about is DigitalOcean.
And I've told you a bunch of things about it last time we talked about projects.
And that's pretty awesome.
Brian, you've heard of Heroku, right?
Yeah.
Do you know what Doku is?
Doku?
I think it's Doku, actually.
No, I don't.
So Doku is apparently a miniature Heroku powered by Docker.
So if you want to have your own Heroku, you can spin up a Linux machine or machines and install this thing called Doku.
And then treat it like Heroku for deploying and managing your code.
Oh, that's neat.
That's neat.
So over at DigitalOcean, they have a one-click deploy for creating Doku servers.
So you should go over there, create a new droplet, and you just choose, I want a Doku thing all set up.
Click.
Boom.
You're ready to make your own little version of Heroku.
Wow.
That's really cool.
Yeah, it's really cool.
So they have all these cool one-click things to create different types of environments.
And I just thought I'd highlight this Doku one because I know people use Heroku a lot.
And if you want some flexibility, this is a really awesome way to do it.
So anyway, check them out at pythonbytes.fm/DigitalOcean.
Get $100 credit if you're a new user to play with Doku or whatever else you want to play with.
All right.
So remember a while ago, we talked about running a Python script as a systemd service.
Yes.
That was a while ago.
It was.
It was quite a while ago.
I'll link to that when we talked about that as well, that episode.
But the idea was I would like to take some Python code and make it, you know, turn on when I boot my Linux system.
The idea was to take a Python script and make it just boot win Linux boots and run in the background.
It's kind of like if you did a cron job, but you have more control, right?
Like your app, your Python code is constantly running.
If it crashes, it'll restart if you configure it that way and stuff.
Well, that's great for Linux.
But what about for Windows, right?
So the equivalent infrastructure in Windows is called a Windows service.
It starts when Windows starts.
If you want it that way, it can run when you're logged out or it can run as a different user.
It can run as a restricted account, all these things.
So if you want your app to run basically as Windows runs, right, then you have to create this Windows service.
Well, it turns out it's quite easy to do this in Python if you use the right libraries.
So there's a cool article that just says how to create a Windows service in Python.
And it's based on this project called PyWin32, which seems to have no documentation.
Maybe I just missed it, but it doesn't seem at all obvious out of use.
But it is on GitHub and pretty popular.
But this article talks about how to create a Windows service.
And the idea is you just derive from a certain class and you override a stop, a start, and a main method.
And boom, you have Python code that runs as if it were part of Windows.
Actually, this is awesome.
I'll use this right away.
Yeah, yeah, it's really cool.
I don't have a good – I've written a couple of Windows services back in the day.
I haven't had a chance or reason to do so recently.
I did take that SystemD idea and do some really cool stuff around my courses for automation there.
But if people are on Windows or they have to deploy to Windows, I think this is a great little example.
So if that appeals to you, if that sounds like something you want to try to do, this is a really accessible way to do it.
That's cool.
Yeah, indeed.
I'm glad that this came out because I thought about doing this before.
I'm like, oh, man, I'm a smart guy.
I can figure this out.
I don't even know where to start looking.
There's like weird registration stuff you have to do.
But it's cool.
You just say like the name of the service is this, the description of the service is this, the way it starts is like on system start or delayed or whatever.
And then you just like run a command to install it.
It's great.
Nice.
Yeah.
You probably want to package it up though when you send it out.
Yeah, you should package it.
Have we talked about packaging before?
I think so.
Not enough.
It's one of the things that should be more accessible.
Okay.
Well, let's talk about it again.
So the Mamu Deshemi, I don't remember how long ago this was.
I don't know if we covered it or not.
But anyway, he wrote an article about different ways to package different levels of Python projects.
And then he did a talk on it.
And then now it's been all this information has been shared with others and edited.
And it's part of the PyPA, the Python Package Authority documentation.
And it's called an overview of packaging for Python.
And it's kind of like how it's broken out.
Because when you want to share Python code, there's different levels.
So if you just want to share a module with somebody, there's that.
And then so, I mean, it starts there and talks about how to just share a simple module with somebody else.
But then you quickly get into packages with source distributions and wheels and possibly binary distributions and how to do that.
And then, but that's complete packages are different than if you want to share an application.
So I do, that's the part where I get fuzzy.
But actually, this is a good starting point that hopefully they'll keep current.
That talks about some of the different ways that you can share applications.
Because, you know, sharing a web application is going to be different.
And if you share it on Heroku, it's going to be different than sharing a desktop application or a command line application or something.
And then also whether or not you want to assume people have Python or whether you want to package Python with it.
And what do you do with the dependencies and all that.
So we've talked about all this before and there's a lot of different solutions.
And so this is a good starting point to take a look at if you want to share some code with somebody else.
Start here and then get lost.
There you go.
I like it.
It brings a lot of the stuff we talked about together.
Like, for example, in the bring your own Python executable section.
It's like, here's a whole bunch of ways to do this.
You could use PyInstaller, CXFreeze, Constructor, which apparently I'd never heard of.
OS Snap.
OSnap.
Another one.
Like, there's a bunch of things in here.
And I actually have only heard about half of them.
So it's pretty cool.
And since applications kind of range, everything else is fairly consistent.
There's like one obvious right way to do it up through packages.
And then it's with applications where it starts going off and feathering out into different solutions.
I think it's because there's a lot of different requirements for applications.
Different than just sharing source code.
So I don't think it's a bad thing that we have a lot of different solutions right now.
We've talked about how it'd be really nice if there was one obvious way.
We're just not there yet.
So here's a good list of some of them.
Yep.
That's really cool.
Well done, Mahmood.
So there was a little bit of drama a few weeks ago around PEP 572 about changing the Python language for an assignment in place assignment operator.
Yeah.
Remember that?
And that was actually, that was part of the, one of the sort of straws that caused Guido to say, all right, I've had it with this.
Tired of fighting over these things.
I'm out.
Well, here's a new PEP that I think proposes a bigger change to the Python language.
This one is not accepted.
It's in draft mode.
So people can respond to it.
However, I actually think the value this one brings is massive.
It's a big deal though.
It's a really big deal.
So the idea is there's PEP 505 now, which is for none aware operators.
And the idea is there are several languages that are proven.
There's some nice design patterns or language patterns that short circuit working with none or null or whatever it is in the Swift.
I forgot about what they call it there, but nothing, right?
So there's basically two ways in which you work with this.
One of these are called null coalescing operators, which lets you substitute a value if you have a none object.
And null aware operators, which lets you chain operations regardless of whether with their null.
So the two languages that are most popular that come to mind have to be C# and Swift.
And both of these have deep support for this concept.
Swift takes it to a whole other level to say you can't even assign the equivalent of none to a variable unless you mark it as nullable explicitly, which is way far out there.
So the idea is like Python could benefit from this.
So there's these two cases, the null coalescing and the null aware.
So I'll give you the null coalescing, right?
It's if you're going to do some kind of weird test, like you'd say something like value, if value is not none else missing, right?
Like this big complicated thing could just be value, question mark, question mark, missing.
It's either going to be the value, but if value is none, then it's the other value missing.
What do you think of that, Brian?
I'm still on the fence.
Okay.
This one to me actually doesn't offer a huge amount of value.
It's okay.
The next one to me is like pretty killer and more with the fluent style of calling functions chained together.
So the null aware member access operator, it's the same basic PEP and the same basic structure, but it's used in a different case.
It lets you chain these fluent interfaces together without testing for none.
So suppose I have a user.
The user has orders.
The orders you can call a first operation, but if it's there's no orders, then it returns none.
But if it does return one, the order has a name, right?
So you would say like if user is none, return none.
User.orders.first.
If the first order is none, then we kind of bail out.
But if you make it through all those tests, you can get first order.name.
So that's a lot of code to be spoken over the air.
But you would now with this new proposal say user?orders.first?name.name.
And it would mean exactly the same thing.
Okay.
If you look at those two lines of code on digital paper, the amount of space on one and the other is ridiculous, right?
Yeah.
Sorry, I didn't mean to cut you off.
First reading of the null aware operators, I was thinking it's just going to make the language more complicated to teach because when you're reading it, it isn't obvious what these things are doing.
However, it does make the language smaller in that you can be more expressive in less code.
And that's a good thing.
I think it would make people more willing to effectively test for none when it could be none without cluttering the code, right?
Like these two examples I gave you, it's like eight lines versus one line.
You're willing to just go, oh, I just think it's there.
I'll just go through it.
And then you'll get none type does not have attributes such and such, right?
Whatever the attribute is, right?
You get that error all the time.
It's one of the most common errors in Python.
But if you just put the question marks in there, that totally goes away.
That's the thing where if you don't think you need that problem, just think about how many times you see that.
Like, you know, none does not have the index operator.
Yes, exactly.
Exactly.
And this basically provides a much nicer way to deal with it.
So my vote is for this feature, especially the null aware member access.
Yeah.
But I would be okay if Python never had it.
But I do think the languages that do have it really make pretty good use of it.
So it's in draft mode.
People can give feedback before some sort of battle erupts over it.
Yeah.
I think it's good.
Yeah.
I do too.
Cool.
It'll let you have a second edition or third edition of your testing book, right?
Because there's probably somewhere you got to put a question mark in there.
You're just adding work for me.
Thanks.
Anyway, it's something for people to check out.
Quite, quite good.
I don't have anything extra to share with the folks.
You, Brian?
I've got a couple of things.
I just noticed the other day that the PyCascades, it's going to be in February, February 23rd and 24th of 2019.
And it's going to be in Seattle this time.
It's the call for proposals is now open.
Oh, that's sweet.
You got a long time too.
You got it now through October 21st to get in your proposal.
Oh, I'm super excited about that.
The opening, the inaugural PyCascades in Vancouver, BC last year was excellent.
And I'm really glad I went.
It was much smaller and it was a really intimate event.
You got to meet a lot of the people who were there.
And I'm looking forward to have it in Seattle.
And it's coming to Portland next year.
So that'll be even better.
I'll try to go and I'll also try to submit something.
That'll be good.
The other thing I wanted to highlight is I just wanted to brag because I got a Test and Code episode out a week or so ago that was with David Heinemeyer Hansen.
And it really went well.
And he's not a Python person, but it's interesting information.
So that was fun.
All right.
So for people who don't know who David Heinemeyer Hansen is, what's his claim to fame?
Well, he's got lots.
But he's the guy that made up Ruby on Rails.
And he's one of the co-founders of Basecamp.
Yeah.
Just that.
So that's pretty awesome.
Yeah.
It's really cool that he was on your show.
And I'm looking forward to checking that out myself.
It's definitely applicable to Python people as well because we talk about language agnostic things like just testing and stuff like that.
Very cool.
Well, thanks for the call out on PyCascades.
Sure.
PyCascades.
But I also was catching up on your podcast, Talk Python.
And they had a couple episodes that were people that learned Python after they didn't get CS degrees and learned Python later, programming later.
And just the whole concept of that and listening to everybody's stories was very interesting.
I like those.
Yeah.
Thank you.
I did a two-part series on that.
And I think it really connected to a lot of people because so many people find their way into our world of programming in Python without going through the traditional CS degree path.
And I think it's just an example of a lot of people doing really great in alternate ways in.
So, yeah, very cool.
Well, good talking to you.
And we'll talk next week.
Yep.
You as well.
Catch you later.
Bye, everyone.
Thank you for listening to Python Bytes.
Follow the show on Twitter via at Python Bytes.
That's Python Bytes as in B-Y-T-E-S.
And get the full show notes at pythonbytes.fm.
If you have a news item you want featured, just visit pythonbytes.fm and send it our way.
We're always on the lookout for sharing something cool.
On behalf of myself and Brian Okken, this is Michael Kennedy.
Thank you for listening and sharing this podcast with your friends and colleagues.