Keynote with Mariatta Wijaya

Yeah, I think that's a good thing. Thank you so much for having me here. Um it's really an honor to be keynoting at DjangoCon. The Django community here is is really special to me. My first conference talk ever was Django Kong in Philadelphia only two years ago. The year the year after that I spoke at seven conferences, five countries, three continents. It's awesome. So I I I really feel excited to be back here and thank you so much for inviting me as a keynote speaker. Thank you.

Um before I start, just I have a few brief announcements. Um Zapier is hiring, GoDale, PyCascades. It's a conference happening in Seattle in February and CFP is still open. Please submit a talk and feel free to follow me on Twitter, but my next tweet probably about my ice cream selfie. Today I'm here to talk to you about core Python development. I'm a Python core developer and there is this one question that I receive. again and again and I think every core developer receives it too. That is when can you merge my pull request I'm sorry.

It's just today there are close to 1,000 open open pull requests to CPython. The problem is we're we're outnumbered. There are only so few of us who can merge your pull request. Meanwhile, there are like hundreds, thousands of you contributing, and I really, really appreciate all your contributions. They are welcome. But I'm sorry we just can 't possibly go through all of them quickly. We we don't have enough time. I'm sorry. I'm Canadian And um the other problem is our workflow is pretty complicated.

Like for every single one pull request that comes, oh my face. What are you going on? Somebody help me? Okay. For every single pull request that comes, we need to ask all of these questions. Bug or feature. It's it's really important. Did you sign the CLA? You do you wrote tests? Are the tests passed? Have they passed? Is do you is there an issue created? Did somebody actually ask for this change? You write a news entry, blah blah blah, backport. It's very complicated. And we do this for every single pull request. So today I just want to focus on one of these issues, the issue with backporting

Perhaps you don't even know what is backport and why is it even important for Python. So backporting is the act of applying changes from the newer version software. to the older versions of the same software. And perhaps it will help if you realize that Python still maintains these different versions. The master branch will become Python 3. 8. The Alpha release is due January. 371 and 367 still maintained. We still accept bug fixes to those. And it's due to be out October 20th. Backports is how you will receive 371-367.

Without backporting, you will not get that. So it's it's very important process in our development. So let's think about this scenario. Say you're using Python, you're using F strings, Python 3. 6, and then you discover a bug, maybe there's a problem with the documentation or whatnot. You come to the bug tracker and tell me, hey, there's a problem with F-strings. And because it's F strings, you thought Alright, this applies to Python 3. 6. Then you decided to help and fix the bug, making a pull request to the 3. 6 branch. But that's that's that that's not how it works. This happens all the time. People making pull requests to the wrong branch and we have to tell them

no close that please read the deaf guide. This is how it really works. You discover a bug in F strings, that's fine. Please do tell us about it. But then we're going to tell you that all right, this applies to all these branches. But first, please make the pull request to the master branch. We're going to review it, criticize it, um make sure you sign the CLA, make sure you have tests, write the docs and everything. Once we're happy with that, we'll merge it. Only after we merge it, we'll backboard it. Okay, sounds simple, right? Backporting Now let's go to figure out how this happens.

This is how we used to do it, to backport a single commit. We need to type out all of these lines. Git fetch, create a branch, cherry pick, push it to master, push it to origin, check out another branch, delete it. Stop, leave the terminal, go to the GitHub, create a pull request, and go back and do all over again. It's tedious. It's a lot of work. You're typing a lot of things and it's boring repetitive task. So thankfully we figured out a way to make it a little bit better. So I wrote this utility script called Cherry Picker.

And it's on PyPI. So all we need to do is install this instead of typing out all those really long lines. We type one line. Great. And it takes care of all those boring details, the leading branch, creating branch. It even opens up the web browser, pointing to the GitHub. pull request creation. So it's it's much simpler. And you think that we'll be happy with this? I was not. I I have problems. Problem is I have to be on a computer. That's a blocker for me. Like If if I were I was able to review pull requests on a phone, why do I have to move and wait until I go to my computer?

To finish the backport. I want to be able to do it from like if I'm vacationing on a beach on my phone, reviewing for requests, let's merge it, backport it. I don't want to be on the on the computer That's a problem. And because sometimes I have to wait until I'm back on the computer, I forgot to do it. So backports were missed. People start asking me to do it. The other problem is I started receiving back reports. Um One people complained that it was a pain having to install Cherry Picker because they didn't have Python 36

yet. It's a pain for them to install Python 3. 6. They missing out on F strings. And the other problem is some people use it on Windows and it's actually it's great. Python on Windows is awesome. The problem is me. I don't have Windows machine. So I don't know how to help them, how to support them So it is a problem for me. But even though it's simple, it's still just Boring. I I do not get smarter running cherry picker. I don't earn anything. It's it's not

intelligent activity It's boring and um I don't feel it's worth my time doing. But it's important, we need it And in fact, it was during PyCon in Portland last year, I spoke at another contributor and he asked the same thing, like when were you review my pull request and merge it and I'm I was I'm sorry I like I think I made up excuses I mumble a little bit about backboarding And it was him who suggested that, okay, how about code verbs don't worry about this? Let contributors, hundreds of contributors

take care of this for you. Let us do the backboards. And I feel like It sounds like a good idea at first, but I don't know. I don't think his time, your time is any less valuable than mine. If I'm not willing to do this, I don't think it's worth my time. Why should I ask you to do it? So It's time to build a bot. Automate this. The sooner we have the bot, the better. Problem is I didn't actually know how to build a bot at that time. time. I actually was waiting for another core developer to build the bot.

But time passed and we we need So I thought maybe it's a good opportunity for me to learn how to build a bot. So I I started learning. I started reading GitHub Developer Guide. And I learned about webhooks. So basically, you can configure your repository. You can add webhooks. To your GitHub repo and let GitHub notifies you whenever there's something happens inside your repo. It can send you events like, hey, there's a new pull request Somebody open it, somebody close it, or it can tell you about there's change in status. If uh the commit um the status change

to pending success failure all that it can tell you about pull request review was submitted dismissed there's a lot of different events that you can you can subscribe to. So when GitHub sends you all these webhook events, it will actually make a post request to a You just need to tell GitHub where to send that post request. So that's where your bot will be. You will be creating a web service. Anything you want, flask, Django, but it's a web service with a URL for GitHub to send that request. Now for Python, of course we're going to build

it with Python and pretty much in PSF 's infrastructure are run on Heroku So that's what I started learning, like how to build things, web, services with Python on Heroku. So that this is how the webhooks work. That's one direction. The other direction is how do you want you want to make actions to GitHub from your web service. Maybe you want to be able to create pull requests. You want to be able to Able to merge pull requests, leave comments, apply labels and all that. Turns out there are a lot of things you can do. Whatever you can do on GitHub, on the web, by yourself, there is an API for it.

So it got me really excited. So I thought, okay, now I think I understand how to build a bot. And I've started thinking like this is how it will work. I will Subscribe to the PR merge event, have my bot run cherry picker, and then I'm going to open a pull request with the REST API. Sounds simple. But remember, CherryPickle was really running git cherrypick. So it needs a copy. It needs to first clone CPython repo, right? Remember CPython is twenty-seven years old, is big. Cloning CPython takes two minutes.

at least is long and then I discovered something. In Heroku request times out after 30 seconds. So what what do I do here? I I I didn't know I've never built bots, I never do anything with Heroku. So I checked with um with Ernest, PSF infrastructure. He works for infrastructure for PSF. He's the director of infrastructure. And I thought maybe we need to set up our own server, not Heroku, something that's not limited. By 30 seconds of timers or anything. But he said, yes, totally. We can do this in Heroku. You will use salary, do it in the background dyno, in a worker dyno as a background.

task and there is no time so on it. Alright. Okay, I'm learning Heroku. I'm learning about GitHub API, so now I don't know how salary works. I guess I will learn about it it too. So he sent me documentation to salary. It was really straightforward. I was able to figure it out quickly. So sorry. So this is how it works now. This is the bot. I will have a web service. All it does is it receives the webhook event. And then it's going to just start a background task on salary. That is so the background task is we'll do all those long running things like cloning CPython, running cherry picker, opening pull requests

happens in the background This bot is called Miss Islington. So Miss Islington is open source. You can check out the code. It is Python 36. Um it uses AIOHTP, which is an async web server and client But the real magic that lets building bots working with GitHub API easy is GitGHub. GitGHub is a library maintained by Brad Cannon who is Another Python core developer. This is the library you want to use if you want to build your own integrations with GitHub. You want to build bots with Python

3. 6 and above. You skit it up. And you've seen the diagram, it seems. Sorry. It seems straightforward. Um, this is the actual code from Miss Islington from top to bottom. I will tell it when I receive the pull request close event, that's the first line. I'm going to check if the pull request has been merged. That's the if statement. From there I will receive the commit hash. I will know which branches. If we tell it to backboard, it will check the labels. And then basically in the end, that's the the

backport task delay, that's the that's the command to start a salary task. It's it's simple. It's easy. This is how hap this is possible because of GitHub. And when we built the bot, we deployed it, it actually worked, and it was Exciting. This is the very first time ever three of our bots talk to each other automatically I didn't realize this to work actually. I didn't expect it to work because I didn't write any unit test. I tested in production. I didn't know how to write tests at the time.

But It worked and it's like magic and now like all the problems I used to have with cherry pick girl, it's gone. I can do this from my phone. I no longer forget about it. I don't receive bot reports anymore because I only need to make sure it works with the bot. And it's no longer boring because I'm like, it's fun. Working with Ms. Link is fun. So but now there is another problem. It's still taking time. The time is related to part of our other workflow, waiting for tests to pass.

CPython runs lots of tests and status checks like this. We use Travis, App VR , Azure pipelines Waiting for all of this to finish on good days takes at least thirty minutes. For one pair. It's long. I'm I'm not gonna wait for this. So I need to do something else, try to come back to it later. Check if it's finished. If not, okay, wait again, come back again. Is it done? And so on. Like it it's it's boring, it's long That's a person waiting for time to pass. Spiderweb starts to form.

Wouldn't it be nice if I have a bot that waits for this and just tell me when it's done. Like that, right? So yes, we can totally do this. We have this. Spoiler. It works. This is how it I got inspired and I started figuring out how to build a bot that does this. So basically that is the Status event webhook you can subscribe to and then it will give you the commit hash and you can make another API call to get the combined status, all the status of the single pull request, of the single commit, and then you can post a comment saying that whether it's finished or not

Now, here comes one problem as I was trying to build this. I didn't know which pull request that commit comes from. That webhook event in the first that didn't tell me which pull request. request is from just the commit hash. And it took me time to figure it out. I I search in the pull request documentation, I search in issues documentation, could not figure out Which pull request contains this commit hash The answer was buried hidden in the search issues API. So it actually says this, like you can use if you know the commit, you can use the search issues.

API to search pull requests. Alright. So I I wrote out this um utility function. It's pretty straightforward. That's the URL, basically Just pass in the commit hash, make a request using GitGitHub, and I get it. So it looks more like this now. I received the webhook event. I'm gonna get combined status. Now I know it. Now if it's finished, I'm going to figure out the pull request containing that and then post a comment in that pull request. So that's that's how it works. And now all my problems really, really gone. Not only I figure out how to take care of the backboard, waiting for test

to pass is no longer boring. Like I feel so accomplished. My life is complete. Yay, let's start figuring out how to fix these other issues. And this email came. from another co-developer, Victor. He said, oh, what if what if we let Mrs. Lington merge the pull request herself? Um sure you want a bot that merge pull request to CPython? You're asking, can we give Python commitment to a bot?

We we don't even do this easily to people, real intelligent human. Doing this to a bot. Okay, I guess As long as it 's written Python with F strings, maybe it's okay. So we did it. We built a bot that merged codes into CPython and it is written. Win Python. This is this is the code with using GitGitHub async. io. This is how we merge code to CPython. You just need to You call that API GH put. So making a put request to GitHub pass the commit title, commit message, squash,

commit hash. That's all. Simple. That's how we did this. So now our workflow looks like this. If you realize just how little core developers are doing now. The bot does all the boring things And this is this is awesome. Like now core developers get to spend more time doing the important things, like actually reviewing pull requests. They're not bothered by waiting for your test to pass. They don't even merge it. Let the bot do it.

At this time, I realized something. First I didn't expect it we would ever want to let a bot do this. But now we have it. We can't live without it. If we are back to this, everybody's going to be mad and like telling me, enable Miss Islington again. We thought we were happy with just this. This is what we wanted in the first place. But this once we learn more of how to build more automation, more bots. We really can't live without. We let it. I know I know some other people now asking me to do more bots for CPython.

We're spoiled But now that I know how to build bots, I started asking myself, like, alright, I realized how simple it was. I started asking like what other bots can I build? I'm sure I can find more boring things in my life and let the bot do it for me. And in fact I now get grumpy when I know that Bots can do this. Why am I still doing this? So one of the bots I'm still working on is called Blackout. This is um this is a project I wanna finish soon Black is an uncompromising code for matter for Python.

And it first came out, I got excited, I love it, I want this, I want black everywhere I can, I want integrated in my life. And then I read the instruction on how to integrate black into my life. And there is this long list of instructions like If you use this ID, download this ID, if you use that ID, download this plugin and um pre-commit, configure it, and I'm like Just the mere thought of God I need to download something. No! What if I have more than one computer? I have to download the same thing again?

No! I don't wanna do it. So I wrote a bot that runs black So basically I don't even need to install black again or update it on my computer. I I just tell the bot, okay, I apply the label, blackout. The bot formats it for me. That's it. Nobody needs to argue. Nobody needs to install a plugin or whatever. Let the bot do it. I'm uh I have this working, I'm just I'm still not fully satisfied with how it works. So this is my next project. If you're interested, that's the source code Blackout Marietta Blackout

I use this other bot called PyUp. Basically, it's a service that helps keep my Python dependencies up to date. And it's really useful. I dec I recommend you all use it. Um, it's free for open source projects. So when one of your dependencies has an update, they made a new release to PyPI, you get a pull request updating your requirements file. It's great. My problem is I have to wait for the test to pass before I can merge it. And then I have to merge it. And delete the branch. If a bot can create the pull request, why can't it merge it itself?

Why do I have to wa wait for this? In my mind, throughout this whole workflow, the only thing that requires human intelligence, that requires my involvement is reviewing the change log. And to be honest, I don't even review change logs anymore. As long as it passes. My coverage didn't drop. Merge it. Right? I know a bot can merge this. So this is an actual open pull request in Mrs. LinkedIn and I'm purposely I don't want to merge this because I

I refuse to press that green button because I know a bot can do it. So if if PyUp doesn't provide auto merge ability soon enough, I'm I'm going to to build my own bot that just merge and delete the the the the branch like build a button. This is an idea that I have. Um I've been working on adding the PSF code of conduct to some of our GitHub repositories. So I added the code of conduct to CPython to the DevGuide tabs code workflow. By working on this, I meant I copy paste things. And you know, it's boring.

Basically I create the same pull request again and again. I think a bot could totally do this. There should be a bot that can go through every public repos under Python organization and add the code of conduct. I think it's important to have So I wish there's a such a body that can just create this for me, merge it itself, delete the branch. Don't I don't need to be bothered because a bot can totally do this. Since last month, I've been taking a break from any volunteering

activities, not contributing to open source since September uh yes, September and also this month. So I figured Maybe I should have one similar to out-of-office autoreply. I will have an out-of-office, out-of-open source autoreply on GitHub. So if you all mention me on GitHub this month, my bot will tell you that I'm sorry. Mary 's not gonna look at this for until November. I kind of didn't want to include this here just because it may be seen as product page, but a lot of people asking me how they can set this up for themselves.

This is done by Zapier. This is a multi-step Zap by Zapier. Um it's triggered by if there is a new mention of me. I still have to filter it like if If the mention is outside of Zapir, so if my coworkers mention me, it's fine. But everybody else, they'll get this message I I still get to run Python. I I there's a step for me that I need to extract out the pull request well number where the mention happened. And then I tell Zapiel to create the comment. So if you're interested, um talk to Zapier,

try out Zapir. Dave basely needs a bot and he didn't even realize it. Um he earlier this year he asked on Twitter like Still an easy way to allow issues but not pull requests. And he had good reason for doing this. I think one of the reply was moved to something else. That doesn't sound really easy. You can totally do a bot that takes care of this. It's it's as simple as this. Receive the PR event Close it. Much simpler than Miss Islington. I know this can be done. I wrote it. Those are

Those are the lines of code you need to write in order to accomplish this. It's easy. So I hope that I've given you ideas to automate your life. I mean I've give I've used GitHub examples, GitHub bot examples. samples throughout this talk just because that's where I spent my life most of the time. But don't be limited to just GitHub. Automate your life. Don't be a robot. Build a bot. And if it if it is GitHub bot you're after, I did wrote a tutorial

Telling you how to do it, go down GitHub Bot Tutorial. But yeah, automate Don't don't do the boring things. You know, look at your own life. I'm sure you will find even more boring things. So thank you so much. My name is Mariara. Thanks for coming to my talk. I hope you enjoy it. Follow me on Twitter or if you would like to get in touch. You can email me merit at python dot org. Thank you so much.