Manage.py as the task runner

Speaker 1: Hello and welcome back live on stage. We have Eustus. And uh you're gonna tell us uh something about Task Runners. And uh I think we should start by giving you a big round of applause. The stage is yours

Speaker 2: Hey, thank you very much. And hello for you all. So did you like the uh Yes. Great. Nice thing to be in front of you all. And as I said, I'm talking about running tasks in a development projekt. But first as we start, I'm Justus Hammerleinen and I work as a It's hard to say because sort of like one problem with my sort of way that I I do my work is

Speaker 2: I I don't like to choose only a one job. I like to work in various kind of situations, in different kinds of projektin. With sort of like business side, with the tech side, sort of talking about what we should do next. uh moment talking about should we run uh things with uh uh pass scripts or like like how we should manage uh sort of like details. Currently I'm working on a small team and we work with a few projektia. One is Vegemi. Vegemi is a food education software that pitää familia jail eat more fruit. veggies. It tries to make it more fun. It tries to make people to understand better what are the benefits of of eating

Speaker 2: uh So the fruits and veggies. Then the second thing is I'm uh part of is Franklin that does electrical construction and home like building automation. You can already see the synergies between those two projects. And then the third thing that uh my team is working on is we work with a Pooh real estate agency that sort of helps people to find homes. And one thing that combines all of those projects is that they use Django and that's why sort of like I'm here. talking about something. And I'm talking about managed. py. And basically what it does, it already sort of like uh

Speaker 2: does a whole maintenance tasks that you have have in your uh dango projects and we already saw a lot of uh sort of uh commands that you use when you are interacting with uh bodelos and and schemas and creating sort of migrations for your database. And for a brief background, sort of like how I ended up being interested in Manage Spikes Mikä andttää why it's suin holista -projekte. Mutta alkaa sortta, how kodebas easilä yhdessä ja easilä. use and how because one of those things that I have noticed in many projects that you have a readme

Speaker 2: that is pretty long that sort of like has all of the steps that you need to be able to run the process that you are. How many you have a sort of readme in your project that you are working on now that sort of tells how to set up the system. And how many README 's are up to date? You are like how many of you are 100% certain that if I would take that README, I get the project running? Yeah, yeah. And this was one of those things that sort of like I noticed that they are not up to date. And then the other problem that I don't like to read. Like I'm I'm I'm sort of hired to write code, not to read paragraphs. So sort of like, uh

Speaker 2: how could you make things easier? And I came up with sort of like test free commands that if you have it makes a lot easier to sort of start interacting with a new project. Först är start skript. Basic om du har ett skript som vill sätta upp eller at tell att du har installat sam dependens som är näkött för projektet vill fa och tell that to. So you have one sort of like paste and it documents what you need to do to be able to run your project. Then test script and it basically runs all of those things that can fail. So you there's one command that you can run after your changes and no does the sort of continuous integration fail. And then the final one is deploy.

Speaker 2: And the reason is that there's often things that sort of like are hard or not totally working or needs changes. uh uh uh related to deployment process and really often nobody sort of like wants to work on that part of the application stack and I have maybe sort of like I have a this kind of like traumas of of being that person that ends up working on the deployment process and with all of the easy to use cloud tools and CI tools. and and sort of like being a little bit off of afraid of them. Well, that's a sort of like thing that let's say ten years I've sort of like wanted to get these scripts like working But the real problem is that when you are doing a sort of script that you want to work straight

Speaker 2: out of the sort of box , you'll end up using the lowest common denominator. And basically it means the pawn cell. And it I don't mean pash, I mean the sort of like uh pawn shell. And for example I personally use a corn cell in in In my development. So you end up in a situation that you don't have a language that is really nice jay to use Then you end up in a situation that you have some tools that you can use, but then for example if people are using Linux, there might be a curl or VJ installed by default. And then well you need a git. And did you all know that sort of like even if you have a program that is named the same but uh

Speaker 2: the GNU Linux people didn't want to use the real Unix flags? So you have this kind of like a Big problem with sort of like getting everything work. And then you end up installing kustom software and you basically are sort of writing a README to sort of get the start script running that sort of was point. to make the README less important. And even if you get this working, then there's somebody that comes to your project and starts using Windows and then sort of like everything breaks. Yeah. So it's sort of like But the thing is, I would like to have a widely supported

Speaker 2: Ken and R purpose language that would have a large user base and it would be already available that you wouldn't have to install anything And this is me understanding that we are working in a dungo project, so we have a Python already installed. And well this is I'll show you a picture of our next uh development meeting where I sort of told my idea what we want to do. So it's me tied other developers että hei, i it's a really good idea that we'll start using Managed. py for everything. So rather than calling Managed Py from other

Speaker 2: programs, we'll call other programs for. manage. py. And why to choose manage. py? Why not to use some other task runner or some other sort of thing? The main thing is that when we are working in a dungo project, we already are using it. We'll have to run migrations or like Micrate. Sorry, uh run server or test. All of those are the things that we are already running. So why wouldn't we sort of like go all the way there? And next let's check like how the managed pie works. So uh Do you hey how many of you know how many ways you

Speaker 2: you can uh call dango management commands? There's somebody sort of like has three fingers up. That's good. So we have dango admin. We have the managed. py and we have the python module django. And the thing is, all of them are the same thing, essentially. So sort of like all of them work the same way. You call it, you give them a command name and then it sort of finds that command jaunks it. But how the commands like where they are, how to sort of like we tells that what to run when you call. Mikrata oli runserver. Väjälen kommand on stillittanut the samma vai. Ja this on sort of great way

Speaker 2: with Dangho. There is no sort of special kaka within the dangho environment. You have basically app name, then you have management and command directories and within those directories you have a command name. py Which sort of like means that for start project you have a core management commands start. py for runserver core management commands runserver. For collect static contribute static files management commands collect static. py you'll start to see a pattern here So that's the way that Jangen

Speaker 2: knows the applications. But well, then there's a question that how it knows where to look thätään The core is easy because they are always available. But how all of the other stuff is found? And how many sort of like though the great thing for me with Dango is that basically everything plus comes back to the installed apps. Like almost everything, sort of like the great thing with Django is that it works the same way whatever you are doing. And this is a good example about So same way kuin static filet. Based on the order

Speaker 2: you have in your installed apps, it suits which static files based on the path it will load. You can then override existing static files, you can override existing templates, you can override existing commands, you can create new commands. And let's check now like what is the minimum like like thing that you need to do to be able to have a new command in Dango So to create a new command, we'll first need to create an app. And let's call it hello C. It would be hello command, but I don't have enough horizontal space with the font size I'm using, so we'll have to use short names for the

Speaker 2: Then in the settings files we'll add the hello uh command app and this is something that if you have a problem basically always for me it's the thing that take the time to put things in the installed apps. And if we check the hello C -direktori, we have management, commands jamand. py Then of course normal app files like apps py models. py. Then within the hello command. py We'll import from the Dung Core Management base command. Then we create a class that is named command. It's important it's named command. Command. Dango finds the right thing to

Speaker 2: run. Help, it's used to sort of like tell the user uh what what the command should do, then we'll just override handle uh method. And that's all. Like those basically like four lines of code, and then we can do anything that we want within a Python. So it's pretty easy from like from my point of view. of view. And then w after you have done of all of this you can run the command Python managed by hello command and it will answer. Simple. Easy. And it's not all. The best thing is that sort of with as with Dungo, somebody's already sort of worked on this, had a real

Speaker 2: like problems and needed to extend things. So it's really too easy to add for example flags or sort of uh uh uh options to your command. So you'll just uh uh override add arguments uh method where you get self and parser. The parser is instance of arc parser that is a sort of like Python module for parsing command-line arguments and with that you can just add for example parser add argument a name option uh and then action store tells that will want to keep the value that you are providing and help just this thing that you want to show the user

Speaker 2: uh when they ask for the help. And now what you get, you still have handle and in the keywords you will get the variable or the option that you have said uh in the name that you said on the other argument. And then you can do what you want with those argument. And again when you run the command and you pass uh pass that argument it will sort of like work. So it's really easy to add those things to those uh to to start creating command line tools with managed. py and then the extending part so so of like you can create new commands that is great but you can also

Speaker 2: change break and sort of totally modify what the sort of existing command Like there was an idea here that you can add a sort of validator that your pizzas don't have pineapple. But sort of you could create a make migrations tool that sort of just removes that information there. It doesn't sort of like take uh doesn't allow uh pineapples to be removed. Yeah. But let's see how to to integrate a tailwind uh to the run server process. So again we'll create a new app, detail, and again, short name because we have only so much horizontal space.

Speaker 2: We add the detail to the settings app and then we'll create the command Nyt näytän detail server. Ja koska meillä on detailillä ensin, niin dango näyttää runserverän kommand , itse kats ourän, not korva. When we are extending we need to do things a little bit differently. So we'll import from Dango Core Management Commands run server. So the command that we want to uh extent and we need to import because dango works in a way that it loads the file and it runs the class named command And if we want to have a command in our file, we need to

Speaker 2: have our class named command. So we need to rename it. because then you know what the command is that you are extending. You create class command and here you do override run argument Do people notice what we are doing differently compared on the last example? Does anybody want to yell? Yeah. And we are overriding run, not handle. But this is sort of like I didn't point point that out to you so much on the

Speaker 2: Nyt on, että why overridin run here, not handle? If we vote on the core server. py file jackän, kuinka Its handelkomi says restarting app, kunsaut filet. Ja sitten saa self-run, joka sähta. Starting servoilla itsellä ja sitten cleanupia. So because the sort of command class is just a normal Python object or normal Python class, you sort of like can Look at it and you can extend different parts parts of it so that you can sort of like piggyback on the things that it's already doing. So

Speaker 2: Basically, we'll extend the run if we want to extend the run server command. And again, we can do anything we want before we call the super run. So we can sort of like well, we have unlimited possibilities We can sort of like you can you can break things really horrible horribly or sort of you can do uh nice sort of like like really hard to catch bugs if you sort of like likes try to do two two complex things. But now we have a power of doing anything. So let's continue of of integrating Tailwind. Again we have a the directory, we have a management command

Speaker 2: Run servers. Tässä source that is our input. CSS. If we want to add Tailwind-related direktivoissä. Tässä static detail. This is our output. We output the Tailwind CSS, normal files, then of course package JSON, Tailwindissä NPM, sort of package. So you need to install the dependencies and then you need to add Tailwind config. And with Tailwind Config one of those things that we Because the directory that we are running, it's the detailed directory. We go one directory back and then just search all of the templates statik files

Speaker 2: and we generate the CSS that we wanna. for all of our uh things. And now we'll do the so this is the class command class and what we need to do to be able to run this kind of like a background running uh service. So we'll need to care get a process ID so so that it 's not c garbage collected before our code uh exits. So in the run we first want to have the command directory. That means the command directory is the uh detail command management commands directory. And from that we'll want to go to the app app direct directory itself so that we'll run things in right places.

Speaker 2: And then whatever sort of process we want to call we'll use subprocess So we open a pipe. And here we can example call the tailwind. uh to sort of watch and uh build our uh sort of CSS uh this at the same time we are running the run server. And here is the sort of we want to uh set the current working directory to the app directory so that it finds all of the files shell so it sort of like uh works and everything Exists what many sort of like like like uh terminal applications accept expect.

Speaker 2: Standard in uh would be Like prosesspipe. Then there's self-standard out jaf-standard error. So we'll just direct everything that happens to our application. will know what is happening there. And then we'll call super command self-run. And this way we can call here an esimerkiksi Tailwind, but we could call any application we want. So this is sort of like now we are using the Python managed by a shell. or any any application or any program that we can think of. And I'll show an example of calling async

Speaker 2: libraries because after that you you have the information of doing basically anything in managed by side. So if we would have async example command, we would have some async sort of like API calls or something that we want to do. We'll uh wait all of those calls to be ready. Then we have the normal command and then we'll just in the handle use the asynchron to run those those things. So we'll move to Async side if we need to and back to the sort of like sync side after it's done. And it's okay because all of the management commands are something that we run that start and end. They are sort of like synchronous by nature. And the great thing is

Speaker 2: when you start doing that is that it's just Python. So you sort of like anything that you can think of you can now do as part of your sort of like process. You can use any package. So if you want, you can take pandas, you can start creating sort of like like processes that take your data, generate some kind of of like information about it. You can call any program, you can call kit, you can call anything that you want and manage those things through the Python. hand side. It handles a lot of those things that are sort of like cross uh operating system, cross-platform things. And makes everything really easy in that way. You have access to the settings pie, so you don't

Speaker 2: need to manage environment variables. Y voidaan käyttää joita settingskilaan ja lisää tämä filma. Ja nyt on kommit Git ja miten python, not se, että se on samen skrivat. Ja näyttää nähdä, että se on environment. that needs to be set. And now sort of like after giving all of this power to you, like what you should do with it And it's I have a few ideas. So for example, NPN install. Tailwind is one of those tools that I like to use, but one of those things that it's a sort of node uh program so you sort of like have to handle different things

Speaker 2: but you can create a script that sort of handles all of that setting of setting up of the uh like like like uh installing all the dependencies, running it, being part of the run server command, being part of the collect static command, all of that you can do. yourself you could have a more complex sort of like build process for your front end and it also would be something that when you start run server it would automatically run. And it helps again people that are not so maybe sort of like in the front end side that you can set up things without having to move between two different uh world. You could uh have dev fixtures so that you have a command that uh would go take an SSH connection your to your dev uh servers

Speaker 2: would pull the media files from there, would pull the data that you have you have running there. And now you have the sort of like environment set up with data that is sort of like sane or at least more sane that you can sort of start learning about the application managing secret so you could use any but one example one password uh software development git you can use anything that you sort of like need, but you can start integrating more uh stuff within your application, not so that it's sort of like integrated, for example, in the CI side. You don't need to use GitHub Action, you could use Python Slack as DKI. you can

Speaker 2: generate a sort of send message uh command that sends a slack message if you uh using chat ops and now you have something that is in The Python side it's easier to debug because you can run it on your own machine. You don't need to sort of like go back and forth with the GitHub actions, sending a comment that I changed something that will it work. As it's dango, you have batteries included and it means that sort of you can get easily colored text. There are roles that you can sort of like use to set uh different colors and then there's teams so sort of like you don't have to say että always green with black background it handles okay and then the no-kolor situation

Speaker 2: Because there are situations where you don't have colors. Automatic bash komplet and of course call command. So that when you do something as a Command you can easily use it also in the dango code. So if you have some processing that you want to trigger within the sort of uh managing uh request you can easily sort of do that and I'll tell you one case where we use use it uh it's part of how we handle uh certificates so So we'll have a we have a GitHub action that is basically just a time trigger and it calls managed. py uh

Speaker 2: uh manage certificates. It pulls certificates from one password. If they are still valid, it doesn't do anything. But if there's less than one week validating them, it generates new certificates with DNS challenge and saves them to one password. And now our deploy process can read them from the one password and set up on the servers working okay. And this was important for us because we On the Franklin sanoissa deploymin asiassa lokal netvoikissä, johä on acessa ja kannattaa HTTP challenge. Mutta we still want to know acessa uh uh our sort of like internal services through an HTTPS

Speaker 2: without managing the whole sort of like self-signed certificate process But that's that's all. Thank you

Speaker 1: Thank you, Eustace. Do we have a question or two or three we can certainly manage a question yes question there in the back

Speaker 3: Hello, thank you for your interesting talk. You've certainly made some very compelling arguments, I think, for using managed PY. However, do you think there are any cases where it's not advisable to use it?

Speaker 2: I do I I have like I can come up with sort of like arguments against it and I have heard a lot of arguments against it against it. But I'll have to say that right now, and this might be a sort of like that changes after some time. But right now I feel that that's one of the best if like again we are working so that we'll commit fully on dango so we'll you use sort of not the best parts but the whole dungo thing so I think for our cases it's been a really That will do everything within that. Koskuita, secret management and environment variable management, that easier to understand and sort of like map. how things are used. So

Speaker 2: not really sort of like if you are not sort of committing totally to dango then it sort of like might not be that Good idea. But those cases where you are you are going to commit, it's really good idea. At least now, I feel like right now is so So I'll after two years I'll come to here with a talk that you don't want to do everything in Money Spy

Speaker 1: Great. Um more questions? I'd like to ask maybe about what kind of effect it has had on your README 's. Um compared to some before situation.

Speaker 2: Well, it it hasn't really affected readmes of projects that I I'm running. But it has changed so changed things so that people that are not me can also sort of contribute. So they are now sort of like at least codes that documents what needs to be done to get things running. So it's

Speaker 1: all right. So we are going to uh switch directly to the to the next talk. So hang around. But let's give Eustus a big round of applause. Thank you so much for the amazing talk. Thank you