Can packaging improve Django deployments?

Speaker 1: Yeah, my name is Markus. Um Very excited to be here today to give this talk. I also want to thank the organizers for doing such a great job organizing this conference here. It's great to be here. Um just a brief note before I start. My slides are online. Um I will show the You are at the end of my talk, so you don't have to write down everything that's on the slides. Okay. So few words about me. I started creating websites at the end of the nineties because I was fascinated by communication possibilities that the intent provided. So I started with Perl and CGI, moved to PHP, JavaScript, and

Speaker 1: finally discovered Python and later Django. Because yeah, I call myself like an open source software developer. This is because I never visited university Uh didn't really work out with me in the university. Never got there. Um and one of the main motivations for me to be active in this community is to give something back so that people that Also can't follow the usual path of learning. Um can we can all learn from each other I'm a member of the Leipzig Peiten dieser Group, so I'm from Leipzig, which is a city here in um Southeast Germany. And um I want to give a huge shout out to the uh folks over there because um I was able to give a

Speaker 1: preview of my talk over there and they helped me a lot with their feedback to improve the talk. So yeah, thank you very much over there in Leipzig. Um I'm also a founding member of the Django German Django Association, which was founded to do the DjangoCon 2010 in Berlin, first community edition. And uh I'm also an active Django supporter. We had like a workshop yesterday. And I'm a Open Knowledge Lab founding member, which is something about open data and Yeah, I'm getting people to learn more about coding. And finally, I'm the CTO of PicturePipe. It's a company that is offering some services around video submission and coding and streaming. Yeah, so that's the question today.

Speaker 1: How can packaging make deployments easier, faster, and more reliable? But before I start discussing the question, I have to uh explain three things. First This talk's not about publishing a junk project to the Python packaging index. It's just about the deployment. But using these tools for that Second, I'm not a packaging expert, so these are just my personal experience with these tools and what I did and what worked for me. But uh I'm still very happy about any feedback after my talk. So yeah, key topics I'm going to talk about first thing is like the quest. Second thing is packaging a Django project.

Speaker 1: Then after we package it Then a quick comparison of uh packaging tools and finally a short summary of what I have shown. So yeah. Um The quest is that I start with a short story about my first Python project. This was in 2007. I had to build a phone book web application with a web application construction kit called Nouveau. Does anyone know it or has used it? No. Okay. It's something that is built on top of Twisted. And uh this was the first experience for me uh with with Python and the web

Speaker 1: was very interesting. And the phone book uh was uh hooked up to an LDAP server And um it was deployed on a Windows server. And um the thing I built was uh bundled together in a single installer so uh that you can simply um install the single installer, it also installs the service, runs the servers all the time and it can run behind an um internet information server, Windows server. And this was a really uh exciting uh experience for me because before that I just deployed code via FTP and You never had any idea what happened on the other side and if if everything is like you would ha it like to be. And

Speaker 1: so my quest for reproducible and deterministic deployments began Okay, so let's have a look at Django projects and how it looks like if it's modified. to be packaged. Um so this is more or less what uh most people know But it has been a little bit modified so that it can be packaged using setup tools. There are a few important changes. First thing is that there is a manifest file there at the top. There you see the new. Oh doesn't work here. Yeah. And um

Speaker 1: then there is also a setup C VG and setup Py file here at the bottom. And um then all apps have been moved into a separate directory. Many people do that, but uh here it's done in a way that everything is Python packages and um all the configuration files have been moved into a directory uh called config. And yeah So let's have a look at these files and how they look now. This is more or less your uh standard managed pie file. The only difference is in line six uh here at the At the end. This doesn't really work here. Okay.

Speaker 1: Normally it says just my project settings. And this is so that it still can pick up this original settings file. And the WHI file file has been updated accordingly. So uh Small change. Uh these are the changes that uh have to be uh made in uh settings file It shows only the settings that have been updated. So of course SameSpy is a little bit bigger and has more settings, but it's only the stuff that needs to be updated. So you can see the app that is being installed also uses this My Project apps. uh namespace now and the uh UL root UL conf and WSGI application settings also use the myproject config namespaces

Speaker 1: Then you could in uh create a few more directories, for example for translations, for uh static files and for templates Uh there are no media or static root directories. You could create them if you want to, but it doesn't make so much sense because if you later deploy that, uh you can't uh Collect your static files where you installed that package and you also can't bring your user uploads where you install that package because When you deploy GAM, all will be erased and so all the user content will be erased and this is not what you really want.

Speaker 1: Settings has then up to be updated again after we added these three directories so that uh these directories are uh in the appropriate settings for locates, static files and templates. Uh it's just joining the base pass that is already provided by a settings pie with the directory name so also nothing so fancy so far. Yeah. So usually people use requirements. txt files to define the dependencies for a project. I think most of the people here do this. And setup Py instead of GSV files are used for packaging libraries. So if you publish a a generic Python library or a Django app that you want to be uh installed into different projects, you usually use

Speaker 1: setupy and set of GG. But uh why not use setfy and setfgfg uh um for both, so for dependencies and for packaging. And usually a setup pie looks a little bit like this. Um you have a few um functions, some code at the top, and then you have this huge setup function with all the arguments there. And usually these functions are used to fetch some information from somewhere else and inject it into the setup um function call. Uh but uh yeah. The code is so small because it's not really interesting.

Speaker 1: It's not what we want to do. Want to do something else. Because um In December 2016, Setup Tools 30. 3. 0 has been released and it has support for putting all that metadata into Setup CG. It's not used by so many people because um The people that want to install your package have to have the right setup tools version. And so the uh Python packaging authority people do not announce as widely to use it. But I'm using it and it works. And as you can see, everything is much more readable than before. You don't have any code here that is being executed any longer. And Um

Speaker 1: yeah, so it's a nice alternative to having this setup Py file. And it also brings a few uh interesting features here. You have uh you see it with a long description, you can say file colon, readme RST, and then it's reading the content from that readme RSD and it's it's putting it into the long description so you still have this feature that you before wrote with everyone with their own function in setup pie. So um no before there was this metadata section and because not everything did fit on a single slide I have a second slide here which uh has the options part Which defines a few more options uh to include the uh package data, which would be templates under the stuff in uh case of Django, and also defines the name of the

Speaker 1: uh package which is here my project just for the demo. The interesting thing is you can define all the install requires here and you could even pin them because um Um we are not relying uh we we are not uh um our project is not used by another project. We are like the the end of the uh of the uh chain and so we can pin dependencies as we want because um yeah nobody else will get trouble of this. If this will be a library, you shouldn't do this this way, but if you have a byte Shingo project uh it helps you a lot to get reproducible and deterministic installations. You can also use Python records at the bottom

Speaker 1: to uh limit the Python versions this can be used with Um so it would refuse to install if you have a different Python version. And the setup pie now looks like this. Nice. Isn't it? It's just importing the setup function and calling it and all the other stuff comes from the setf file. So the days of having nasty code in setup pie are finally over. And as I said, I'm really using this for Django projects and it really works. Looks a little bit strange, but yeah. So if you're doing stuff like this, there's one tool I can recommend, which is bump version. It's a tool to bump versions. And uh it can also be configured in setup TFG, so you can put the configuration of bump version into setup TFG, so you only have a single configuration file for everything.

Speaker 1: And it makes it very easy to bump that version of your package. What else you would need? What you have seen in that directory listing at the beginning is the manifest file. The manifest file is used to control which files beside the Python source code go into the distribution. And the important thing here is that it's evaluated from top to bottom So uh if you like include some stuff at the top and then later delete it at the bottom, it will not be inside the package. So you have to think a little bit about the ordering of the stuff. And what we do here is like we include our RST files, stuff like readme RST. With graph we say Graph the whole

Speaker 1: take the whole project directory and take all the data files that you find inside there, like templates and I don't know JSON files with some pictures or this stuff. images. But we say prune media and static root in case of uh someone um uh had left after development some files error so that these files not go into the package. So we take everything except these two directories. And we exclude all YAML files because usually you have like a lot of YAML files lying around the root of the URL um project. We also exclude the managed by file because it makes no sense to have a managed by file inside a package. And we exclude the uh cached

Speaker 1: um bytecode files and directories. Um a nice tool if you uh packaging stuff Also for other reasons. And having a manifest file is Check Manifest because Check Manifest uh looks at your good repository and looks at your manifest file and tells you if there the if there's something in your git repository that is not being taken care of with the manifest file. Um so that you don't by um by accident do not have stuff inside your package that you want don't want to have inside. So what about Manage Py? Usually you need Manage Py to do all the uh execute all the commands and we have just included it in the manifest in.

Speaker 1: So the answer is you can uh add a options entry points uh section to the setup CFG file. Entry points were also possible in setup Pi, so it's nothing really new. And you can define a command called, for example, side admin, but you can name it like whatever you want. And it simply calls the same uh function that managed by calls. And all the rest is only environment. So it's the same functionality that you have a managed pie. But the bonus here is you can execute wherever you are, because instance inside the past and you don't have to be inside the directory a managed piece, or you don't have to remember my managed by was. Yeah. So and building this package is one simple command, Python setupy uh

Speaker 1: bdist wheel, and then you get a wheel archive. Wheels like the uh the modern way of packaging Python distributions. And this would look like then like this. So you have a this directory where all the wheels are throughout in and because our project was named my project and it is uh the version 1. 0 the name is my project 1. Two point zero and um uh because we have had this uh universal true um Setting in the setup CFG is also for all Python versions and platforms. But you could configure this differently if you would have the need. But usually Django stuff is not with any C

Speaker 1: code also inside of not much. Okay, so how to install this stuff now? If you have packages Django project now and have the wheel lying around somewhere. Um what I've been using for a longer time is pip tools. I know that there are nowadays like modern more modern tools But I think at the moment these more modern tools are uh not really ready to be used, at least not from my perspective Especially because if you use some kind of service to upgrade your dependencies, then you will uh very uh quickly we recognize that these services uh don't work very well with

Speaker 1: uh stuff like pipfile. Um so yeah piptool has a few commands. The first command you can run is pipcompile. And uh uh if you say pip compile it looks at your setup pi and setup CFG files and looks at all the dependencies and compiles a complete dependency tree from that. And we put this into a file we call constraints. txt. You can even do that with hashes. So in this case, uh pipcompile hashes every package it finds and puts the hash also in the constraints um file. But then you also would have to hash your own package as well because if you hash, you have to hash everything you install, otherwise nothing can be hashed. So it's a take it or leave it approach

Speaker 1: And it has a pipsync command which can be used with the uh constraints or requirements txt file to uh Install everything that is inside it and uninstall everything which is not inside it, which is inside a virtual environment or even somewhere else. But maybe this is not such a good idea to execute this in your lecture uh OS installation. And this is how uh constraints TXT file looks like And the interesting um difference here to a uh requirement stick C5 generated by uh um by paprize is that you have all these comments.

Speaker 1: So you can clearly see what were your project dependencies and what are transitive dependencies that have been brought in by other packages that you are using And I have tried to choose a few libraries here that have multiple dependencies so you can even see that requests or six is used by uh several of these packages. Um So yeah, how do you install this? You simply call pip install and say um dash c constraints txt dash e dot because we want to install for development. So dash E means edit table so that you don't install the build package, you just install the source code that you have. So if you change it you can still use that source code.

Speaker 1: And the dash C Uh option of pip is something that um I don't know how long they have it, but it's fairly new. And the difference between dash uh R and a requirements file is that a constraints file is only um uh Um yeah deciding which version is installed. So um if the package which is inside the constraints file is not installed at all, it Also won't be installed. So it just helps you if if some package says okay, I want to install this, what the other package should be. And because the real requirements already find them set up uh

Speaker 1: CFG. But if we would install something else and uh that would want to install, for example, a newer Django version, then we would still stick to the Django version that we we have defined in constraints TXT. So even if we have a package at the end of our packages that wants to have a very very new Django version, we still would stick with that old Django version. Um and uh so you maybe think about what what about development tools? What we had now were only the production requirements we have for application Um and here you can use another setup TFG uh section which is called Options Extralls Require and there you can define sections and you can create a section called Dev

Speaker 1: and put if you um libraries you want to install in there or distributions uh and maybe because it's development stuff you don't even pin it And then you install it simply like this. So behind the dot you simply put in brackets the name of the section you have there and you can even have multiple sections and separate these sections with a comma if you wanted So, question is now, we have talked about development so far. So, how serve this package and how to get it really to the systems you want to want it to be deployed? So one thing you could do, you could simply install it from the file system. Pip can install um packages from the file system. So if you can copy that package somewhere.

Speaker 1: Where it's visible on the target server, you could simply install it this way. You could also use any HTTP server that serves the directory which contains all your packages. Or you could use a tool like DevPy. So the thing there is a link to the website. And DevPy is a project which can host your own packages and also mirror the PyPI server so that you have like the best of both worlds and you can even cascade DevPi instances. They're interesting too. Um And if you were to install that on the server now, this would be the approach you would take to installing it from the file system if you're in the same directory. This will be the approach to install it from a different path, somewhere else on the file system.

Speaker 1: And if you have this extra index URL uh option, then you could use a tool like DevPy, which is uh really behaving like a packaging index And so you could say okay install all the other stuff from PyPI but install my package from this extra index UR. Or if you use DevPy you could even use it alone because it will mirror all the packages from PyPI. So how to change settings now? Because after we installed this package, all the code is in site packages directory inside our Python installation, and of course we can't go there and fill around the settings files. So the best way to go with there is use environment variables for that. Except for the secrets. So if you have a vault, you should use

Speaker 1: this if it's possible. If not, then use everything uh use environment of variables for everything. And there are a few nice Python and Django solutions to manage um environment variables. NVDr , which is keeping all the stuff in a directory in single files, Django configurations , Which uh allows you to write very nice Django configuration files which can um inherit stuff from environment variables and n paths and environment config also help you to cast um stuff from the environment into specific types because if you use an OS environ you have usually a problem with this everything is a string and sometimes you need an integer or something else and this has all been done here so you don't have to reinvent the reel

Speaker 1: here. And finally running this with the WSGI server is also fairly simple because it's uh inside the Python path, you simply have to say GunnyCorm my project config. WSGI and then it's starting up and your application is running. So quick comparison of packaging tools. Um You could or should use NPM or Yarn for all your JavaScript dependency management, so all the stuff that you do in the front end. Because uh Uh especially Yarn is very good with this and you shouldn't try to reinvent the video here and uh if you use JavaScript stuff then use JavaScript tools to handle this.

Speaker 1: And in the end you could simply build bundles with a tool like WebPap or um other tools. And uh then um the uh the bundle that you create you simply put into the static directory. Um Python packages are something like the lowest common uh denominator Um because um not all packages uh not all platforms have package managers, but all platforms have Python because we want to run Django there, so Python will be installed. So Also uh pip will be installed and so it's very easy for us to install other stuff. And now that we have a Python package, we can even go further and use it in other environments. For example, you could use Condor.

Speaker 1: Condor is something from the scientific community, but uh the good thing about condor is that uh Uh packages can also be installed without having Python installed and uh you can even install non-Python dependencies with Condor. So it can help you with that too. Or you could use tools like PEX or Snapcraft, which create also like standalone Python applications, but I haven't used that with Django. Or you could use the platform package managers if it's necessary because your organization depends on RPM or DBN packages And it's fairly easy to convert a Python package into a platform-specific package. And of course you could also use Docker if it's necessary.

Speaker 1: Um because uh every Docker container that is equipped with Python can install Python packages. So summary. Uh if you package your Django application as a Python package, you are hosting solution independent. Because every hosting solution that uh hosts Django has Python, so you can install your package and you don't have to rely on the tools that they provide to you Um you use tools you already know and uh that you use to install other dependencies and you also Try not to um get into this not invented here syndrome so that you build tools just because you think it's better and uh someone else has already done this.

Speaker 1: And um it improves also the deployment of many servers because you don't have to do all the work on each server again. You just ship the package there and install it and that's it And uh the same release is also used everywhere. You could build build this package on a CI server, and after that run the unit test on the CI server with a package. Then you could Uh use that package on the staging server and if that's also going well you could use this on a production server and if it's not going so well you could use this package in a dev environment to figure out what's really the problem with that and not have to rebuild everything again And of course it's easy to do rollback because you just have to install the other package again. It's already there, everything is compiled, so nothing to be done. And a build distribution requires no build steps.

Speaker 1: So if you have things like C dependencies or other stuff that takes time to compile, it's already done So if you pip install this, it's simply put onto the platform and requires no compiler. And so you avoid to have tools like Git. GCC, GetHext, Node. js and all the other stuff on your production and staging systems because you just need it on the build system where you build that package. And after that, you simply have this artifact and ship it around. So thank you very much for listening to my talk. As promised, the slides are at the URL at the top. Uh they are also on my GitHub account. Um these are ways you can

Speaker 1: reach me. And of course I would love feedback on the ideas I've presented here because as I said it's only what what I've been doing so far and I'm not sure if it's like the best way to do it and what other people think about it and if it's really a a good way other people would also use. So come and find me and talk to me about uh the stuff I've presented here. I would love your feedback

Speaker 2: Thank you, Marcus.