The 101 guide to deploying Django

Speaker 1: Um we have Julia ready and um Julia our next speaker is gonna talk to us about deploying Django so maybe you have had uh an experience with uh setting up a Django project and then the next questions come in how do I turn my locally running Django into a fully fledged website that everyone else can see. So Yulia, thank you so much for for joining. And uh I guess uh we'll see you soon and see if we have some questions from the internet. Um Enjoy the talk and we'll be enjoying the talk as well.

Speaker 2: Thank you. So uh hi, um let's talk a bit about myself first. Uh my name is Julia and I have been a software developer for some time now. Uh well five years exactly, and five years might sound a lot in technology years, but in human years I'm still a bit of a toddler. And one thing that defines toddlers is that they are very curious beings and they like to push the big red buttons to see uh what it might do So being curious myself, I sometimes get into pickles uh where I realize that I didn't know where I am or what I'm doing. And during one of those experiences, I ended up doing a lot of DevOps

Speaker 2: work for one of the projects that I uh worked on The problem was that I didn't understand much of uh DevOps. Uh I mean I thought deployments easy. Everyone can do it, right? Uh it's just that thing that gets the project into production. So um I I I didn't have any problem getting into it, but I realized that it's a little bit more than just pushing a button to to get things working. So since then I have quenched that curiosity with knowledge and based on that now I want to take you through a very short journey about what happens when you deploy and what are the steps Basically, I'm having this talk so maybe I can save you from reading

Speaker 2: a hundred articles that are actually tutorials. My job here is not to give you a tutorial or to give you a recipe Uh mainly I just want to to let you know what what you can be looking for. And uh hopefully I I will give you the tools to to just go and search exactly for your specific need instead of wondering Where am I where am I going to start? Like I did almost a year ago. So your application is ready. You packed your bags and you're ready to show the world what you can do. The app is perfect utopia. All the tests are working, all the functionality is there.

Speaker 2: So uh what are the first steps in your journey? And the first step is understanding. So what exactly happens when you deploy an application? Uh basically deploying is the first uh process of taking your local development environment and making it available uh on a remote uh host so that all the users can then access your application. So basically you just wash your code onto a server That server can be uh anything that has a continuous process and a unique IP attached to it. So You could turn your personal laptop into into that server, but that is not recommended. But just so you know, you can do it.

Speaker 2: Here are a couple of examples of servers that usually go with Django because we are a Django conference. I have grouped on the on the left side the the ones that do a specific thing and on the right side the ones that do another specific thing which I'm going to go into more detail a bit later. So um How does the code get from your laptop to your wild server? There are three main ways you can do that. Uh the first one you could just go and install it directly. It's just one of the things that you know you can do but maybe shouldn't know. It's very hard to maintain. It's a lot of hard work

Speaker 2: when you shouldn't be putting all that hard work. And that is just for one server. So Imagine you have something that skills and then you will have to go and SSH into each of those machines, install Python, install Django, copy the text from one side to the other. It's it's not really it's not really okay. But just you know, you can do it if you really want to. The second way is to use a container. Um for easy replications such as Docker. This way you can basically do what you did in the previous step, but instead of going through all those steps, you just unpack the container into your server. And this is uh this combined with shipping those containers into clusters such as Kubernetes is what most of the developers do.

Speaker 2: The next thing is to to go serverless. You have no infrastructure to manage and you but you depend on the vendor. Um, I don't have a lot of information about how well Django goes with serverless. I would imagine that since it takes a lot for the Django app to start, that it wouldn't be exactly optimized to go in a serverless situation. But uh that's something that I need to learn more about before having an opinion. But it's an option that also you should be knowing about. So before that, uh we need a server for the server. Django app comes with a server of of its own when you run uh Django

Speaker 2: uh run server, you you don't have just the application, you have a small server for it. But we need something uh other to access it. And Uh the the reason for that is that because Django is uh I mean the original Django application is single threaded, you you're you're going to want to access it more from more than one entry point. And the official Django documentation specifies two m main methods of deploying The first one is using WSGI and the second one is using ASGI or WISCI and ASCII for simplification. The main difference about them is that the first one only supports synchronous code

Speaker 2: and the one the second one is asynchronous friendly. That doesn't mean that it doesn't support synchronous code because it's backwards compatible. But uh usually it's mostly used for the asynchronous uh way of doing things. Uh so let's go to each of one of them and uh talk a little bit about them So WSGI stands for Web Server Gateway Interface, and it's essentially the standard that talks about how Python web applications are deployed. It first appeared in 2003, which in technology years means a long, long time ago. Python back then was on the rise and there were a lot of emergent frameworks to choose from. But

Speaker 2: uh having to to to choose one frame meant that you had to learn all the stacks. So people needed to to pay a lot of attention before deciding which one to to use. It would be like a a really difficult conversation whether to use jungle or flask nowadays, which we uh hope we don't have any more. Uh so there was no singular way of communicating with the servers. And um There was a need for that. So basically what uh PEP333 and later PEP3333 for Python 3 in 2010 uh proposed was a way of doing this communication that is singular and you could just plug in whichever framework you were using without having to to learn how to to communicate with the server

Speaker 2: And uh in order to understand better that uh better about that, we need to think about how users communicate with the application. Um I I bet you all remember that old story about what happens when you type something in your browser and press send So basically your request travels a long way and then it gets to the server and then you receive a request back. uh and in order uh and once it reaches that server then it needs to get to the application and basically um Um that what basically it means WSGI come and said that uh Let's let's have

Speaker 2: let's split this into two big components, a server slash gateway and an application slash framework. And then let's define a protocol in which how this communication is going to happen. So basically the server is going to demand that uh the the application is going to provide the server with a callable in order to send the request And then the application is going to send back a response. There is also the concept of middlewares where Basically the middleware could play both the role of a server and an application, but for this talk I'm not going into them. Um so Django

Speaker 2: already comes with this in place and uh I if you uh created a project from scratch or if you If you looked at the project skeleton in whatever project you looked for, you would see that there is a file called whiskey. py, at which until this year I have not paid attention to And uh let's look at a bit about uh at what this file is doing. So um after we import everything we have this line where we set an environment variable uh that defaults to the the settings file and then um you uh return something from a function and

Speaker 2: you uh assign that that uh return thing to to the application and let's see what's inside this function in order to understand better And here we're going to see that we have this line where we initialize the app registry. And when that set prefix is true, we're also setting the URL resolver script refrige to some predefined value. But if false, well, we don't sort uh set anything of that sort. And then we uh we initialize this class and if you look inside it we can uh see there is a call method, so this is the callable that WSGI was talking about.

Speaker 2: that is going to have two arguments. And the arguments are very important because they are described in the whiskey specifications. The first one is something that should be an object, a dictionary that should contain CGI style environment variables. And the second one is a callable that is accepting two positional arguments and one optional a status which is a string and a response header which is a list of tuples containing the header name and value like accept and then the URLs over there Or sometimes it might also contain a second um

Speaker 2: a third argument, which is only used when the application returns an error. And we can see, for example, the start response here that is going to contain those headers and the status and uh there then we send a response. And in order to better understand how this should look like, uh I have we have an example here taken from the PEP3G3 where we see that the the status is a string saying 200 okay and how the headers should should look like. But there are some limitations to the whiskey. The the first most important is that it's only synchronous

Speaker 2: and that means that every request that comes uh is blocked until it's done. So basically it's similar to waiting in a queue at a shop and uh if you have a lot of those those uh requests coming in and you have to wait a lot in order for the last one to to get solved. And this is why Whiskey opens a new thread for each request But what happens when we reach the thread limit? Well nothing. We just wait. And developers have uh have solved this problem by scaling horizontally, so you can have uh Multiple servers that can in turn spawn multiple threads, but you can only scale horizontally so much until all the resources are exhausted, which is something they call the C10K

Speaker 2: problem. Um there are examples of genre applications that are very big, but that's only to the expense of well, a lot of money basically. Also, uh another limitation of the uh of whiskey is that it only works with the HTTP protocol. And Uh we know now that there is HTTP2 and even HTTP3 which is not exactly mature, but HTTP1 is good for most of the requests, but it is stateless. So in order to maintain a session, you would have to send the information of that session with every request So uh and so for for some small small thing you could

Speaker 2: you might potentially have to do a lot of requests, for example uh if you are a stock application and you need to make sure that you have the latest information. Also only the client can access the server so there is no bidirectional communication Um so is ASCII the the hero in this situation? Um ASCII came as a spiritual successor, a successor to Whiskey, and um it came to introduce asynchronous web interfaces and handle all these bi-directional protocols. It can support a sync and await uh operation, it can uh also supports web

Speaker 2: sockets So there is one continuous um uh stream of information between server and the client throughout the the duration of how much that connection is being opened. Um So neither the client nor the server have to wait on each other to communicate. Also, it's compatible with Whiskey, so uh technically you could still have uh synchronous code. inside ASCII or you could uh go from a whiskey compatible uh application to an ASCII one And if we look at the ASCII. py file inside the project, we will see that it's very similar to WISC.

Speaker 2: So technically we could just change where we impro uh import the function from and actually and also change the function because in the other place it was getASCI application here is get uh It was GetWisky application, his Get ASCII application. So it's very, very similar. Now let's take a look at inside this application as well to understand ASCII better. Well, if we go inside this function, we will see that it's very similar to the other one. The only differences are these two things. So we know that we are using uh ASCII 3 as the version of ASCII and that we have a different handler. And if we look at the handler, now

Speaker 2: we see that the call function is an async function and also it has a different number of arguments and the arguments are very different. So let's take a look first at the arguments. The arguments are firstly the scope, which is a dictionary, with which which has to have at least a type key in order to specified incoming protocol and that type key should be for example um HTTP for example You need to have a receive callable, which is a waitable that will yield an event dictionary and also uh a send callable which uh

Speaker 2: is going to take an event dictionary as a parameter and then return the response at the end Uh and if we took it take a look at examples, we can see that the the the scope uh uh argument it's it follows the whiskey uh convention in a bit and it's very similar and there's actually a a map around the ASCII documentation and how to get uh the the whisky environment variables and turn them into ASCII scope. And you can also see for example how the event dictionary looks inside send. Um and after everything uh has been said uh

Speaker 2: you you see that we await for the response at the end. And um So how can ASCII save the day? Whiskey is okay for simple web applications, or uh as I said, there are cases where there are some really big Django applications. that have enough money to scale them in order to work synchronously. Uh but if you think about it, what does the software do? Well the Well, it queries the database. It does nothing. It just waits on each query to finish. And that is a lot of lost time. So uh ASCII can come and bring in performance that Whiskey was not capable of at this point.

Speaker 2: However, even if I said earlier that uh ASCII is somewhat backwards compatible and uh you can have synchronous code in s uh that works with ASCII. It is recommended to not mix synchronous and asynchronous code, mostly because synchronous code uh can block uh an event loop uh in asynchronous code. Such a situation can bring your application to a standstill if you're not being careful about it. But mostly it's it seems like ASCII is a thing of the future. So we are not there yet. Until now we decided that we want our server to be either whiskey compatible or ESCI compatible.

Speaker 2: But we're still still need to connect it to a server and then possibly to add a load balancer or reverse proxy and just put it out there. So packaging our software into Docker containers usually very helpful. Um or something else other than Docker, I actually realized while writing these slides that Docker is very close to becoming its own verb such as to Google something or Netflix and chill. I'm not exactly sure how many of us think of containers in terms of not Docker, but that's beside the point. Let's go back to the application. So we have a Django app that was going to spawn its own server.

Speaker 2: And uh we need to make it accessible from the outside world. So in order to make our lives easier and not have to install it every time, we're going to put all of this into a container so that we can easily unpack it everywhere. Most of you will know that in order to use Docker you need a Docker file. So let's look at a very basic Docker file that will help us deploy Um so the stockphilers by no means compete, but it does it basically just Does it. You first you need to import like the package that is going to hold the OS and everything. And for this I chose Python 3

Speaker 2: point eight alpine and then we're going to to copy from our local environment to to the Docker container requirements After that we're going to start install them with pip install. And after that we're going to copy the actual application and we're going to run the server on zero point zero point zero point zero at eight uh at port eight hundred eight thousand sorry um and Okay, that means that we're going to have a Django application that is accessible to only one point and we need to make it uh multi-threaded basically. So now we need to install another server on top of our Jango

Speaker 2: app. So let's imagine that I chose a WSGI compliance server for no particular reason. When uh it comes to whiskey, there are three main contestants. uh mod whiskey gunicorn and UWSGI. UWSGI and GUICorn are don't run in Windows, so if you run Windows uh you usually use Apache and m m mod wsgi or you install uh some WSL and from Ubuntu on it like I did because I have a Windows computer right now. Uh GUnicorn is the most recommended one out there. But I chose this one because I was familiar with it.

Speaker 2: So but it doesn't really matter that much. Any any whiskey compliance server will do. So basically all you need to do is just scratch that last uh command and replace it with a new command that We'll start the US key server and we'll use this initializer file where you have all your parameters. And you need inside parameters such as which socket you use , which is the module is actually the file that contains the whiskey. py file or whatever. How many workers you need, what do you want to do when you exit? Is this the main thread or not?

Speaker 2: Things like that. There are a lot of examples on the internet about how to do it. And now at this point we have more than one service and not all of them are going to be in the same container because we might need if we follow the the usual tutorial to to add something like Nginx on top of it for um delivering static file for example or maybe we want a load balancer or Whatever, like that depends to our needs. But now we have more than one service and uh we can't put them in the same container Actually, we could do it, but there's this rule of thumb that you said that says you shouldn't have one uh more than one process per container. So let's let's be nice and

Speaker 2: don't do it So we have Docker Compose to the rescue. Basically what Docker Compose help us do is just create everything like you would do just with the Docker run, but create multiple services at the same time and make them dependent on one another So in this particularly very, very brief example, we build a container for the application that is either whiskey or ASCII compliant. We also build a container for the reverse proxy and we link it to the initial application. We will not forget that we will, for example, in the case of Nginx, we will need another Docker file for it and we will need to add parameters.

Speaker 2: And if we do it for the static files, we will need to set those uh environment variables as well. We will need to pay attention to the port binding because that can give you a lot of headaches and two days of debugging and not know what's going on. And now that we have that, we go to the next step where we use some container orchestration tool and ship it somewhere else. And these tools basically do management of container-based microservice application across multiple clusters. And they have become essential in the development of very big systems, uh, and they're basically a buzzword for smaller companies And for good reason. They are scalable and configurable

Speaker 2: and easily maintainable. And they're actually not that hard to learn, maybe a bit harder to master, just like Python, but uh there are a ton of resources out there. There are a bunch of such tools among which Kubernetes or Amazon Elastic Container Service or ECS. uh Docker platform and more, but all of them will have to have some sort of configuration that you will need to ship alongside the deployment. uh which would look something like this. Basically you declare what kind of service you have, what kind of ports you're using. and what kind of dependencies you have amongst each other. So now that we have like the basics out there,

Speaker 2: let's take a look at some Best practices. This in time while experimenting with deployment, I have made a lot of mistakes that helped me learn some lessons. And I chose five. Yeah, five of these. The first one is always use the checklist. The Django checklist is there for a reason and checklists have been proved to help in remembering not to skip any step. And I I didn't know in the beginning how useful that checklist that uh exists in the documentation is, but really take a look at it and take note of it because it's there for your help. Also, it's very important to monitor, log everything that you can log

Speaker 2: and not just log, just Also read these logs, put them into nice forms and tables. Use every tool available that you have and it will save you a lot of headache. in uh finding out where things go wrong or uh where the application is not exactly as fast as you would like to where are the bottlenecks and stuff like that like trust in the power of logs Um be careful of your users and their sensitive data. Uh use environment variables with trust. Um be suspicious when it comes to security. I know that it's very annoying for developers, but not uh you you should make sure that you give them permission

Speaker 2: to to um deploy and debug in deployment carefully. And Just just take care of security. Like good security makes happy companies and happy customers and everyone is happy. And the next big one is to always take care of your Docker files. Well, if you're using Docker and not something else Basically the order in which you add all those commands matters because you could get up to very big build times if you're not careful in which order you're doing things. Also , I was talking about security earlier. It's very important to remember not to give root permissions to the server.

Speaker 2: So make use of that. Um but most importantly, your needs matter. Just define what you want and stick to it. You can read some tutorial Or you can listen to somebody else's opinions about how to deploy, but at the end of the day, you know better what you need. For example, at some point in the projects that I worked on, we decided to skip entirely Nginx. And just use WSGI with Django and then use another um another gateway on top of that So it was very it was very difficult at that time to to find something to make it work, but it was possible. And most of the tutorials out there would tell you why why would you want to do that?

Speaker 2: But yeah So now we're on to the future. I guess the emergence of serverless will make it easier for the developers to deploy, perhaps uh ASCII will become the norm. It's up to us to to make the tools that we want work for us. And um I guess it's time for me to wrap it up. So thank you for listening to me. And if you have any questions, I would be more than happy to answer them, both here and in the comments, or you can hit me up and on Twitter or GitHub or whatever. Thank you

Speaker 1: Thank you so so much. Uh Julia, it was impressive uh that you got around all these uh subjects. Um and uh thanks so much also for for sharing best practices at the end um that we are gonna share on as well. Um and um it was super interesting to see also that uh how how whiskey and ASDI look uh behind the scenes. The um Um internet might have questions. I'm not sure. But we are out of time Um so can I can only ask a very quick question because I wondered. So UWisky actually also supports ASDI despite its uh

Speaker 1: name?

Speaker 2: So what?

Speaker 1: U U Whiskey. It supports ASDI as well, despite the name

Speaker 2: Yeah, uh yeah, I I I I suppose it does. I I don't have a lot of experience with uh ASCII, so I only know about the whiskey part of it.

Speaker 1: It's uh I'm I'm uh I'm certainly looking forward to jumping into ASGI and and uh I'm also wondering should uh should we stop deploying with the W H D I now?

Speaker 2: Definitely not. I mean the not all the situations are such situation in in which you need asynchronous code. And asynchronous code comes with its own difficulties and challenges because you you need to make sure that everything is in perfect sync and the data is correct and stuff like that. Uh so I I don't think that there are all the cases with uh would be I don't know perfect for SP. And I think with the going to be the perfect solution for some applications, but it's good as the same ones that benefit from having No

Speaker 1: simple answers here.

Speaker 2: Yeah.

Speaker 1: Thank you so so much.

Speaker 2: It depends.

Speaker 1: And uh you'll be on uh on Sullib after this uh if there's any more questions. Thank you so much. We can have