New ways to deploy your Django app - Tom Dyson

Hi everybody, I'm Tom Dyson and I'm here to talk to you about alternative approaches for hosting your Django project. Uh, let me share my screen. Great. So I'm from Torchbox. We're an agency in the UK. We've been using Wag uh Django for uh almost since the beginning, pretty much since the beginning. We also the agency behind Wagtail, the open source content management system that I hope many of you are familiar with. But I'm not talking about Wagtail today. I would encourage some of you to, or any any of you, to follow along with this talk using the link here, tomd. org forward slash deploy.

Most of this talk is going to be a live demo and um the the the steps that I'm going to be going through are exactly uh the same ones on those notes and uh It may be helpful for you to go along with those. You will just need a few things in order to complete all the steps. The most important one is a computer with Python on it, which I would guess most of you have. And then it would be useful to have a GitHub account, some sort of disposable Linux virtual machine. And I've put some links in in these notes. with uh some a couple of sites that have a credit, $100 credit to get you started if you don't have one already, and a Google account. So the the purpose of this talk is to kind of address this question that I see a lot in Django support forums. We see it a lot, in fact, in the Wagtail support forums as well, but I think this is

it's more particularly around Django generally. It's people who've who've gone through the Django tutorial, who've worked out how to build a site, and then they get stuck about, and then the next step is making it live and they get stuck there. And you see a lot a lot of people making you know asking for help like this and they often kind of apologize, which I think is a shame because you know they shouldn't have to apologize for this. The the skills for deploying sites are pretty different to the skills to create sites. And um I feel like I I really like this last point here where someone replies to somebody else who's been struggling, where they say, um, uh it's good to know I'm not the only one. gonna keep on struggling till I don't realize it's a struggle. And that's it's funny and it's nicely written, but it was also I think it's it's a shame because um

I I think it's true. And for those of us who have um who have deployed a Django sites using Nginx and or Apache and Gunicorn Supervisor. or worked out how to use Heroku or these other systems, then if you do it enough times, then then you know it's hard to remember how much of a mountain it is to climb for people who have just learnt Django and are suddenly presented with all these options. And I should be clear that there are lots of fantastic options for deploying sites as you become more familiar with it. So there's the do-it-yourself approach, like the one I described, where you you have your own Linux servers and you install Nginx. and Gunicorn or USGI and so on. And then there are kind of great platforms like Divio where a lot of this and they provide 24-7

support and a lot of this is handled for you. So but but what I want to talk about today is the three options that um are not so well known and I think could be Really a good useful starting point for people who who have just built their first Django site and uh and and want to find quickest ways of of making it live. The first one is called Doku. And the way a way you could think about Doku is like a really simple open source Heroku. So it's actually a long bash script that manages Docker containers on a Linux server, but you don't really need to know how it works. You just need to know the two lines to install it and then how to push your site to it to make it live. The second one is static site generation. And this feels a bit like Back to the Future.

When I was first building websites in the late 90s, I would uh handcraft HTML files and um FTP them to uh to a shared server somewhere And this is kind of going, it feels a bit like going back to that because you're you're building sites out of static HTML pages, but with a very different tooling approach. And there turned out to be quite a lot of benefits to this, which uh I'll go through towards the end And the last one is a relatively new product from Google called Cloud Run , which is like you take your docking containers and as they say in their marketing somewhere, they convert containers into URLs And there are a few rough edges around this, but I think the model, this kind of serverless model of serverless containers, that sounds like really uh jargony, but I think this serverless model is a really interesting one and I predict it's going to become the

the standard way of uh of deploying applications. So it's interesting to see how that's working now. All right. The the most part of this talk is going to be a live demo and I'm going to try and do a lot of these steps in the time that we've got available. It's going to be a bit of a race against time. So you'll have to forgive me if it doesn't go perfectly and we don't get through everything. And I hope some of you will be able to follow along too using that link tomd slash. org slash deploy, which will take you to here. So we are going to start with providing we have something some of these things by building a Wagtail site locally. I've used Wagtail here. This can be any Django app. Wagtail is just a kind of is an example of a fully featured Django app that's quite quick to install.

So that's why I've used it here. But this is none of this is specific to Wagtail. So in my local environment here, I would create environment and install Wagtail. I've done those first three steps so you don't have to watch the that scrolling past, but I'm now going to Start a new Wagtus site and go into that directory. And the next thing I'm going to do is install a package that actually I wrote about a month or so ago called Wagtail Fake News, which is um uh a way to create a lot of dummy content on your site. I did it really for testing, particularly for testing um uh volume so to see how well it's kind of scales when you have more volume. So we add that and put it in our requirements.

txt. And then we're going to add it to our installed apps in settings. So of course there are more elegant ways of doing this, but I'm just going to push that line to the bottom of my settings file. And um and the next step is to run and migrate. So this is common again. This is a standard thing that as a Django developer you'll be used to. So creating all the tables that Django needs and then that Wagtail needs and that Wagtail fake news needs. And that's all done. And now we have this new management command. I didn't copy that correctly. Make fake items. So this is the one that comes from WagTale Fake News, and this is going to create 50 fake news pages and 50 images, and it's going to associate images with each one.

That happens pretty quickly, seven seconds. And one last little step. Magtower has a default home page, which we don't want, so we're just going to overwrite that template quickly with that line and then we're going to run the server. So most of this now it's hopefully feeling quite familiar for uh for Django sites. So now I'm running my server and let's check see how that's worked. So in a new browser tab, I am Checking here that this news is linked and I've just gonna check something actually in the Slack to make sure that my I'm not seeing anything this

like. I'm just hoping my screen share is working. I guess otherwise someone would have told me, but I'm just not seeing the screen share indication over my screen. Right, so now we are seeing Can someone just like give me a positive sign in the general channel on Slack just to say that you can see my screen? Working fine. Thank you, Darren Jones. Uh a little moment of panic there. Okay, so we can see now that this uh my site is running. Um And that's great. So that's like, I don't know, how far are we in now? Five minutes and we've got our local Django site running. That's basically, you know, just credit to Django for making that site really easy. The next step now, now we're getting on to deploying.

So now we are going to install Docu on a Linux server. And I would generally use one apps a service like DigitalOcean or Volta. And so I would create something like let me see if that's going to log me in. Create a new server on Volta or DigitalOcean, one of these services. Um so go here and cloud compute and then I can choose my location. I usually like I always feel like servers in Paris a little bit more stylish than anywhere else. So and then get the latest Debian. The small one is fine as long as you've got about a gigabyte of memory and uh add your SSH key to it and probably give it a name and hit deploy now.

I'm not gonna do deploy now because I've already done one and it just takes a couple of seconds, but uh you might have done something a bit like that as I was talking. So here is my the server I've created and then I'm going to log into that one. So here in this tab with a black background I've logged into this that that server And the next thing to do would be to install Docu, which is these two lines. So it gets the bootstrap. sh and then runs it. I've done that already, it takes about two minutes, so I didn't you didn't need to see that running up my screen Once that's done, then you go to your server's IP address in a browser. So if I get the IP address here And make a new tab and go to the browser.

That loads this initial Doku page. So Doku doesn't have a web interface generally, apart from this initial screen, where you just check, you can provide your SSH key. and give it a host name. I'm just going to use the same IP address and tell it to start. That's done. And then it's just going to redirect me away to the Docky Docs which I don't need. So I'm going to close that tab. Okay, so Doku 's installed. The next thing I need to do is create an app on the server. So if you use Heroku, this will seem pretty familiar. This um uh a lot of the commands are quite similar to uh you could say they've been inspired by Heroku. So we've created the app and now we're going to install the Docu. Postgres plugin. So Docker has the concept of plugins and um

most of them, I think they probably all are themselves Docker containers. Again, you don't really need to know that, but they're ways of providing different services on your Docu instance. So the Postgres one is probably the one of the most popular ones and uh that means that we now have Postgres eleven point six running on our Docu instance Once that has finished, which shouldn't take much longer, we can create our first database. So we now get this Postgres create command that didn't exist before because we've installed the plugin So that's going to create a database called Demo DB , which will take a second. And then we're going to link that demoDB to our new app. So we created our app called Demo App

and we're going to link the database to it. And again, this is a bit like how uh how Heroku works, uh where you make these variables uh like the database URL available to your apps. Okay, so that's all done. Something worth knowing about Docu, and in fact lots of these kind of mod services, and Heroku is similar, anything where you're using Docker generally. is that they are ephemeral. That means that every time you deploy them, they're recreated from scratch. Which feels a bit weird when you're used to it, but it turns out to be a good thing and it helps you design your your application in in I think in a correct way. So uh follows this this set of rules, principles called the 12-factor design, which you may have come across. I I I recommend looking about.

But it means that uh if you for example, if your app has user content, you are you have a content management system and you're uploading images, then um th they're going to be lost every time you redeploy, which is no good. So we need to have some sort of persistent storage. So one approach might be use something like S3. But um Docu has uh an option for this as well. So you 'll create a directory and then we're gonna mount it. inside docu giving it the name and we're again we're going to make that available as a variable called media root which our application is going to need to listen to on Right, our application that we've built so far, so this is going back into the white tab where we built our application.

Um, that uh because it's uh like a Django app doesn't use Postgres by default So we need to make sure the Postgres is installed and that it's in our requirements. txt. And then we need to tell our app using settings to use these variables that are now available. So we now have this database URL and media root available So I'm gonna add those, just plonk them at the bottom of the file, which isn't really best practice, but it's fine for now. Okay, so we now have our app. And we have installed Doku. Now we're going to do our first deploy. So deployments in Doku

are all about Git. So we're going to create our Git repository, add everything to it, first commit, and then we're going to make a new remote. And so I need to put in the IP address of our server. which should be somewhere in my clipboard history. There we go. And adding a new remote called Doku. And then the final step is to git push to Doku. This is the bit that's going to take a minute or two. And while that's happening, I can just kind of recap a bit on what we've got to so far. And I've got actually some slides to kind of help us think about this. So we made a simple Django site We installed Docu on a VM using those two lines. We enabled Postgres in that Doku

instance. We added persistent storage. And we just ran git push to deploy. So the um the the making into simple Django site is what you do normally. Installing Docu and the Postgres things are one time only. Um after that and and and the persistent storage is probably for each app and then the git push to deploy is what you do for for all your apps afterwards. So we can see what's happening now. We pushed that up to Docu, and Doku immediately saw the Docker file that existed in our in our app and so understood what to do with it. So it um It's following the instructions in the Docker file. It's currently now installing all the contents of requirements. txt. So all those things are going to be available to our app.

It will link up the Postgres database, hopefully, and make that available to our app. And once it's happy that everything's there, it will deploy it. And one of the things that Docu does is check that everything's working before it makes the before it di deploys it live. which is really a kind of valuable step, but it does slightly slow things down. And you can do it in a more intelligent way if you like by creating your own custom checks. This isn't the most interesting part while it's just uh waiting to see what the installation part is. Um and then but now it's doing the

deployment. So uh these steps will look familiar to anyone who's done uh Docker before. And maybe this is just while this takes another 30 seconds, I could just start talking a bit about Docker, which is we don't have time in this talk to kind of go into information on to kind of full background on Docker. But Docker is a way of packaging up your your applications in a that's a in a like a very lightweight virtual machine. And it's increasingly becoming a standard for deploying your applications across lots of different systems. And it also can be useful for local development. So when you create a Wagtail site, it creates a simple Docker file for you. But there are other ways of doing that.

So if you have a Django app, if you follow, for example, one of the cookie cutter steps. then um uh the the cookie cutter the Django cookie cutter then you'll also get a Docker file. So that's now finished and it's telling us that our application is deployed on a live server and we hold our breaths type it in and that that that it's live so we've built our site we have uh installed docker do we have uh deployed a site to do and it's now live however there are no news items on it because we haven't created any content on that one yet. So we're going to need to run a couple of tasks. And this is back on the server. We're going to run a management task. And you can see the syntax for this is Pretty much like the normal Manager. py that you you start it with docu run and then you specify the name of the app.

It takes a few seconds longer than than a normal management task because it has to spin up a container quickly in order to make that available. And then similarly we're going to create some fake items just like we did before but this time we're running it on docu so again a few seconds for it to spend the container and then it starts creating all those fake items. So now if we look again In on that one, but on our live server and go to fake index, we see all our site and all the pages that have been created, and just to confirm that the admin user worked I can hopefully log in now using the details I just created

and browse through Wagtail and let's edit a page. Maybe this first one. box last resource check this is working I guess mark we'll change that image to audience democratic modern and uh bolden some of that and preview it and that all looks good. I must say I find these uh I find these randomly generated content really intriguing and uh often makes me think Sometimes it does the kind of sound really weirdly profound. Never direction what? Price might sell. Rate really positive.

Sometimes I feel like uh who needs artificial intelligence when uh randomness is this good? Okay, so to recap on where we've got to. We have Git push to deploy. We now have a live site production ready running on our $5 instance. This could handle probably five or six reasonable sized Django apps and for five dollars a month and you get pushed to deploy and it's a you know pretty pretty quick process so far. Okay, next up is static site generation. There are quite a few ways of generating a static version of your site. The one I'm going to use is based on Django Bakery, which is open sourced by the LA Times In fact, we're going to use Wagtail Bakery, which is just a sort of bit of a wrapper around Django Bakery that does the more convenient things for Wagtail sites.

So install that, add it to our requirements. txt, and then make a few changes to our settings again. Demo settings base. py, add it to the bottom. So we're adding bakery and wagtail bakery. We're telling it to build out to temp slash dist, and we're saying that we want all our published pages. Those are going to be the pages that we want to be live. And this gives us a new management command, build. And build will take that view and create published versions of all those pages Usually takes about five or six to this like that gives you nine seconds. And so if I now

uh check in my Where do we say temp slash dist. There we can see we have it's created some pages, and we can see for each of those pages it's created a directory and uh a page inside it. So it's got we now have static content for all those pages, which is great, but it's just running on our laptop, still on our development environment. So now we need to publish that. For that I'm going to use Netlify. There are lots of other ways we could do this, but Netlify is uh Netlify has a lot of benefits, so it's an excellent service. And you you can install the command line version like this. I've done that already But uh you can do that, it'll just take you a couple of minutes. And um

I now I'm gonna s go into that directory. So this is the one with all the statically generated content. And I'm going to run Netlify init. So the first time you do this, you'll probably need to authenticate, and you can do that against your GitHub credentials. I've done that already, so I can just go ahead like this. choose my team uh hello jumper con you hope no one has used that before yes so that inside has now been created um but now I need to deploy So the first one was just creating the site. I'm going to do from this current directory, it's going to check and see if there are any files exist up there already and they don't because it's the first time we've done it. And then it's going to push all those files up.

And once that's done, it is hopefully going to deploy them. And this is normally super quick. Waiting for deploy to go live. Just while that's going, I'll show you what the UI looks like. in um Netlify on the web UI. So here I'm now seeing the manual deploy. So that's now being created. The site's not being deployed as we know. This is sort of hanging at that last minute. So I'm going to do the classic, turn it off and turn it on again, stop it and try start again. This I'm going to do prod, which is going to make it go live straight away. That's a usually a separate step, but uh This time I'm just going to do it in one gray. This time we're getting some nice emojis that we didn't get last time, which is a bit odd.

And this step, uh, once it goes live, should give us a static version of the site hosted on Netlify, which has its own collection of CDNs and works across clouds and has a lot of other interesting features, particularly around things like serverless functions and authentication and so on, which we're not going to go into in this talk I'm hoping that's going to finish in a minute and I can do a big kind of uh celebratory ta-da. If it doesn't, I'm gonna have to wait and go back to a previous one to show that I that it is working so this is one I did just slightly before and uh click on it here and you can see now this is running on netlify. app And uh all these pages are available.

So the content of our Django site, the output of our Django site is now running on Netlify. You can see on Netlify. app, Netlify knows nothing about Django or Python or databases is hosting the static pages, but it's doing them in a very efficient way and also interestingly a very cheap way. That one hasn't gone live, so it's lucky I had that back up for me. Hopefully it will do before the end of my talk. So you'll believe me. Okay, so where have we got to? Live demo two. We generated our static pages. We used um we used Django Bakery, but there's a few other approaches we could have used too. We installed Netlify. And we deployed to Netlify. And great, next one, then the final one of this demo is using Google Cloud. So like I said before, this is a new product, new ish product, recently come out of beta

from Google and called Cloud Run. And you have to start by installing the Google Cloud SDK , which I have already, but you could do, doesn't take too long. Cancel that one now so we can go through this. And then you have to create so Google Cloud has this concept of projects. So you can kind of group all your stuff together in different projects. So This is what it looks like in the console. Here I've created a project called DeployTalk, which has that ID. So I I need to tell Google Cloud The name of my talk uh project. Um And then I

you have to enable the minimum APIs, which I've done already. That says which what bits of Google Cloud we want to use. You have to set your project ID and region. That's going to be useful for some uh uh shell scripts later. And we're gonna it's this approach is going to be a little bit similar to how to the Docu one where we get a um a Docker a Docker image and we we submit it. So now from our app This is kind of there's two key steps with this. The first most of it is kind of boilerplate really. This is the first key step where we're going to take our current app. Actually, we're not going to do this now because I'm in the wrong place. I need to go back to my demo app. And we're going to take this current directory and submit it as a

submit a build for it. So it's going to create a Docker image based on the contents of my directory, which apparently I've got about 19 gigabytes in total, and it's gonna push all that up to Google Cloud's image storage. and make it available for other services to use. We had this first step called use Canico, which is quite a recent feature. Uh it's a sort of strange name, but it uh basically enables caching of your build step. So you can see in some of the lines coming out here that This build is quicker than it would have been otherwise because it's it I've done it once before and everything that hasn't changed in my app can just be reused from the cached layers

So having done all that and runManage. py, it's creating that , adding all this to the cache, and it's going to put the the whole image, make the whole image available for us to deploy. There we go, done took 48 seconds, so it's created this image and the final step is to deploy that image. So I'm gonna paste this in and then and run it and then I'm going to tell you what it means. So we are saying gcloud is the general term. Run is because this is about um Google Cloud Run, the action is deploy, and demo app cloud run is the name of the service that we want to deploy. And it's going to deploy a service from this image, which is the one that we just uploaded before. So we created an image for demo app

image. and we're now deploying it. And this should be fairly quick because Google Run service just needs to fetch that image from locally from elsewhere in the Google Cloud. and create a revision for it. And that is now done. So and it a bit like Netflix, it signs you a um uh a new URL. And I paste that in and hit enter. And then one thing you'll notice here is that this is slow. And this is there's the this is because of um one of the issues with serverlessness generally is this this so-called cold starts. And cold starts are because uh in this case when we the first time we request the app it has to uh

has to s start that container. And this container is a rather inefficient container because the way that the Docker file works in that you get out of the box in Wagtail, it runs a migration to make sure that your database is up to date. Which you don't re you really don't want to run all your migrations the first time anyone requests your site. Of course it's uh Then and there's a lot you can do to optimize that. So you uh that's not kind of best practice for production Docker files, and actually you should move that step out of your uh Docker file, which um and there's lots of good documentation on on how to do that. Once you once it is up and running, then uh Google Cloud does a really good job of preventing cold starts afterwards. But for that first one, uh

you you want to find ways of reducing that cold start. So we now have our set that same app running in, I think, three different places. We have it on Docu, we have it on Netlify, and we have it on Cloud Run. Let me just quickly show you how that looks on Cloud Run in their UI. This is a couple of interesting things about uh the setup here. Also, I find that this UI quite slow sometimes. Um It'd be a shame if that doesn't come up quickly while that's going. I'm just gonna recap on what we've done. We installed the GCloud CLI that made this possible. We built an image. And we pushed it up to Google Cloud Storage and then we deployed it to Google Cloud Run. This is still loading.

Sometimes you just give it a little tweak The thing I wanted to show you here is how you can configure your the scaling of your instances, which I think is the really interesting thing about uh Google Cloud Run is the way that it scales not just upwards so uh you can scale to uh a thousand by default it's uh a thousand instances of your container each one of which but by default can handle 80 concurrent requests, but also it scales to zero. So here's the revision I just created. For my next one, I could set some other variables like How much memory? So this is currently running on a tiny amount of memory, so I'd probably want more like half a gig.

And I could change the number of CPUs. And I say how many requests I have. But this is, I think, the really interesting part. So this will currently scale up to a thousand instances if you need them without you having to do anything. Just as your traffic increases, it will create new instances when it needs them. But also it will scale down to zero. I gave a talk about serverlessness in in uh Florence, I think three years ago, is it two or three years ago? And uh I talked about this idea of turning the lights off when you leave the room, which I think is one of the most interesting aspects of of serverlessness. that um when people aren't on your site, it's kind of not financial or ecological sense to keep that server running. And that's that's what I like about this sort of serverless approach is that You are only paying when people are hitting your site in this.

And this the servers are even only alive when you're hitting them. So I think scaling to zero is a kind of really interesting component of all this. All right, I want to talk very in the last remaining minutes very briefly about the pros and cons of these different approaches. So Doku, I think this is the easiest way of getting your site into production. So we did it, I think, in under 10 minutes. This is a standard Django site you can go live in 10 minutes if you have a Linux box available, a $5 Linux box. It also encourages good coding practices. So this that 12-factor design I talked about before and it's super cheap. You can run A few Django sites are for five dollars a month and these are like you know that you can do this in production and we've we've done this successfully at Torchbox. On the other hand, Docu doesn't have horizontal scaling, so If you need to handle, if you need more instances, you can't.

You can't have lots of Docu instances next to each other. And in a similar way, there's no redundancy. If the hard disk dies on your machine, then all your sites will go. The infrastructure is on you. Static site generation, the pros are pretty convincing. It's the cheapest way to host your site. And you cannot hack a static site. So it's you know it's it's it's pretty compelling for all these reasons On the other hand, you get this build time delay. So if you have a really big site and you have to generate the static version each time, then there's a there can be a pause between making the content and having it live. There are more moving parts because you need a Django server and a static site server like Netlify. And it may not suit your app. So it works well for kind of static, like a content manager type sites.

But if you need personalization or you have logins or lots of forms, then it's probably not going to be the approach for you. On the Google Run side, it scales forever, not quite forever, but if you have if you need more than a thousand instances handling 80 concurrent users each, then you've got bigger problems Scales to zero, as I mentioned before, I think that's really important. And it's the future, that's a bit glib, but I do think this model is the is is in is the way that uh that app hosting is gonna go On the other hand, you have to deal with cold starts. It's still complicated. So I showed you the easy parts, but once you get into connecting databases and persistent storage, there's some more complicated instructions, which I've linked to at the bottom of my notes And finally, databases are not serverless. So you there are some sort of serverless database products and

Aurora on AWS serverless is one, but it's It's not it's I think it's just really hard for a relational database to be available as a serverless. So it's um uh it's it's currently still feels a bit hybrid, but I do think that this is the way that increasingly apps will be hosted. Okay, if you need to deploy as quickly as possible, then I think Docu is a good approach. If your site is mainly about information, then static site generation gives you that speed and robustness and it's super cheap. If you have to scale, then I think a cloud run type approach is the way to go. That is it. I'm sorry for going on so long and I'm going to stop now, but I'm really happy to answer questions either on Slack or uh in the uh in the q a section afterwards. Thank you all very much.