Deploy Django: GitOps & Kubernetes Made Easy - Calvin Hendryx-Parker

Speaker 1: Thank you all to the Pi Ohio organizers for making such an awesome conference that I was at I was at the very first one when Catherine did it in Columbus at the library there. And so I've been a fan ever since. Now I want to talk to you all about something even more near and dear to my heart, which is deployment. Who here loves deploying software? It's like your favorite part of the job. Oh oh I I know, I I got one, two, okay. It's not always a thing we love doing, but it's a thing that has to happen for us to deliver what we we actually produce as value, which is our code. So specifically I'm going to talk about deploying Django, because that's a lot of work we do. And so it might serve as a good blueprint for some techniques and tricks and and

Speaker 1: ways that you can actually deploy your own software using Kubernetes and GitOps. And I even put the word easy in there. That's a word I've been trying to strike from my vocabulary normally. That might simple and things like that. But I I do feel like what we've put together tries to make this as push button as possible. All the slides are actually available on GitHub. There's a GitHub link, so if you just go to the sixfeetup org on GitHub, it should be like one of the first repositories. Or that QR code takes you to exactly that link. And it's not nothing nefarious, I swear. It's a it's a just a link to that uh page right there. And you can learn more about uh this you can follow along, have the slides. And it all the things I'm going to show uh

Speaker 1: because I one of the things I love doing as part of giving talks is demos. If I only get to give a demo, there's no point in me being up here. So we're gonna kick off actually with the demo because part of my demo takes about five minutes. to 10 minutes in the background to run. And I don't want to like make that, I want to make sure we get through all that. So what we're going to do first actually is kick off with our demo. And so whoops. What we're gonna do is we're gonna deploy some software today. We're gonna go from nothing on the file system to a full uh GitOps workflow deploym. to AWS Cloud in Kubernetes uh set setup here, all the way from we're gonna actually create the repository here. Oh create We're going to call it six feet up. And I've deployed probably 40 Kubernetes

Speaker 1: clusters over the last two days, so you don't have to because I'm testing a lot. Um all right. So this is obviously iteration number 17. Oh, I forgot the public tag. Public. And if you wanted to follow along, you can also see that that Repository. But we're going to use a tool that we developed at 6P 's open source called SCAF, and I'll tell talk more about that as well. And we're just going to make a new project and we're going to use our full stack template. I'm going to name it simply that. And then we use this copy under the covers. And though this, if I last year at this time, I gave a talk about Scaff in the developer experience. This is the other half of that talk, which is the deployment experience. We really have focused on making an integrated developer experience that is easy and fun to use and so that folks can

Speaker 1: have a sane environment for working together. So right now I'm just filling out some questions. This is like where I want my site to actually be deployed to. So when we're done the goal would be that we can actually Go to a URL that has certificates and the whole bit all set up and it should work. And we're going to deploy it into US East. Um I actually have the uh uh ID of my account right there. We're gonna put an XJS front end on it. We have a choice between K3S and Talos Linux, which are two different uh distributions of Kubernetes available to us. We're not gonna use not gonna use One last bit I need to change is make sure that this goes into the six feet up org. And bang. So behind the scenes, this has actually started a brand new fresh project on our

Speaker 1: file system. I had nothing on my disk at all. Uh and then it's going to put all the the bits we need in the right places and give us that developer experience. What I'm going to do is stall a little bit right now while it does this because Because I want to actually kick off this the deploy of this into Amazon, which means I have to have code in a GitHub repository and I have to have a CI pipeline running in there to make this all kind of magically them together. There we go. So right now it's actually going and pinning all of our dependencies. One thing we want to make sure as developers is we're always pinning all of our dependencies. And that whether they're Python dependencies or node dependencies or later on I'll show you even the Nix dependencies we use for command line tools as part of our developer workflow. We actually pin those as well to make sure that the experience today is the same experience I would get six months from now.

Speaker 1: or 18 months from now when I come back to this project after a hiatus and I want it all to still work and actually be productive and develop, that's that's one of the problems we really tried to solve with the SCAF package. that we put out there as a as a way to get started with your projects. While that's actually running, I will come back over here real quick. So the other question you may be asking is like what what's what even is GitOps. I mean it's it's an operating model for you to deploy software in a way that is repeatable, reproducible. It provides you with that single source of truth. All of the the bits that get deployed only happen because They went into your Git repository in some way or fashion. Like there's no click ops or command line cowboy going on here. Uh you

Speaker 1: everything that gets deployed comes out of a single source of truth. It's all declarative, uh which means that It's uh you describe what you want to have, not how you want to do it. So you basically shape your world, tell it what you want it to look like, and then it goes off and does this for you. Of of being in the GitOps model. That automated reconciliation means that the system is continuously ensuring that the state of your deployment matches what it's supposed to. So if something in the state changes, the system uh will deploy those changes. If something in the deployed state changes in an adverse function way because someone maybe manually went in and did something, it can actually go in and actively fix that for you. It uses poll-based deployments. You're no longer from your own laptop running kubectl apply from your own machines because that's obviously dangerous and untrackable.

Speaker 1: and you don't know who did what, uh this sits and watches your repository or looks for polls coming in from uh the the CI pipeline and deploys it for you on your behalf. half so you now know it's been deployed correctly. Another bonus bit is you really get a good documentation of your infrastructure because you've defined what you want. It it's all in uh you know terraform code and Kubernetes. manifest, you know what the code for the infrastructure looks like. So it's basically a documentation as code for your infrastructure and your application itself. All right, let's uh look real quick over here. Oh yeah. So we now have a a project on our file system, PyOhio 17. Heading back into demo later Here. We use DureInv, and this is kind of part of making sure all of everyone's environments are all line up and have the right things.

Speaker 1: But one of the pieces we're using right here is a system call called Nix. If you've not checked out Nix, that's a whole talk on its own. But Nix makes sure that I have the same versions of the binaries that Frank has and that Maxime has, and then we and we're not Like running into weird edge cases where you're using a version of said that is from someplace else that doesn't have the same command line flags as my version of core utils said that was on my Linux box. And what that does next though is it does change our file system and puts In a lock file for that. So we're going to check in that file real quick. Oops. And now everyone who to polls this project will get exactly the same versions I have on my my machine right now of those things. And then you can move them forward just like you would with any other part of your dependency chain

Speaker 1: process. So we now got uh our thing here I'm going to actually do We're gonna check out a develop branch because we're releasing the sandbox. I don't want to release the production. I'm gonna well I've got kind of my GitHub's GitOps pipeline set up so that anything going on the develop branch, like if you did a pull request and merge into develop, it should release automatically into that sandbox. So we're in sandbox, right? I'm going to do a git push of this to develop. And I use yep another cool trick there, uh one password has an SSH agent. So all of my SSH keys are actually in one password and not on my file system. So no private keys, no private secrets on my machine anywhere. Another tenant we like try to really strictly follow as we do the scaff stuff too.

Speaker 1: You'll notice I don't have any secrets anywhere on my machine or in the GitHub other than if they've been encrypted. We use sealed secrets for that. Okay, so that pushed over to um uh to GitHub. So actually we can go look at that right now. Whoops. Oh this this monitor resolution is gonna mess with me. Here we go. We should see if Hiohio Seventeen. Here we go. And that kicked off uh GitHub Actions. It's actually in the background building right now. Um some

Speaker 1: So the next step is actually going to be to go in here and we're going to deploy. We're using task files to deploy, so we're going to do deploy. Oh wait, once, yeah. One one more thing. I've done a couple demos. I want to make sure my kubectl Kubeconf is not set. Perfect, it's not. And I want to make sure that's set. Uh there's a very dangerous mode in here I just enabled. Don't do this at home, but for the sake of the demo, it's it's okay. gonna be it's all gonna be fine. Okay now we're gonna deploy our sandbox. Deploy sandbox. This is actually going to set up a security group for the IP address of where I'm sitting right now. So only I can access this well now. And you all on the public Wi-Fi who are in this building can access this.

Speaker 1: And it's going to take Terraform and start building out the infrastructure in AWS for all the infrastructure pieces we need to make this all all the magic happen. While that's going, we'll go back into the uh magic land over here and continue the uh presentation. Okay. So why GitOps then for Django itself? Because you are deploying an application, you may have the needs where you want to be able to run rollback. If a if an error or a bug creeps into production, you'd want to be able to have that quick feedback and rollback. Tools like GitOps like Argo CD, which I'm going to show, or you know, Flux C D. CD and other CDI CD type tools, give this quick rollback feature. You basically can go typically through a web UI and able to push a button and say rollback.

Speaker 1: Or you can roll forward, which is kind of my preferred stances, fix the bug quick. quickly, small little you know, atomic changes that are rolled forward are actually a better approach because now you can get back on track and and continue to be uh releasing new features and have and maintain That velocity. One other thing you get with GitOffs is going to be environment drift protection. I mentioned this a little bit, but dev staging and production as you create new settings or new environment variables, things like that. Those are the typical things that developers forget to put onto the next environment. You're developing a new feature, you've implemented some new environment variable as part of that feature, uh you forgot to put it on sandbox. And when we need to go to deploy to production, you forgot to put it in production. And now things broke because you forgot to have that environment variable there. Well these these tools actually help you set up kind of base

Speaker 1: within the layers that can go on top of it so that I define a the environment variable for everybody up front and then I customize it literally using a tool called customize for the various specific environments. So you can now have uh least privilege and and um you can have environments that are set up specifically for production so developers don't have have production access, but they do have access to sandbox. These are the kind of things that are enabled. That's security. No CI CD needs cluster credentials. The way we're using our CI CD is we have our our tool pull our GitHub for um new changes instead of having more credentials kind of out in the wild in various places. The CI pipeline we've deployed has no credentials in it. it at all. It builds images and because of the OIDC uh O uh

Speaker 1: the uh OAuth connection with like GitHub and Amazon we can push into ECR without with roles instead of having credentials. It's just much cleaner and a lot less to worry about. And then compliance. With GitOps you have a full audit trail of who has done what, when, and where. Because it's all in Git. Now you know who pushed a change, you know what when it built, you know, what image had that change in it, and when it went out the door, and when it got deployed into a specific environment more importantly too. You also can control I mentioned this with the environments. You also get to control who acts who has access to the kube config. So if you only want again developers to have access to the sandbox environment, they don't have there's a different kube config than there is for production environment. So So now you can have people who are more in the operation of responsibility and accountability knowing that their stuff can't be messed with by kind of rogue developers, you know, meddling when they shouldn't be.

Speaker 1: I know they're trying to help, but sometimes uh their help may not be wanted. And then but people ask me this often is why Kubernetes for Django? My number one reason I I like about Kubernetes is the control plane. It gives me that ability to Orchestrate it and automate it with an API. There's an endpoint that gives me that control from other systems that now can talk to it. These other reasons are all awesome as well. Obviously, you could do get the whole orchestration and scaling bit. Uh the zero downtime deployments is a nice bonus. Uh you it will scale down or it'll it'll scale up new pods as it does releases with your code on it, keeping the old ones in place until it's moved all the traffic over to the new ones and then it goes ahead and shuts down the old ones. It's really nice that that's just an out of the box thing you get. You get the self-healing.

Speaker 1: If containers fail for some reason or if a node fails for some reason, uh Kubernetes is watching for all those things and bringing them back up for you on on demand. Service discovery is also nice. You can just point at the database, like the Postgres, the Redis, the celery or whatever the things may be, and those names are automatically hooked up inside the cluster and discoverable. Secrets management, out of the box you get sealed secrets, which means you can check secrets Secrets into GitHub and they're tied to a specific environment and they're encrypted at rest and in the repository. And they only apply to, say, the sandbox environment. The sealed secrets that work on sandbox don't work on production. So again you get nice kind of secrets management. Uh there are better ways to do that, but that's a good default, like a sane starting place for that.

Speaker 1: I already mentioned control plane. I love the deploy story. You know, it matches our developer. environments. So not only did I just spin up uh a Kubernetes cluster in Amazon, I've also spun up a Kubernetes cluster on my local machine. So I've actually got two clusters I've spun up here uh in real time with you all. Actually let's check in on uh How's it going over here? Oh yeah, this is the long part. Spinning up cloud front distributions. Uh I've got a whole architecture diagram and talk that goes over everything that's in there. This will take another two minutes, so we'll be back in two minutes for that. Uh all right. And then another reason for Kubernetes is you now have a cloud agnostic way of deploying uh whether you want to This is a demo done on Amazon AWS, but this will work on Azure. It can work in any managed Kubernetes clusters. It can work on your bare metal.

Speaker 1: So if you want to be in Hessner or some other like really inexpensive Provider, this gives you the ability to move things around and everything still works the same way generally. You've got little add-ons around the edges, like for example, we're using Amazon. ECR for our container registry, but you could just as easily be using Docker Hub. You could just as easily be using Cloudflare instead of CloudFront on AWS. So there's there's some pieces you can swap. pop in and out. But it's nice is that you've got a base where you can now do that. You can actually plug and play pieces from one to the other and have Terraform and Kubernetes orchestrate it all for you. And then the magic behind the GitHub GitOps part of this actually is Argo CD for us. There's a couple great open source GitOps platforms. We chose Argo. It has a nice, really slick UI, which makes it kind of attractive for people who are not in

Speaker 1: in the GitOps or DevOps world as much. But it is Kubernetes native continuous delivery. The Argo CD runs in the cluster for us in this case. So we're running it in Sand We'll be running it in staging and running it in production. And each of those are responsible for their various deployment environments. So we're deploying Kubernetes or Argo CD into Kubernetes. It's pulling our GitHub repository. It has a webhook mode but we're using the polling uh option for that. And then it synchronizes automatically on demand. When the manifests update and there's new revisions that are tagged for the sandbox environment, it knows it's got to go into action and deploy. those pods onto your uh system. It's basically a powerful wrapper around the kubectl command. Anything you do with kubectl on your local machine, Argo can now do for you in an automated automated in

Speaker 1: controlled fashion and it's all again all auditable. And it does support other extensions like Helm, Customize, and Plain YAML. Actually we're using all of those features today in this demo. the scenes. And I mentioned this a couple times now. This is all enabled through our Scaff tool. We basically wanted to have day zero GitOps. We didn't want to bolt GitOps onto it afterward. We don't want to want to bolt deployment in as an afterthought. We wanted to start with the end in mind so that everything should just flow nicely. Like day one of of the project experience should have something live on the internet that you can click and like navigate to and see if it's like a web application for example. Uh we it is battle tested. This is something I started back probably in 2008 as a Django template, uh pre Kubernetes, pre-all these other things.

Speaker 1: I didn't start doing more container type work until about 2021, 2020 timeframe. And then I really started getting more invested in this ecosystem of how do we how do we simplify and take the hands off the keyboard. and turn this more into a a developer experience where I get to do just the pieces I want to do. That that cognitive load reduction of focusing on building software and not worrying about deployment, but having all the deployment follow best practices. What I really, really wanted. Another side effect of this is choosing the right kind of tools. We chose Talos Linux for our production level Kubernetes distribution because of the immutability of it. When you deploy a Talos Linux uh virtual machine on Amazon or any other platform, it's also cross-platform, it doesn't even have an SSH shell.

Speaker 1: You can't shell it. into it and fiddle. It has a tallow CTL for doing updates and things, and you've got kubectl for doing things with your cluster and your application. But it is a secure platform that is immutable by default. You can't change files on the containers again following that stateless informal um you know way of doing things. You know that this this is basically the magic. This is best practices out of the box you know for that you know solving that kind of blank page problem, how do I get started? This is a great way to get started. Alright, let's see where things are right now. Why keep wanting to take me over there? Ah, sweet. Okay. So uh CloudFront's done. We're getting which means we're getting close to the Kubernetes pit 's all starting.

Speaker 1: All that's happened so far is infrastructure via Uh Terraform using OpenTOFU is being created over in Amazon. There you can see the secrets that we output from the Terraform run that then will be combined into a sealed secrets. Here in a moment. It's installing the an ECR credential provider so that it can get pull images from ECR uh very easily. And right now it's installing K3S. So I mentioned Talos Linux, that's one of our preferred production Linux additions. Distributions for Kubernetes. For sandbox instances, for local running stuff, uh you probably want something a little more lightweight. And K3S is a very lightweight single node distribution of Kubernetes that is uh perfect for a use case like this where I've just got one single EC2 instance running and that's what

Speaker 1: is actually happening right here. Now it's now the actual Helm chart is being used for Argo C D, getting that installed, and then should be up. This will take a a hot minute uh before it'll be ready to go, but I'll come right back to that. So I think these are definitely easier to describe when you've got a picture in front of you. This is generally the overview of the the experience. I've been trying to lead up to is that the developers themselves code on their local machines, they're using Kubernetes locally. We use Kind as a Kubernetes distribution. on our local machine ends up ultimately in a pull request. And that pull request could be reviewed. And then once it's merged into the develop framework or develop um uh branch, you now have something that can be released. So you can see here the GitHub uh flow on the top, which is like basically a GitHub repository that runs the GitHub actions for CI.

Speaker 1: Those are responsible for building images that go into AWS ECR, the Argo uh controller in the cluster is now watching for those um images to be built and when they are it goes into action and deploys This is probably like a month of DevOpsy things rolled into it happens in about five minutes of type uh s time saving. So it's it's legitimately that that nice uh of a system for you to use. All right, let's I don't want to keep going to that. Okay. Sealed secrets are coming up. I'm gonna go into this directory here Now what's nice is you if you're a developer on your local machine and you want to interact with the sandbox.

Speaker 1: Just go get our kubeconfig. You just have a kubeconfig for your sandbox in the repository ready to go for developers. Uh I can now look for things. like certificates for example. So the system hasn't gotten any uh TLS certificates from certificate manager yet so I can't actually go to the the web UI for Argo, but it will be there soon. So as soon as those show up, we'll be able to go to Argo and then kind of start showing you a little bit of tour of Argo CD itself. Hopefully if it happens in the next five minutes. or so here. Which you should. Um keep fingers crossed. This was running a little slow for me uh this morning. So for example, if you're using kubectl, I I again I am not a Kubernetes expert. Uh there are people in the room who know way more about Kubernetes than I actually do. But for example, we can look at the things that have been deployed currently.

Speaker 1: There we go. All the Argo CD stuff's up. The sealed secrets controller is not up yet, so you can see the Argo stuff at the top. Uh it's it's gone off and you know put Together all the magic, the stuff I don't know how to necessarily do myself, but it does for me, which is really nice. We should see sealed secrets coming in here real soon. And once we do, uh then I'll be able to actually get to what we were talking about earlier. Let me see if the certificates. Nope, no certificates yet. That'll take a hot second. All right, we'll come right back to that. Okay. Now I mentioned that uh again part of our workflow involves Nix. I wanted to have a little sub -thing in here. The issue we had two days ago literally was all the developers who were working on this ta uh scaffolding project were all using Linux laptops. Everything worked great on their laptops.

Speaker 1: I am on my Mac OS Apple Silicon laptop. I went to pull it out and nothing worked. Uh it would bomb out like every step of the way. Ultimately what it was was a few command lines. tools we're leveraging in some of the scripts had different flags from Mac OS to GNU Linux. So using Nix allows me to get the core utilities versions of those same exact utilities over on my side so I don't run into these kinds of stumbling. blocks. If you haven't checked out Nix, I highly recommend it just from a standpoint of the works on my machine thing is really painful. You don't want to spend days spinning your wheels just because you had the wrong version. Now you you would think the base 64 command line tool uh would not bomb out, but there's literally a flag on Linux for wrapping or not.

Speaker 1: And on Mac OS, the default is to grab. And you don't need the flag. And there is no flag. It doesn't exist. Helping another developer shouldn't be hard. And that's What we're trying to solve for here. So if you are in a team and you're collaborating with other people, this is a really nice feature to have for you. Some new things we also added over the last two days. For a while we were pretty coupled into one package. We were storing the deploy keys there and a lot of our secrets there. Not a bad thing, but from a simplicity and eliminating some extra tools around the edges, we decoupled from one password. Sealed secrets still happens. We're also using um SSM parameter store in Amazon to store things like the deploy key and then we put the deploy key into GitHub and then now uh Argo will have access to the deploy key on demand via that secrets management in Amazon.

Speaker 1: So again the cluster itself doesn't maintain the secrets, an outside secrets management tool is doing that for us. I mentioned the dangerous mode. As long as you don't know what that magic environment is , variable is you don't have to worry about it. But if you did want to turn it on, you could. We did automate sealed secrets creation, and that's actually what's happening right now on on here. Back up to this. It's still waiting for it. So it's actually this part was pretty manual before where we'd like pull the secrets, create the secrets, push the secrets. We managed to get that all pretty pretty automated. Uh it does take a little time, unfortunately. And then you can now tear down an environment. If I just go in here and type task tear down sandbox, it's going to go through and run the Terraform to de d undeploy all the infrastructure so I can stop paying Amazon per minute for all the stuff.

Speaker 1: I've just deployed. We added in some GitHub automation as well. So part of this process is actually push some variables into the GitHub repository when I push the code up there. And then we updated Argo C D to the latest. The traditional deployment before this was, you know, you SSH'd into servers. If I never have to SSH into another server again, I'll probably be a pretty happy person. It makes me happy to see all the this you know kind of Rube Goldberg of mechanics going on behind the scenes, saving me a ton of time. There's no more manual Docker Compose ups. So even if you are using containers, but you're still using something like Compose for deployment, you're still having to do manual stuff. We tried to eliminate the works for me bit. Deployment should not be anxiety. That's another piece we're fixing for as well. And we want to know who did what and how do we get back to it. This is the post-GitHub's

Speaker 1: GitOps in world, which is a Git push should result in a deploy if you're in the right branches. It's automated and consistent, it should work everywhere. Should provide provide confidence and everyone should be able to do this. If you want to get started today, I'll have some links in the prep uh presentation for Scaff. Uh it's a one-line install. You'll be able to type like I did, Scaff my project. It'll build a Django application. application on your local system. And then log into the AWS CLI, you'll be able to do all the rest of the pieces. And be able to edit the things and watch the manage and deploy. You don't need to be a Kubernetes expert. You don't need complicated CICD pipeline. You have one main pipeline building images. It's not very big. And you don't need a huge team.

Speaker 2: This should not be too Uh all the risk on you. Oh boy. Come on. Actually, okay. That's bigger than Oh , There it is. Okay. That 's the part I really want to see. We may still make the demo happen by the end. So it watches for that pod to be created. You saw on the very bottom set sealed secrets right there. Once it's ready, it's going to push the sealed secrets from my environment into it. And now Argo

Speaker 2: CD will have the ability to do the rest of it 's magic, which is actually right here. There, you push the secrets into the GitHub repository, which is triggering the next build. key bit of this right here. Go back over into our repository. There it is. That sealed secrets just got uploaded and it's kicked off last bill. We now go into Archivo And everyone in services has a public URL for actually managing the DNS node inside the Amazon for us. That's not right yet. No, uh so the that's why I'm waiting for the certificates to be made.

Speaker 2: Once the certificates are made. Every cross your fingers? Maybe it's wrong thing. Once I think it's replaced, I'll be able to get to the RV by actually while borrowing that because I think it's the R The Argo password, when you install Argo CD, kubectl command right here that you can just run to get to your Argo password. And you just make sure you export it out of QCPR. And I can run that. And there's that the password for that. And we didn't know that and

Speaker 2: jobs are um but we are now at our lunch break for today. Um So I would recommend uh if it's more than visual personal um , uh true, okay. That's a good sign. Bingo. So we are now logged into our CD. It has a root application and it also has all the little pieces that are getting deployed right here. Can you all read that stuff? Here we go. This is waiting for the

Speaker 2: uh what we just submitted in the GitHub. When those images, as soon as those images get built over on the GitHub Action side, they're going to be pushed over into ECR and the manifest will update and this error will go away. There it is. So you can now see all the components of our application lining up and going green. I won't have time to go log into all the apps, but The Django app is coming out, the Redis is coming out, the database is all coming out, like everything now, it's basically becoming live. When this app gets to green, you can go log in to your Django app. This is all running in Kubernetes. on Amazon, hook into my repositories, if I make a code change locally now and it will now deploy, it takes about a minute or so.

Speaker 2: But if I don't have to I I can sit back in the developer and focus on just my code which is exactly what I want to do developer. And last thing, the links in the presentation, and I am not a hardware to define. Please talk to me. I love what we're chatting about with Thank you all so much. And then I'll be on the I'm I'm one minute over. I didn't break it now.