One Thousand and One Django Sites with Vince Salvino

Speaker 1: It's wonderful to be back at DjangoCon. I always love this conference and it's been quite a few years since I've given a talk, so very excited to be here. I'm going to be talking about 1001 Django sites. And this is kind of a play on the Arabian Knights, also known as 1001 Knights. So it will be a little bit different, hopefully, kind of a storybook format. So as all good stories start, once upon a time. There was an agency named Code Red, which is where I work. Building many Django and Wagtail sites, they must host them in a manner that would be pleasing to clients and developers. As sites numbered in the dozens, then in the hundreds, the magic lamp

Speaker 1: stack became burdensome. Weary with maintenance, fatigued by three-letter acronyms from cloud providers, they set sail for a new platform. So this will be a story of adventure, of monsters, and of wisdom on our journey to hosting a thousand and one Django websites. So kind of the backstory behind this here is that I'm with Code Red and that's our uh we're a small small company in Cleveland, Ohio. We've built a platform called Code Red Cloud, which we launched in 2020 And uh it's a hosting platform, so we've recently crossed a major milestone of uh hosting a thousand uh websites on our platform.

Speaker 1: So I wanted to share just some of the absolutely crazy stuff that goes on behind the scenes when you're just dealing with hosting this many Django sites. So yeah, that's my shameless plug. You can check us out at code red. cloud. We have free accounts, all that good stuff. The other thing I wanted to mention is we specialize in Django, but you hear me talk about Wagtail a lot. So this is just a completely random side note, but a shout out to the Wagtail Open Source Project. It's a It's a huge I'm a contributor to that. And this is what Wagtail looks like. It's kind of like the uh Django admin, if you think of that, but just on steroids, really amped up, uh really amazing products. So you'll hear me talk about Django and Wagtail, sometimes interchangeably, because uh you know Wagtail is built on Django, so

Speaker 1: uh a lot of our stuff is just built around Wagtail So, alright, enough background. Let's get back to the story. So, we're going to start with our first story of A Thousand and One Nights, which is Sinbad the Sailor. And this is going to be about finding the technical architecture for hosting this many Django sites. Setting sail. Sinbad is a young man looking to make his way in the world. Like many famous great world leaders of antiquity, he decided to use Django. I'm embellishing a little bit That's not the original story. Armed with Django, he seeks out fame, fortune, and riches beyond imagination. That's what we all get when we use Django, right?

Speaker 1: So, okay. So his first voyage, he takes many voyages in the story. His first voyage is that he begins by developing a few Django sites. For his clients, and this is a great success. Setting sail to host these first Django sites Uh Sinbad chooses the traditional uh time-honored lamp stack. This is the framework that his forefathers used, the lamp stack. The Cs are choppy and the Linux is unforgiving, but Django sites run admirably well on this lamp stack. An excellent choice, he exclaims. But Sinbad has yet to encounter his real first challenge.

Speaker 1: And that is the den of pythons. As more sites are added, Synbad quickly learns that each has wildly different resource usage, memory space, CPU time, Python interpreters. They must all be restricted and managed and handled on a per site basis. How can Sinbad tame these pythons when there's so many of them on one system? He once again employs the tools of his forefathers, ModWISG , subprocesses, virtual ENVs, system quotas. Uh all to no avail. And of course, these things can be used to manage when you're running multiple different Python processes on one server, but it proves to be quite a complex challenge.

Speaker 1: So, with this, Sinbad seeks his second great adventure, which is the great whale. So, looking for a safe haven, Sinbad discovers an island, the island of Docker. The island at first appears to be a land of ease and low-hanging fruit, as Docker is, right? But soon Sinbad hears a rumble. The island actually turns out to be a great whale in disguise. And this whale destroys much of Simbad's boat, his crew, everything. So I I just thought that this story was such a great analogy for Docker because several years ago when the when this was all new, exciting stuff, we kind of jumped on that bandwagon

Speaker 1: and Lamp stack and Docker and you know you get into even things like Kubernetes and uh those are all great frameworks, but they can easily, you know, you can easily kind of lose control of those things if you get a little bit, you know, beyond your depth in them. um nautical pun intended. So uh eventually the Docker is tamed and put to good use. So we eventually figured out how to use you know this in a kind of a unique way to handle so many websites. Uh but once you get one Django site running on Docker, it proves to be Almost uh is just as much of a problem um having many Dockers as it is having many individual Django sites because they're all different, right?

Speaker 1: So to control this beast, a unique approach is taken, and that is we ended up creating a single Docker image, which is almost like the environment. So rather than being a single Docker image for every single site, we create actually one universal Python environment. And kind of mount that into the file system, and every single site is using an identical Python environment with its own unique code running inside of that environment. And to kind of control all of these different environments across all these different servers, we end up creating Agent, which is just sort of our own little utility running on the system to manage all these Docker containers.

Speaker 1: So further kind of taming that, um yeah the the agent manages all of the containers across all of the servers. You might be saying, why don't you use Kubernetes or Docker Swarm or many of the other tools that have been invented for this purpose? But those tools are almost always used for running many copies of the same site, right? So if you're Netflix. Or if you're Google and you say, oh, I have a million users, you know, it's Friday night and there's a million users trying to watch Stranger Things. I'm gonna call up my Kubernetes and have it fire up uh a hundred more Netflix dashboards uh to handle all of this traffic.

Speaker 1: So that's great for Kubernetes, but what if you're running a hundred different Django websites for a hundred different clients? You don't want more copies of those because every site is different. So that's why we are kind of in a little bit of a unique situation. And you know, any agency really who is managing lots of websites is going to have that same problem. So with this, you know, kind of single universal image uh and as an environment, we can now spin that up for Any random site, you know, any random Django code, Wagtail code, what have you, can just be loaded into a single environment and called up and kind of spun up on the server. So, you know, and once again, as I mentioned in the beginning, this has to be good for the clients, but also for the developers, because we do not want to spend time managing this kind of stuff.

Speaker 1: So I have just a really quick uh demo, just a break from the story and get into a little bit of uh coding here. Um This is a Django project, right? There's a manage. py, there's a requirements. txt, this is on my local, which I just did this morning. So there's a V and V in there. So How would I go about deploying this? Well, normally you might say we gotta build a Docker image, we got to deploy, you know, push that image up to a registry, or we gotta use some kind of AWS services. Don't forget about your S3 buckets and all that kind of good stuff. So we want to deploy this in like 30 seconds. You know, we don't want our people spending time on this, and I don't want to spend all day deploying a Django site

Speaker 1: So kind of to show you the end result of this whole system we've built, we've of course created a little command line tool that helps automate some of that. So This is just, you know, if I'm in my uh command line here, that's my little project, which is just a boilerplate Django site. I'm just going to call my uh deployment script and that's going to go ahead and package up all this Django code, upload it to a server. Where that server then creates a brand new, fresh Python environment using a container kind of under the hood, loads all that code into it, starts up the container. And bam, I've got everything is fresh. It issues domain names, it issues SSL certificates, it runs migrations, it runs collect static, pip

Speaker 1: install, all that good stuff. So that entire process has just been automated down to like a you know 30-second uh deployment. So awesome. That's exactly what we were looking for from the technical side. And we finally achieved that using kind of a combination of many different techniques. So the second story that I'm going to tell you is about Aladdin. And this kind of goes more into the business side, right? So we kind of figured out the technical side that may have taken a few years to get to that point. Uh but now the business side comes into play where, well, all this stuff costs money, right? So um And I think the uh Latin analogy is perfect. Your wish is my command, right?

Speaker 1: Has anyone here ever been a freelancer or works in an agency or something like that? And you're probably very familiar with the term your wish is my command. Because uh sometimes you know business challenges get in the way and you have to just uh do everything. So the story of Aladdin, he's trapped in a cave. Aladdin receives a visit from his first client, who is just so happens to be a magical sorcerer The sorcerer will reward him with riches if Aladdin can help him with a difficult task. Has anyone ever promised that to you? So uh soon Aladdin finds himself trapped in a dark cave. And I think this analogy

Speaker 1: is sort of uh trapped in a service contract. Um As Aladdin soon learns, web hosting requires an immense amount of services, right? So we're not just talking about getting a site up and running. It has to run 24-7, 365, 99. 9 % uptime. If that goes down, someone's gonna be really angry and they're gonna be calling you and they're gonna be, you know, it's gonna be really bad. So uh but then just beyond that, there's disaster recovery, right? And we usually think about disaster recovery being like, oh well, a hurricane just hit the East Coast and all the power's out and all the data centers are, you know. But that's not really the most common disaster recovery.

Speaker 1: The common one that happens literally every single day is Somebody pushes some bad code, right? Someone pushes someone makes a mistake, it all happens, we've all done it. You push a bad code to production and something, maybe a bad database migration happens Oh boy, now your database is messed up. The data is maybe got wiped out, and that's really, really bad. And You've got some downtime on your hands, and this just happened within a few minutes. Now you're freaking out. So that's like the disaster recovery that literally happens every single day. And this is stuff we've had to build against. So the biggest one is backups, like a complete backup of everything. Media files, static files, database snapshot. a code, literally, you know, the

Speaker 1: the uh version of every pit package that's installed on the system, like the exact version, an entire snapshot. That way if you push a bad deployment and you have bad code or you accidentally delete something in your database or you accidentally delete media files, Uh you can just pull down a backup and within five minutes you're back up and running and uh you know it's no big deal. So Building all the tools for this and and you know accounting for all that is one of the like major business challenges that happens with uh hosting a site. You know, you don't usually think of those things up front until all these problems start happening. So, you know, how are we going to do all that? Well in Aladdin's case, he quickly learns the

Speaker 1: financial burdens of providing these services, and they grow exponentially, you know, the more he gets, the more problems he has. And uh the only solution for him is just this superhuman, you know, um supernatural uh level of uh efficiency and for him he has a magic lamp where he can summon the jinn or the genie. For us, we don't quite have that same luxury, so we instead use uh miracles of automation. So the way that we just automate the heck out of this, I mean creating the servers, creating the containers, creating the databases, SSL, DNS updates, you know, all these little bits and pieces.

Speaker 1: We are automating to happen in that 30-second demo that I showed you. All this stuff happens behind the scenes during that 30 seconds. The next thing is sort of disaster recovery, so automating some of that, right? If there is an error on the site, if the site starts throwing 500 errors, or you know, it can't connect to a database. Instead of someone manually having to intervene, could we potentially automatically fix that? So the air monitor detects that there's a database problem. It goes in and it looks at the database and it restarts it. And the error kind of heals itself seamlessly. Uh and hopefully the client never notices and no one ever notices

Speaker 1: because the problem only happened for 20 or 30 seconds. The next uh problem that we needed to automate was security monitoring, because security, as we all know The internet is like what 90% bots or something like that? You know, make up whatever statistic you want because there's just a huge number of bots. Uh, and it plagues every single website. So what happens when bots just start pummeling your your website or you know trying to brute force something or just trying to spam you? You know, that stuff can take down a Django site. You know, Django can only handle, you know, uh WISGE or GUNACORN or whatever can only handle so many Python processes before it just kind of

Speaker 1: Runs out of memory or just kind of runs out of allocated processes and just stops. So things like that monitoring the traffic, which leads me into the next story, our final story Which is Alibaba and the 40 thieves. So this is how uh this is the story of how the spammers and bots and malicious uh sites can actually really just uh bring you to your knees. And it all starts with open sesame. So Alibaba sees that a large portion of web traffic is evil bots attempting to brute force URLs. In the story, Alibaba uh finds a cave of treasure, and the the 40 thieves

Speaker 1: want to break into the cave of treasure. In particular, a Wagtail site is particularly, I won't say vulnerable, but it can be very easily taken down simply from 404s. And a lot of Django sites suffer from this too because you might have a URL, maybe it's an API endpoint, maybe it's something like a 404 or a search endpoint that can take a query. Right. Um, and it has to look things up in the database. And maybe you've got several levels of looking up where you look something up in the database. If it's not there, then you hit an API, and maybe if that's not there, you look up something else. And if that's not there, then you finally tell the user, oh, sorry, I couldn't find what you were looking for.

Speaker 1: Uh right, so these like very uh back end heavy endpoints. So uh what happens if you have a bot? that wants to just keep hitting that really heavy endpoint over and over. Maybe it's sending programmatically generated data to that endpoint. So it's busting any kind of caching you have. It's just creating a huge amount of problems. And the example I use is XMLRPC. php. And that really is means nothing to Django. But um that means a lot to a WordPress site because that's the single most vulnerable endpoint of a WordPress site. So every single bot is gonna just crawl along and say, oh, here's a website. I wonder if they have a vulnerability in their XMLRPC.

Speaker 1: php that I can easily exploit. And it's just gonna pound that endpoint and just keep hitting it. And we were seeing this. I mean, we had sites that were just getting hit by this and having so many 404s that it was actually taking down a Django site because of a WordPress vulnerability. Uh so yeah, you know, thank you, WordPress. But um WordPress is does a lot of things really well. So but uh just that one particular issue. So how do we stop these bad bots? Alibaba must find a way to stop the lesions of bots. But how can he distinguish a bot from a human? Well in this case we wrote a little sentinel middleware in the reverse proxy. So this kind of sits in that Nginx uh you know

Speaker 1: layer proxy and simply analyzes all the traffic that comes in, watches all the traffic. looks for certain patterns, right? So a bot is probably going to hit an endpoint over and over and it's going to maybe be generating data probably at a regular interval. Whereas a human is going to be making requests where they hit an endpoint, then they're gonna download a bunch of CSS, then they're gonna download a bunch of JavaScript, then they're gonna sit there for probably 30, 40 seconds. Then they're going to hit another endpoint and get all the CSS and JavaScript again. So you can kind of identify these different patterns. So uh using that, that was a pretty good way of blocking bots, and we were able, and it's one of those things where you see the graph and it's just like traffic, traffic, traffic, traffic.

Speaker 1: Deploy this, traffic just drops, you know, instantly drops. So that fixed a huge amount of our issues. But how do you stop bad humans? This is a completely different issue. So the 40 thieves like to use fraudulent info. To open fake accounts and abuse and steal resources. And anyone who's running sort of a SaaS product or a web service has probably had this issue. Email addresses are meaningless because you can go out there and create Gmail addresses by the thousands and they are real legitimate email addresses that you can two-factor authenticate against. Uh you can create Google Voice numbers, you can create, you know, fake phone numbers, fake emails.

Speaker 1: Like there's really no way to verify this. How we're gonna do it. So this is the case where we actually this was causing us so many problems because we were having literally thousands of accounts signing up for us and just abusing the resources and they were real people doing it. So in this case, I just want to pass this along to everyone in the room. We used a third-party MaxMind. They make the geo IP database that a lot of people use. It used to be included on like Linux servers and stuff. They have an API called MinFraud, and MinFraud actually can detect based on a lot of different parameters that you feed it. It can detect the likelihood that that person is a malicious actor, you know, if they're spoofing their info, if they're using fake info, et cetera.

Speaker 1: So Anyhow, the very last thing I will tell you about was code injection. So we stopped the bots, we stopped the humans, but there's still a little small amount of bad humans getting through. And we all know what a managed PUI file does, right? Manage PUI run server, manage PY collect static. Can your managed. py file download binary executables? Does your managed. py file open an HTTP proxy to evade law enforcement? Does your managed. py uh py file run a Tor network? How about mining Bitcoin? Can your managed. py file do that? Uh yes it can actually. A manage. This is a legal manage. py file. I'm gonna run python managed.

Speaker 1: py migrate And uh it's not actually going to migrate anything. It's going to download Node. js and it's going to unpack Node. js and run that on the system. And this is something that we encountered a shocking number of times with all variety of frameworks. This is like a very tame example. So uh, you know, that was code injection is kind of the last uh step of security that we had to address. So uh, you know Thank you. I'm just about out of time, but this is a little glimpse into uh the story of what it took to host the thousand and one Django sites. Uh there will be endless challenges, technical challenges, financial security, but you can get through it.

Speaker 1: It can be done. Thank you very much.

Speaker 2: Hi. Yeah. About the code injection, uh do you um in your experience uh do you think that that happened more like when it the the server was attacked or was in the development process or in which case

Speaker 1: good question. Yeah so the uh code injection can happen Uh one from having a malicious developer. We've never seen it from an actual uh hack on the server side, but uh It can be a bad pit package. That's a really common way is somebody poisons one of the pit packages. And then, you know, maybe it's not managed PY, but maybe it is. uh you know someone's installing a request spelled with a Z or something like that that is doing this code injection, you know, because it was a bad PIP package. But also we have seen straight up just malicious use where someone is trying to, you know, evade terms of service and things like that. So

Speaker 1: it's just one of those things you have to deal with. Yeah. Great question.

Speaker 3: All right. Anybody? Thank you so much, Vince. Appreciate you being here. And uh we'll have our next speaker up in five minutes