One Thousand and One Wagtail Sites

Speaker 1: Hello everyone. We're going to start up again. Before I introduce our next speaker, I'd like to let you know that it's not too late to join the Lightning Talks if you Go on the the Wagtail Slack, you can you can find that. And with that, I'd like to introduce our next speaker. Vince Salvino is a technical director of Code Red, and he is also a Wagtail core team member. He is going to be talking about 1,011 Wagtail sites. Thanks, then.

Speaker 2: Thanks everybody. Thanks for that introduction. So yes, we're going to talk about 1001 wagons tail sites and this is gonna be hopefully a little bit fun. I got you all after lunch so everyone's full and sleepy and ready to take a nap. So uh I will forgive you if you fall asleep during my talk. So um Yeah, this is going to be told in the style of uh 1001 Knights or Arabian Knights, so hopefully it'll be something a little bit different here. Okay, so Okay, there we go. So once upon a time, there was an agency named CodeRed.

Speaker 2: Building many Django and Wagtail sites, they must host them in a manner that would be pleasing to clients. As the sites numbered in the dozens, then in the hundreds, the magical lamp stack became burdensome. Weary with maintenance and fatigued by three-letter acronyms from the Almighty Cloud providers, they set sail for a new platform. So this will be a story of adventure, of treasure, of monsters, and of wisdom on our journey to hosting 1001 Pac Tales. So the inspiration for this was we launched a hosting platform in 2020. Because of the pandemic and all that, we did not really uh

Speaker 2: get to promote it or anything. It didn't really launch as we intended. So uh it's been kind of a slow start, but just recently this year we have crossed uh the milestone of hosting over a thousand uh sites on our platform that's a thousand separate individual sites so it's a really cool platform or a really cool uh milestone and kind of in combination with 10 years of Wagtail, just a lot of great milestones. So I thought it would be fun to do a talk about it. So this is my quick shameless plug. I promise this is not going to be a sales pitch, but I'm with Code RED. You may have seen or heard of our platform. If you hadn't, I encourage you to check it out because this is something we've built specifically for Wagtail and Django. So okay, here we go into

Speaker 2: the story So we'll start with Sinbad the Sailor, and this is going to be an analogy for finding the ideal technical architecture for hosting many different sites. So it starts as as all good stories start. Sinbad is setting sail and he decides to choose the traditional lamp stack. That's the Linux Apache, MySQL, and P can be PHP or Python or Perl. So business is good. His small boat has no problem handling a few clients here. there. It's really easy to manage a simple little lamp stack putting one or two sites on it. But something happens.

Speaker 2: His first challenge is the den of pythons. As more sites are added, Simbad quickly learns that each has wildly different resources. Memory space, CPU time, Python interpreters, they must all be managed, they must all be restricted on a per-site basis. So what ends up happening is when you have a lamp stack or something simple a setup like that, one site, sure, it's fine. You add two sites, okay, that's not bad. Then you start adding more and more and more, and when you have sort of this single, you know, master Apache process running, uh, you know, a bunch of uh sub mod WISGs uh the resources start to kind of get out of control. One site might take up too many resources and then starve another site of resources.

Speaker 2: And that's really bad if these are two separate sites and one has the ability to kind of choke out the other and that that's not something that you can that's not very reliable that's not you know um so it you quickly start hitting some limits on things like that. And that was what we started encountering in the very early days. So early days we were an agency like many other agencies in the room, and we you know you have the client says oh thanks for building this site can you host it for us say okay you throw it up so that's how we started pretty much like everyone else So seeking a solution for this growing problem , we find the great whale. And there's not a better analogy story for this.

Speaker 2: Seeking safe haven, Sinbad discovers the island of Dokker. And in the story, he sees a small island. He goes to the island, but it proves to only be the tip of the iceberg. For the island is actually a giant whale underwater, and it basically destroys his boat, destroys his crew And I think this is such a great analogy for Docker because you get into it and you start with, you know, I have one little Docker file and I run one little Wagtail site locally. You know, great, that works perfectly fine. Let's use this in production. And things start to get out of control really quickly with Docker. So the Docker is eventually tamed.

Speaker 2: We eventually kind of figure out a good way of doing that. And what happens is you've now created a monster. So having if you have a hundred websites and you decide to run a Docker container and you have a hundred different Docker containers Well, you've just created just as many problems as if you had 100 sites to start with. You haven't really fixed anything. You've just kind of rehashed it. So to control this monster, we've taken a unique approach. Basically, a single image a single Docker image for all of the sites and then the sites are managed in the file system and we mount the sites into the Docker image using the Docker image as basically a Python environment.

Speaker 2: So the image is the image is not aware of your app at all. The image has nothing specific to your app. It's just a basically a perfectly set up Python environment that you can then load your site into a runtime and do everything. So the site is not being built with Docker. The site really has no connection to Docker. It's just getting loaded in at runtime. because it has all of the Python stuff perfectly configured. But you still have the problem of all these dockers everywhere. And you might be tempted to say What about Kubernetes? Or what about Swarm or some of these other Docker approaches? And those are really not the right fit for this scenario because

Speaker 2: Those often are for having one site where you're running hundreds or thousands of instances of that site distributed over a cluster, for example. So like if you're Netflix or if you're Google or somebody of that size and you need to run a thousand Netflix workers to handle the latest uh episode of Stranger Things that just dropped or something like that. Then Kubernetes is probably a great fit because you're just you're gonna have Kubernetes say give me a thousand workers of my site and it spins them all up But when you're running a thousand different websites, that's not how it works. Kubernetes is not going to manage that for you.

Speaker 2: So those types of tools are not the right fit for this particular way of using Docker. So essentially, we have created an agent that runs on the server. The agent controls the dockers in a very specific way where we say, I want to deploy this site, I want to Update the Python version of this site. I want to apply a security update here and the agent can generate that new Docker container on the fly and load everything in. So okay, that works really good So this is this took many years by the way to develop. So essentially we've boiled it down to a brutally simple deployment process. And I'm just going to take a minute here to kind of show off.

Speaker 2: So this is a dead simple Wagtail site. If you run pip install wagtail, Wagtail Start Test Project, this is what it spits out. As a matter of fact, I just did this this morning while sitting at the registration desk. So this should look familiar to everyone. It created a home app, it created a manage. py, it created a requirements. txt, nothing special here. This is how simple we wanted it to be to deploy. We don't want to do Docker builds. You notice there's no Docker file here. You'll notice there's no special uh Heroku file or WISGI file or anything like that. This is just a vanilla Django project.

Speaker 2: So what I'm going to show you here is kind of the final deployment process that we've created. And I have to say I was really glad to see Tom 's talk this morning. I'm always inspired by Tom's State of Wagtail talk every year. I love seeing that. And uh this morning he mentioned something about What if we had manage. py deploy, right? And yes, uh we have had the exact same vision for that. What if we had a one-line deployment script? So That is actually what we have. It is specific to the Code Red platform, but that is the test project that I just showed you that I created this morning.

Speaker 2: This is my one-line deployment script The server is actually we're copying the code up to the server, we're deploying, uh, we're queuing a deployment. which that agent that's running on the specific server is doing. And as it fires up that new container under the hood, it pip installs everything. The pip install actually takes the longest because of the many dependencies. So we're always trying to minimize dependencies after it pip installs. It is going to run collect static. It's going to run migrate and apply the migrations. And it is actually going to wire up the static and the media URLs that's in your vanilla Django

Speaker 2: settings and fire up the server. And that is it. Now the last line you could see here is that it created a URL for you, testproject. code red dot cloud. And you can actually go to that right now and you will see that there's a vanilla wagtail site there. So this is our one-line deployment process. And because of this, we're super high like this is the end goal that we wanted. We wanted to build this for ourselves because we had so many clients and Even if you're not an agency, if you work at a big company or if you work at a IT department, you are probably that central resource internally, right? I've had many People tell me they want to build their own

Speaker 2: hosting platform for use inside their company. And I sympathize with that because that's the exact same problem we had. And you know, it's quite a challenging thing, as you can see from this this journey here, but this is kind of the end result. So this makes it simple enough that we can literally manage a thousand Wagtail websites with no problem at all. And our customers can manage their own sites. you know, themselves without even having to interact with us unless they have a problem. So it's just really, really great all around. There's that being said, there's many improvements that we still need to make, but we're definitely going in the right direction with this So, okay, now we're going to move on to another story.

Speaker 2: This is the story of Aladdin. And many of you are familiar with this because it's been retold in many Disney films over the years. But essentially your wishes might command. And that refers to the business challenges. When you're working for a client Very often you are of the your wishes my command mentality and the same as if you're working in an IT department and you're a shared service. You know, uh you want to be able to fulfill your users' wishes in a very timely and effective way. So what happens to Aladdin? Well, he receives a visit from his first client, and this client happens to be a magical sorcerer.

Speaker 2: The sorcerer. Will reward him with ridges if Aladdin can help him with a difficult task. However, Aladdin soon finds himself trapped inside of a dark cave And I don't mean this to be condescending, but when you are offering business and you're working with a client, you're working with it, you may find yourself trapped in a contract that is a little bit more difficult to fulfill than you initially thought. Right? Okay. I know some people in the room have probably experienced this. So uh Aladdin learns that web hosting ends up requiring a immense amount of other services that everyone just kind of assumes are there, right?

Speaker 2: So the site must run reliably 24 hours a day, 7 days a week, 365 days a year. And it must have a 99. 9% SLA So that's a lot of time that you don't want to spend having somebody sitting at a desk watching a blinking light making sure that it stays on the whole time. And 99. 9 % , that if you calculate the number of minutes in a month in a 30-day period. 99. 9 % means you can only have a few minutes of downtime per month. I mean a few minutes If it goes above a few minutes, your clients are going to be calling up, they're going to be angry, customers are going to notice that the site's down, you might lose out on business, you might lose out on sales.

Speaker 2: So like Unless you're really monitoring your site, a lot of sites that you start monitoring, you find out they're actually not running at 99. 9. They're running a lot lower than that. Sometimes even 80 or 90% uptime, I've seen. And that's one of the really difficult challenges that you just don't really realize until you start doing this at scale There must also be disaster recovery. So when we talk about disaster recovery, the first thing that comes to mind is A hurricane just hit the East Coast and took out US East 1. We better fail over geographically. But that's that's like the really worst case, like most remote possible thing that could happen.

Speaker 2: The day-to-day disaster recovery that happens is, uh, oh crap, the developer just pushed a bad update. And we had a Django migration that worked locally, but it didn't work on the server. And now everything is throwing a 500 error. What do we do? The migration went bad. That can be prevented in development, but the reality is stuff like that can occasionally slip through and happen. So that's a disaster right there because that's immediately gonna take your site down or have your site start throwing errors where it's unusable. So that's disaster recovery. You need to be able to heal from that really quick before ideally before anyone notices so that you can then fix it and push.

Speaker 2: uh the correct update. So that's the day-to-day disaster recovery. So you need backups for that, right? And sometimes A backup, you're going to need a complete backup, you're going to need a database, you're going to need media files because you can't rely on version control alone Right? If you have something in version control when you push a bad uh Django migration and it messes up something in the database, Nothing you have in version control is going to be able to fix that. You need a database snapshot now to fix that. So just having a complete backup is something that is often overlooked. You need tools for doing these upgrades, downgrades, for rolling back changes, for taking snapshots, things like that.

Speaker 2: So it quickly, you know, you can find yourself getting into more than you bargained for when you start doing something like this at a larger scale. So how is Aladdin going to possibly solve these problems? Well, luckily for him, he can summon the djinn or the genie. Aladdin quickly learns the financial burden of providing these services grows exponentially. For example, the backups. Let's say I'm hosting a 20 gigabyte site that includes 20 gigabytes of media files, database, backup, and code. Not really that big of a website, right? Most of us are probably working with something similar or larger. But think about that 20 gigabytes.

Speaker 2: We store 180 days of backups for some of our for our like professional plan. If you want to store 180 days of backups, that's 180 times 20 gigabytes. And you might say, oh, 20 gigabytes, storage is really cheap. I'll just throw it in S3, or I'll throw it in whatever block storage 20 gigabytes times 180 days works out to over three terabytes of data. Three terabytes is not exactly cheap on any cloud provider at this point. So that might be way more than you bargained for, or maybe you quoted out the cost for hosting 20 gigabytes and you didn't exactly factor in the fact that you're going to need three terabytes of backups.

Speaker 2: uh to store that for six months. So yeah, things like this just really kind of can catch you by surprise. The only solution for this is to for Aladdin to summon Superhuman levels of efficiency. So this is achieved through the miracles of automation. Things like automating the backups and the snapshots, moving them automatically from faster storage to cheaper storage that can be archival. Maybe the the slow storage takes several hours to read and write from. But that's something that you can put a backup that's six months old on and that's sufficient. Also automating things like uh the mini disaster recoveries that need to happen on a day-to-day basis, right?

Speaker 2: Uh having uh kind of an uptime tool monitor the site. If it detects a problem, is it an SSL problem? Is it that the domain expired? These are things that you can proactively watch for to prevent them from happening. And things like a 500 error. Is it the case, if there's an error, is it the case that maybe the site used up all of its resources? So it used up all of its CPU, it used up all of its memory, it's just maxed out 100%. In that case, maybe you can try automatically restarting the service to heal that without even having to have a person involved in it Or maybe you need to fully redeploy it because you can tell that a deploy just happened, then the site went down, and there's a little bit of a pattern there.

Speaker 2: and the solution is going to be to automatically trigger a rolling back from a snapshot. So that helps you keep your 99. 9% uptime and it also doesn't involve a person at all. It's self-healing in a sense. So these are a lot of the tricks and things that we have had to do to be able to work very efficiently and make this happen. So just some of the business challenges that happen kind of under the hood when you're hosting this many websites. And the final story before I run out of time will be Alibaba and the 40 Thieves. So this is probably the most fun anecdote that I often tell people about is just dealing with the sheer number of spammers, bots, malicious sites,

Speaker 2: even malicious customers. So it all starts with an open sesame. Alibaba sees that a large portion of web traffic is evil bots attempting to brute force URLs. A fun fact about this is that a Wagtail site, while it is secure, it can be very easily taken down by 404s , particularly to XMLRPC. php. There's nothing special about that URL. It's the 404 just like any other, except for the fact that bots just absolutely love that one URL. They love posting things to it. They love just hammering it without

Speaker 2: uh relentlessly hammering it. And that's because that is the most vulnerable piece of WordPress. WordPress has nothing to do with Django. except for the fact that every one every bot will just look for vulnerabilities regardless of what the site is. It doesn't care. It's just going to start probing and it's just going to start brute forcing. So We get to inherit many of WordPress's problems by proxy simply by existing. So A 404 in Wagtail is actually the slowest thing that happens. It is one of the more I won't say performance intensive because it's not intensive, but What happens when Wagtail finds a 404?

Speaker 2: Well to do that, first it looks at the Django URLs. Those are really quick. They take, you know, there's no database hits involved, you know, there's no file storage hits involved. That's like loaded in memory. So if you have a Django URL, Django can respond super fast, like in your URLs. py patterns. The next thing is if it doesn't find one of those, it looks at the big catch-all one that Wagtail has, right? It's usually just blank empty string include Wagtail URLs. So to find that it has to look through your entire database to see if you know using the tree beard and all of that to find the URL structure and find a page that matches So if no page matches that, you've just scanned through your entire database looking for something.

Speaker 2: But we're not done yet. There's more. The next thing after that is it has to check the redirects to see, oh, well, if this pay if there's a no Django URL that matches it and there's no URL in the database that matches it, maybe there's something in the redirects table So it looks through all the redirects, and then if it can't find one of those, then it gives you the 404. So the 404 is actually the slowest thing in Wagtail. And pair that with a bot that is hitting a URL which is a 404 that doesn't exist, it is just hammering your database non-stop And something like this can take down a wagtail site quite easily. So that is one weird side effect that we have also discovered.

Speaker 2: Now this can be mitigated in many different ways, one of which is just by caching that. So if you're using Wagtail Cache, it does cache 404s to prevent this problem. But you know, it's just a weird side effect that no one would ever have anticipated. But this happens on WordPress. You know, we host WordPress sites too, we host static sites. So this is a problem on every site just because of the bots. It just has a very unusual performance aspect in Wagtail. So Alibaba must find a way to stop these legions of bots. But how to distinguish a bot from a human? It's actually quite difficult because they can spoof their user agent. They can spoof other things. They can come from a variety of IP addresses.

Speaker 2: But The best way is by identifying their traffic patterns. So a human might actually make more requests than a bot. But that's because they hit one URL that's a page, then they have to download a bunch of CSS and a bunch of JavaScript and some tracking files and it's like okay there's an interesting pattern there that you can kind of see When a bot does something, it's not going to bother downloading the CSS and the JavaScript. It's just going to hammer an endpoint or it's going to start crawling and hammering a bunch of different endpoints usually in regular intervals as well. So a human might go in chunks, right? It might request a URL and then within a matter of milliseconds request

Speaker 2: a dozen or a hundred more URLs to get all the CSS and the images and JavaScript, right? And it's going in kind of chunks like this. The bot is not going to do that. It's going to be much more interval because it's usually running on a timer or a job or something. So there's a little bit of a pattern here. So what we ended up doing to solve because this this was a major problem for us. This was like taking down sites left and right. When you get under a bot attack, you cannot stop it you know, especially when they're coming when they're distributed and they're coming from different data centers. Uh and they might not be targeting you. They're literally just targeting any website out there. They're just going to crawl things nonstop. So, how do you stop this problem? It's a major problem. Well, what we ended up doing was writing a sentinel, we call it, and this is basically a middleware that sits in the reverse proxy

Speaker 2: And it watches all traffic across all sites that are on the CodeRed Cloud platform and it identifies patterns, what IP addresses are requesting what URLs. Is it gets, is it posts, what are the time the timing differences between them, all these different patterns. And when it identifies something that's a bot, it temporarily blocks it from accessing that site. And after we rolled this out, I mean it's the classic graph that you see where it's just sustained, everything's maxed out, you roll something out like this, and it just drops off a cliff. So that's one of the many security challenges that we've had. The second security challenge is preventing fraud. Now this one's a little bit unique because

Speaker 2: if you're running something internally you may not have as many fraud problems But if you're offering a website that has user signups or has users, you know, people signing up, things like that, you're likely to get some fraudulent activity. And you might say, well, let's make them provide a real email address and verify that with like a two-factor code. But unfortunately, real email addresses are no longer very meaningful in the current day and age because they can be generated by the thousands. And things can easily respond to them. I mean, you can't just block mailcatch. This is literally like companies everywhere.

Speaker 2: People can just generate email addresses. This was a major one for us because at a certain point we hit this threshold where people were doing this. And I think it was coming from certain pods of users or bot farms or something that were just generating thousands of accounts and trying to spin up sites. And I will talk about the next issue. that they were doing. But uh, you know, how do you verify? Because they're already verifying their email. They're already providing phone numbers and addresses. Well, the solution in this case was we found an excellent tool that I want to pass on to all of you called MaxMine. Now MaxMind, you may have heard of them because they do the GeoIP database.

Speaker 2: It used to be like a Linux package, and now you have to download it from their website. But you know, the IP address lookup database Now they actually offer some other services. This is just a quick shout-out to them because this has been one of the most amazing services that we integrated with, called their Min Fraud. And basically, you can feed this whatever information you have, IP address, email address, addresses credit cards, anything. You can feed them that and they will provide a detailed score of how risky that information is based on other lists that it's on, where it's originating, if different patterns match up. And this has drastically dropped off our user submissions after doing this

Speaker 2: fraudulent use. So that was a huge win. And the very final thing I'll talk about because I'm probably over time is code injection. So Alibaba notices that the thieves who do get through are very clever. And they will craft their manage. pys to do certain things. Can your manage. py do this? Can it download binary executables? Can it open HTTP proxies? Does your managed. py run a Tor network? How about mining Bitcoin? Can your managed. py do that? Technically, yes, it could do that because it's a Python script.

Speaker 2: And if you just completely ignore Django and write a manage. py that's a few thousand lines long. that uh opens an HTTP proxy and tunnels traffic. Uh yes, it can do that. Running manage. py collect static will not do what you think it's gonna do. So this was just one of the other challenges we've come across, having to monitor processes and monitor things going on on the servers. to prevent people from you know uh from these injection attacks from happening. So yes, that's that is uh the final end of the story. We've hosted over a thousand wagtail sites. Uh you you can do it. Um it is a interesting experience. There will be many challenges, but you know many rewards as well.

Speaker 2: So thank you very much for for having me here and listening to my story. As I said, I'm with CodeRed and you can check us out at codered. cloud. Thank you. Probably don't have time for questions, but if we can squeeze them in. Oh, okay, great. So yeah, happy to take some questions then Yes.

Speaker 3: Can you share those beautiful slides?

Speaker 2: Yeah, yeah, I would be happy to share them. I will post a uh upload them somewhere and post a link. uh afterwards. I think we're also recording this, right?

Speaker 4: Yes.

Speaker 2: So we'll be recorded. Yes.

Speaker 3: Existe hability to attach any domain to

Speaker 2: Yes, good question. So yeah, Wagtail supports multi-site and our platform initially we did not support that, but now we do support multi-site And basically to do that in Wagtail, you just need to point all the domains to the same place. And whenever Wagtail gets a request, it will then handle that with its multi-domain feature So yeah, yes, we support that and Wagtail absolutely supports that. And we are using Let's Encrypt for SSL, so that has also been automated to just generate the certificates for multi-sites. Uh any other questions?

Speaker 3: Okay. Slack channel.

Speaker 2: Oh, okay. Great. Uh yeah, I'll be happy to uh talk to people on Slack afterwards. I will I will check in there. Oh yeah.

Speaker 3: So how many sites did you

Speaker 2: Yes, yes. The question was will we be hosting 1002 sites? So uh yeah, I hope the answer is yes. Uh yeah. The the exact number I is over a thousand. I I don't know the exact number as of right this minute, but uh yeah, we we broke, we passed it a thousand earlier this year. Very exciting. Okay, thanks a lot everyone. Thanks again. I will be answering questions in Slack as well.