Deploy Django Event Platform Using Containers and Terraform | Calvin Hendryx-Parker

Speaker 1: My name is Calvin Hendricks Parker. I am co-founder and CTO of Six Feet Up. We are a Python and cloud consulting company based in the US, but I'm super excited to be here at DjangoCon Europe. I really love this conference. And what we're going to talk about today is actually what you're watching right now. So we're going to talk about how the platform works under the covers. For DjangoCon Europe and how we're deploying it with containers, what the developer experience is like, what the deployment experience is like. I will be watching for questions in Slack. Probably not in Slido because I can't get too many more windows on my screen at once, but it I would love to be fairly interactive. And if you feel like your questions don't get answered during the session, make sure you join me in the face-to-face by clicking

Speaker 1: the link below after the session is over and I'll join there and we can discuss further but I wanted to uh make sure that this is interactive ask questions throw things in Slack uh I will try and watch for those things as well. So as I mentioned, we're talking about DjangoCon Europe , the virtual event platform that is running under the covers here, powering uh the whole um virtual conference platform event we're watching right now. So as you move through the app and as you click in and and use the APIs and and and how it's kind of all working under the cover. So I'm gonna go back to this. But before we talk about how it's deployed and talking about the the production experience. I like focusing on the developer experience and how developers can be productive

Speaker 1: and how we can get you know more done and make sure that it's easy to onboard new developers. And just that whole experience around that. So that's why the first thing you see on my screen is going to be developer experience. So let me bring this up onto the screen. And we can kind of talk through this. And then we'll then we'll go through more of the deployment experience itself. So again, everything we're looking at is real time. What we're actually watching it on is the loudswarm experience. But for us. the the code base. So the developers where they they run and work every day. I feel like developers are most effective and efficient if they are developing locally. You want to have access to all of your tools, to your debuggers, to your editors, to your IDE, you know, whether you're using

Speaker 1: PyCharm, VS Code, whatever the case may be. I've focused on making sure that onboarding new developers into the project is kind of job number one and making sure that the deployment is smooth and easy. The developers don't have to worry about that. We've got you know DevOps engineers who are more involved and in charge for that area of the experience. Now the title of the talk is about containers and Fargate, and we'll definitely get into that, but we're going to start off with containers and building images We have built the platform using Django and we fully containerized it with using Docker containers, but locally the developers are using uh Docker and Docker Compose. to make sure they've got all the dependencies that match what's in production.

Speaker 1: So for example, if we're using Elastichash in production, locally in Docker Compose, we'll have a penned version of Redis running as one of the services under Docker Compose. So we can actually match versions that are going to be in production, even though we're not running locally an AWS cloud in a box on our own system. We're not even using local stack, so if you've not heard of local stack, that's a kind of AWS emulator that you can run locally to simulate various services. Uh our application uses Postgres for a database, uses Redis for uh caching and for the channels broker. We use load balancers and other things that are kind of part of the cloud infrastructure, but many of those things don't really matter to the developer. They just want to be able to write code and

Speaker 1: push changes and get them into production as quick as possible So that's why using Docker Compose locally for us has been a main part of this new developer experience on this specific Django project. So developers are developing locally on Django. They are pushing into a Bitbucket for our source control management. The only reason we're not using like GitHub or any others is just. historical I guess you could replace Bitbucket with GitHub or CodeStar stuff from Amazon if you wanted to. And then new changes as they go into the Bitbucket source control. are then built into the image in a Bitbucket pipeline. And then that image is pushed into the Amazon infrastructure. And this is kind of where our cloud journey starts from a developer standpoint.

Speaker 1: But before we get to this all the AWS cloud infrastructure pieces that we see over here, we're going to talk about Terraform a little bit. Because no one likes to go through the AWS console. and click, click, click, click and produce a beautiful work of cloud infrastructure art, only to not be able to reproduce it into another environment or for another project or to you know, take it and start a whole brand new greenfield project and leverage the kind of lessons learned. It's very difficult to do that if you don't have something like Terraform in place or CloudFormation or some other infrastructure as code component. that is playing a role as part of your uh deployment and developer experience. So the infrastructure's code for Terraform

Speaker 1: is a combination of we've gone through the console, we've gone through the AWS CLI and kind of poked and prodded at the things we need, and now we start describing all that in infrastructure as code using Terraform files. So we can run Terraform Plan and Apply against the AWS Cloud. Now the the infrastructure I'm showing you here inside this AWS Cloud, actually any AWS infrastructure and even uh the bitbucket pipelines and a few other pieces that are uh ancillary to the whole process can all be controlled by infrastructure as code and in our case we're using terraform The Terraform files that are specific to our project, we keep also, I should put an arrow here. Let's just go ahead and do that because that would reflect reality. uh the infrastructure's code is also checked into the Terraform

Speaker 1: repository uh uh that is also in the same repository as the Django code. So inside Bitbucket we have Django code and Terraform code living side by side, which is nice because if you, for example, add a new feature to the application that needs like Redis. and you need to have that infrastructure uh provisioned on the cloud for you enabled to to enable to use that feature, the code that needs it and the infrastructure that needs it is actually contained in say a single pull request and it can be released and rolled back uh all at the same time. So you kind of an atomic view of your code and of your infrastructure all at the same time. So the the result of you know Clicking, probing, discovering, and building, we ultimately will

Speaker 1: build that into a terraform. set of files that will now describe all of our our environments. And there'll be multiple environments. So I what I'm really showing is kind of one environment. But you can imagine the exact same copy of these for example our sandbox environment, our staging environment, and our production environment. So for Loudswarm, we do have those three environments available to us. So If you just we're just gonna kind of talk about one environment, but with Terraform, I've got separate folders for each environment as part of the Terraform files. It allows me to deploy this really easily. So that includes this build pipeline. So in the AWS cloud as part of the developer, you know, their experience, theirs kind of ends once the image is pushed into ECR. But that's where the production

Speaker 1: build part of the pipeline starts. We now have a container inside of Amazon, we can listen for events, for example, like a new image being pushed into the Elastic ECR's Elastic Container Repository. We listen for a new image. We can now kick off uh a code pipeline. In our case we use uh we have a make file in our base root repository for the code and we'll have targets in there for example uh make sandbox release That will send an API call to Code Pipeline to look for the latest version that's been released to ECR, tag it with the sandbox tag, and then start it down the process of actually deploying that code into production. Once that code is deployed, you'll see we also have an integration here

Speaker 1: with Lambda to give release notifications so that it's easy for developers to get feedback on when their latest change has made it into a specific environment. production, staging, sandbox, whatever the case may be. The code pipeline, you may also wonder why we have Bitbucket pipelines and the code pipelines. Again, more historical. We're kind of doing the building right next to where the source code is. This could very easily be moved into the AWS cloud, but just the way the project evolved, we went ahead and used the Bitbucket pipelines. We can also the Bitbucket pipelines are configured and maintained by a Bitbucket configuration file in the repository itself. So it's again easy for the developers to add tests or build steps to the image that would be specific to the developer

Speaker 1: 's experience on the on getting the image released out into production. So once we've gone through the build pipeline, we have code build tasks that will do, for example, deploying the Fargate tasks. So this icon here is a Fargate task. We will run tasks in code build to deploy, for example, the um Django static assets and media. And also in that same image, we have a React application that once build and webpacked compiled and all that, we run it and push it into another S3 bucket for the React application. So the whole application is deployed and pushed from the code pipeline. There's not typically any human hands that are involved in that part of the process.

Speaker 1: which is important because we want to be able to make sure that this is a repeatable process. We could deploy it again for another customer, for another type of project, leverage the same kind of blueprint for any Django project we may do, which Again, makes getting started on a new Django project very easy. So talking now about the developer experience, let's move over and talk about the actual application deployment. Scroll over. So application delivery, obviously a lot more components and moving parts and pieces, and this is where you can really see the benefit to using infrastructure as code type tools like Terraform. Because I don't think anybody wants to try and click through and create all these individual pieces of infrastructure and I'm not even documenting all the

Speaker 1: the network pieces, the subnets, the security groups, the I mean there's that gateway on here, but there's a lot of other moving parts that are under the covers here that aren't being seen. But are all controlled by Terraform. So if someone needs to make an adjustment to allow for a specific service now to hit a new port inside the application. It's much easier to just adjust that in Terraform, run Terraform Plan and Apply, you know, fuse maybe a less than a minute later, all of your changes can now be deployed into whichever target environment you were working on, whether you work on sandbox or production. You can also make sure you have separation of concerns and specific um AI or IAM roles inside your Amazon environment to allow folks specific access to say deploy into

Speaker 1: sandbox, but maybe the developers aren't allowed to deploy into production. Maybe that's limited to just the code pipeline and can only be triggered by specific users who are allowed to do those kinds of releases. So of all these components, the Fargate one is kind of the one we're here to talk about the most. is that the Django application, the WebSockets, the Cellular Workers, and Flower are all technically running the exact same image that we built and pushed into ECR. So Back back here from this step where we were actually running the build pipeline and pushing into ECR, once those things get deployed as Fargate tasks. It's really one image, but they're serving different purposes.

Speaker 1: So we have different uh task groups that can be scaled independent of one another. and serve different, I guess, workload profiles. For example, if you came to my talk earlier this morning where I was talking about hacking Django channels for fund and profit. We're talking a lot about Django channels and WebSockets and how they have a different profile and use usage like workload than say the Django REST API, which is what we're running here. There's no front-end, no server-side rendered Django. in in this application specifically. It's all the the whole front end application is delivered only out of an S3 bucket that's very static. So you can see that cloud front to S3 bucket, that's the whole delivery with the front end you're seeing. Everything else is API calls back into Django

Speaker 1: using just for Django Rust framework, and then the Slack integrations is all the WebSockets workers. But the WebSocket workers have a different Again, different profile, they the their workload is more memory intensive. Each user, so each event attendee that is connecting to a WebSocket will use some amount of RAM to maintain that state. for the WebSocket connection, whereas every at event attendee is not maintaining any kind of state to the Django REST API as part of the application, uh given that you know it's HTTP, it's all stateless. We don't worry about that. Typically the Django portion of this is more CPU intensive when we're doing the queries to the database to bring back the the schedules and the users and the speakers, things like that.

Speaker 1: So we can actually scale the Django REST API separate from scaling the WebSock. We can also use different Fargate container types for the Django processes than we do for the WebSockets. We may want to use more memory and less CPU for WebSockets, and more CPU and less memory for the Django application processes as well. And then the same goes for celery workers. If you have a bottleneck and maybe the workers are starting to queue up for doing specific kinds of tasks. You can now scale the Celery workers independent of the Django app and the WebSockets by adjusting a number inside Terraform, rerunning apply, and now it's going to provision or allow for more of those workers to run. So if you're not familiar with Terraform, I'm sorry, not Terraform, but the uh Fargate.

Speaker 1: It's basically not quite Kubernetes and not quite doing like Docker or Compose. It's in between like orchestration layer that's more managed by Amazon. Like I don't deal with servers. Actually all the things we're doing here are technically serverless. We're not managing any servers except for this one. There's a bastion host. that is used to securely uh access the infrastructure uh pieces here from a shell, but that's the only piece of the infrastructure that you could possibly shell into. The rest of these are all running serverless as Fargate tasks against Amazon's infrastructure. So Amazon manages the hardware, the patch levels, all those things that are related to the underlying infrastructure, like the hardware infrastructure, we're responsible for

Speaker 1: the infrastructure as far as how the containers run, how big the containers are, what the sizes of them are, and those kinds of details are really up to the developers who can make those kinds of choices because they know best how the application is going to need to use those resources. So being able to scale salaries separate from WebSockets and Django is a big benefit here because we can literally pinpoint and target and use our resources most effectively. The last bit here is flower. So we use flour to manage uh or to to monitor and manage uh the celery worker processes. And mostly that is done with CloudWatch through a Lambda so that we can actually get alerts and data into the CloudWatch dashboards. So we're using CloudWatch dashboards to get metrics and data back about the Django process, the WebSockets

Speaker 1: processes, and the workers, and Flowers helping to uh augment that and actually give us alerts if we don't see if we see certain tasks starting to fail, we can actually get alerted because we can watch flower from its API and push that data into CloudRatch metrics so we can get alerts. All that said, all this infrastructure again built by Terraform, we uh put in front of that an ALB. So the Django application WebSockets can actually Run in a single ALB, but we have different uh listeners on the back end depending on whether it's WebSockets traffic or whether it's HTTP traffic. So the ALB determines which one of these boxes services the requests. And since ALBs are fully WebSocket compatible, same thing for CloudFront, it's also compatible with WebSockets.

Speaker 1: So the whole process now allows us to have a globally distributed application running mostly out of S3 buckets. So the first page renders should be very, very quick because we're not actually hitting any kind of an application server like Django to render any of the front end pages for the application. For those of you who don't know, CloudFront is a CDN or content distribution network that allows us to have access to I think over 200 points of presence for delivering the application. So whether you're you know In India, Australia, South America, America, or Europe, it doesn't matter. The application gets delivered from a point that's closest to you. The application also gets a benefit from using Amazon's web application firewall. And so that web

Speaker 1: application firewall allows us to write rules to protect against specific kinds of attacks, whether they're SQL injection or kind of the OWASP top 10 is the base. starting point for the WAF rules that we use and then from there we'll customize based on the type of application, um specific vulnerabilities that maybe or are currently attacking. You can go kind of crazy on the WAF, but I'd start simple. There's some great initial recipes available from the Amazon Labs. like from their documentation kind of reference examples that are good starting points for getting started there. But that allows all the whole the sole entry point for the application though is still cloud front. um whether they're webhooks coming in from um the

Speaker 1: Slack integration so any new Slack messages come in webhooks or whether event attendees who are logging in and maintaining a stateful connection to the Slack WebSocket or viewing the sessions, that all happens here. And the there's some pieces that are outside of the AWS box. that I'll talk about as well. So I've already talked about the conference messenger. So the salary workers are typically watching for sessions that are starting in the next five minutes, and then they will send a message through the Conference Messenger into Slack. The video CDN is actually separate from all the rest of the CDN pieces that are inside of AWS, so that that's being delivered from our third-party video provider. We now support a persistent WebSocket connection to Discord. So we actually actually

Speaker 1: now that was a talk from this morning was how we actually enabled channels to initiate uh persistent connections on Django start to talk to Discord and log in and look for messages. And then we do use Sentry for our monitoring. So we'll use the Sentry middleware for Django. Which is nice because it gives us stats about Postgres, Redis, Django. There's a React plugin. I don't want to sound too much like an ad for Sentry, but I'd say it's a it's a really good product. And since they've added in their performance metrics to that product. It's actually helped us identify some slow points in the application before we launched on specific critical events. Also good to have in place. The other pieces, and again, all the pieces that are inside the AWS Cloud and even some that are out are controlled by Terraform, even down to the DNS.

Speaker 1: So we use Route 53. So when you go to DJC2021. loudsorm. com , we have a specific wildcard in there. We have some Django middleware that works with it to basically make this a fully multi-tenant application. So depending on which subdomain of the URL you go to depends on which tenant of the application you get. We also use SES for email, so any outgoing email notifications like invitations are going out through the Amazon's SES service the CloudWatch metrics, so all the dashboards we can monitor and watch for any performance issues happen there. We use um What's it called? Now I can't think of the top of my head. Guard, I think it's called Guard. Basically it was a checkbox. It turns on some some minimal like security

Speaker 1: uh watching going on inside the infrastructure for potential attacks. So it's a nice easy one to turn on. It doesn't cost a lot extra. I don't think it costs any extra on the Amazon infrastructure. And the last one is actually kind of critical to being able to deploy containers effectively onto AWS or any environment. And this is the service manager property store. In that property store per environment, we can put roles so that when these containers start, they are fed environment variables to configure them. Each image that we build is the exact same image that we use on sandbox, staging, and production. They're completely immutable. Nothing is written to that image. Try to follow the 12 factor

Speaker 1: app as much as possible. So if you aren't familiar with 12 factor act, 12 factor app, go look up the 12 factor app. There's a lot of really good design patterns there. And one of them is to have your applications log to standard out. So that's how we can get our logs into CloudWatch. And the other ones is going to be using environment variables to configure the application. So things like secrets for whatever environment is supposed to run. So things like which Django settings file to run, what Redis connection string to use, what Postgres connection string to use, all those secrets are not inside the container. You obviously want to protect those secrets. You never want to put them into your version control And you also never want to put them into your image repository because it poses a security risk. If someone could get a hold of image, they could get a hold of your database secrets and then start harvesting, you know, obviously data out of your database

Speaker 1: would not be a good thing. But the parameter store secrets allows us to at runtime for these tasks define which environment variables are passed in, and then we use the Django environment. package inside of our Django environment to read those in or use specific defaults that are are set up. And that that works actually for for production deployment, def development deployment or production, sandbox, and staging deployments, but it also is useful back here in the developer experience. You can specify in Docker Compose an environment file, and so we will maintain an environment file inside the repository. that contains developer specific environment variables, but those are the basis for what gets put into the parameter store

Speaker 1: and passed into our Django when it all launches. So there's lots of other credentials that are in there. For example, keys and secrets for third-party services like Slack and Discord. uh keys per environment for century. Now this was a tough one because for example we wanted to be able to make sure our React application can also have the sentry monitoring on it. And so part of that is actually going to happen over here in this code build. The Century one is a little tricky for the React front end because you're going to do a webpack build and you only want to do that once. Like we want to build and push to media storage, but we have one image. We've built one image from you know, the the the original

Speaker 1: code pipeline all the way back here, uh kind of in the in the developer world of things That doesn't that image doesn't get mutated anywhere along this whole pipeline, which is nice for debugging if you have to you know reproduce a a production issue, it's always nicer to be able to pull that image down to your local machine and be able to reproduce it locally than it is to try and reproduce these things in a production environment. So what happens is that during code build, depending on what the target environment is, whether it's sandbox or production, we actually do a little bit of like string replacement inside those binary packages as we push them into the S3 bucket. So the image itself doesn't get modified, but the results that get pushed into the S3 buckets for the React application, for this specific React application, so that bucket and that bucket are the same.

Speaker 1: does get modified depending on which environment it is. And that enables us to get sandbox React application metrics in Sentry separate from the ones that would be coming from production So we have typically a VP this is a VPC inside the Amazon world. We'll have a VPC per per environment, production, sandbox staging, which helps keep those things separate, which also means you can now give certain roles or certain users access to certain environments. Um we can kind of go with a practice of least privilege and only grant folks who need access to specific environments that access. um kind of limits your surface area for you know someone accidentally deleting a specific environment.

Speaker 1: One of the the benefits of like Terraform is being able to reproduce an environment really quickly. One of the downsides of something like Terraform and infrastructure's code in general, it's not specific to Terraform, is it's really easy to also blow it away. Uh if you made a typo and ran plan, you could really quickly see that you could change the state of something that could really know, for example, delete your database uh inadvertently. So it's nice to be able to have the that test environment, that staging environment before you get into production, but it's also nice to not allow the developers direct access to those environments. and have them be more uh deployed on an automated fashion as opposed to uh manual each time where someone could you know fat finger typo something because that that can definitely happen and no one likes that when it does

Speaker 1: I've not seen any questions over in Slack, but if you want to go ahead and post a couple over there, or if it's easier, we can all jump into the Jitsi room or the face-to-face room below. Unless there's any other quick questions, I think I'm going to wrap up and we can go into the Jitsi room directly and talk more uh with specific questions about how things are done 'cause I'm happy to share you know what has been our experience with all this and if there's anything I don't know if there's anything we would change. This has definitely been an iterative process. It didn't start out this big. I think it started out with, you know a couple boxes on the screen and then obviously over time it grows and grows and grows as the application matures and you add in kind of more robustness to the whole process itself. What's nice with this you can Use this process for you know full continuous integration.

Speaker 1: So having your CI tests run here. And at that point you can now enable full continuous delivery. So you could actually make it so that you can do deliveries as many times a day as you wanted. So every pull request can get reviewed, merged, and if they pass on the CI, they can go into each of the environments automatically, or you can still control it manually like we do at the moment, where uh a developer can kick off a specific target to deploy into production or sandbox um once we've determined it's gone through testing and and manual testing as well. All right, well with all that said, I will probably cut it here so I can save time for questions in the face-to-face room because I think there that'll probably be where there's more value at the moment for this I want to thank you all for coming to DjangoCon

Speaker 1: 2021 and I love any feedback you give us on Loudswarm itself as a as a product and we're happy to share any tips and tricks we've learned along the way as part of building this platform using Django. Cool. Thank you very much. I'll see you all over in the face to face. Hello, Mia open up participants. I mean feel free to turn your cameras on and unmute and ask ask any questions you like. I'll do my best against you. Hi

Speaker 2: Calvin, can you hear me?

Speaker 1: Yeah, I can.

Speaker 2: Cool. First off, thanks very much for the for the talk. Thanks for being so open with um how you built your infrastructure. It's really cool to see um you know a live a live platform like that and how you've built it. I'm I'm developing everything on Heroku tomorrow, so hopefully I'll get to that AWS stages at some point. So my question is Something that I kind of wake up in the middle of the night about in sweats is my multi-tenancy. I I build the logic into my views at the moment, okay? And I'm just wondering Which approach you take to the multi-tenancy in terms of your database? Are you using is it shared? Is it partially shared or um is there a specific Django package that you might be using for your multi-tenancy?

Speaker 1: So it's it's a shared database. We didn't do a shard sharded database or uh Postgres has the ability to do like multi DBs per tenant. I've seen both both those. Actually there was a really good talk last year It was either Django Con Europe or it was Python Web Conference. But pretty sure the talks on YouTube about multi-tenancy. So it's kind of two paths you can go down. One which is like the adding a key to the each uh table to say what tenant or what domain it's for. What we did was we wrote a middleware that looks at the request and extracts out of that which tenant is requesting the site. And then that sets up some uh we set that site into the request.

Speaker 1: And then from there, all of our stuff is not, we're not using any views, it's all Django Rust framework. So all we had some base classes in our Django Rust framework that we then inherited all the rest of our API from. So we could basically take advantage of that request and knowing which site uh they're currently limited to. I can see where if you're doing it in the views, you gotta make sure you catch all those spots. Um ours I think our surface area is a little less because we've we're we've made a base class for our Django Rust framework. to enable us to manage that a lot easier.

Speaker 2: Okay, thanks.

Speaker 1: Anytime. Any other questions? Actually, one more thing about you said you were on Heroku. I think that's great. I think really keep it simple until it doesn't need to be simple anymore is a is a great approach. And I I I tell everyone I meet that this is the way to go because you're not managing the infrastructure, you're not managing a whole bunch of like other moving parts. But you'll get to a certain point where you're gonna you're gonna your complexity will demand you kind of move into this next level. And that's how ours grew. I mean ours didn't like I said in the talk, ours didn't start out This big. It grew to be that big because the complexity demanded it.

Speaker 2: Yeah, Heroku was working perfectly for the For us at the moment, it's a team of Wonder Cooper, which is me, and the workflow is really simple. I make some changes in my local Docker environment. push a git up to my repository and then the workflow just kicks in and it pushes into my stage environment and everything is really For one developer to be able to do that, it's really it's really nice without having to um worry about my AWS infrastructure. So yeah, Roku is is is working very well for us.

Speaker 1: Right Cool. Any other questions? Or it could be at anything. You could ask me questions about the talk this morning.

Speaker 3: I have a quick question.

Speaker 1: Yeah.

Speaker 3: I was wondering how do you manage to keep the documentation up to date because um you said you started uh with a small infrastructure, but it's keeps getting uh bigger and bigger and uh you also get new team members um in the development new new developers in your team Uh and I was wondering how how um you keep everybody up to date. Um and how how can everybody wrap up and um know what's going on inside the project?

Speaker 1: That was my big uh push behind using Docker Compose for getting started. Um, because it used to be on Older projects, you know, you'd have to have a specific version of Postgres installed, specific versions of Redis installed. And so there's a lot of developer documentation on how to get set up and going on a project. But with since we moved into using Docker Compose, I tried to make the developer onboarding experience pull this repository Run these one maybe two commands and now you can start editing code. So most of the time it's Pull the repository, you do a make build dev, and then make start, and that should actually have you the whole thing running uh locally. It would be would be the goal.

Speaker 1: So there's obviously less overhead and less cognitive load for a developer to get started. Now it takes more time to get to that point because you've got a lot more things to think about, but it it pays off as soon as you have more than another one other developer on the project. As far as other documentation, you know, we mostly maintain just the README in that repository for how to get started. And then we try to replicate these same practices to every project we're working on. So we've set up a cookie cutter um repository for setting up new Django projects to try and f make sure we're always following the same practice in every project that gets started. So it's kind of another a way to assist. So I wouldn't say we're writing a bunch of documentation that are narrative, but we're trying to put in place like the practices so that it's just a reflex.

Speaker 1: And it just works the way you think it should work. Kind of like Python. I love using Python because most of the time you can guess what some method name may be on some internal built-in because it just follows a sensible pattern.

Speaker 3: Cool. Uh and uh how about the um AWS services that you're using? Um do you manage the all all of them externally? Um or are you uh also interacting with the UI of AWS?

Speaker 1: Oh all externally.

Speaker 3: Okay. Okay.

Speaker 1: Yeah, every last bit, even down to so we use multiple uh organizations or accounts inside of an organization structure. So every project has its own AWS account, but it's all grouped underneath a hierarchy of accounts in the AWS environment and that's actually controlled by Terraform as well. So we have a separate repository for the overall like say six global six feet up uh AWS structures and loudsorm is just one small one inside of our existing infrastructure and then on that whole large thing is one repository and then it produces the count. At that point when we deploy our code The terraform that's inside of the Loudswarm project itself is all specific to just that one account. So the developers have control

Speaker 1: at that level for maybe they need a new service. you know they're adding an Elasticsearch or something else, they can do that now without having to go ask a Sysmin or somebody to to provision it for them. They can they basically can add it. They have the control to add it via Terraform. and then run the Terraform apply and get that that infrastructure for them to use. But yeah, clicking through click clicking through the console is fine for kind of exploratory proof of concept. But we always go back and replace it with a Terraform uh files to make sure that it's reproducible.

Speaker 3: Yeah, cool, good to know. Thanks.

Speaker 4: Hi, hello.

Speaker 1: Hi.

Speaker 4: Uh I have a quick question. Um uh Do different events share containers or do you deploy different uh set of containers or events?

Speaker 1: It's it's all shared. So well they all share there's one image that that represents the currently running production uh loudsform code. And so every every event's running on one. So i ev every change, you know, any any enhancements, every event gets them, because it's just one set of running containers. And we scale those containers. Like if we have a bunch of events running at one time, uh, we can change the number, like the minimum and maximum number of running Tasks. So in Terraform, or sorry, not Terraform, Fargate, they're called tasks, the containers. So we adjust the number of tasks that are available for the application to use, and then the application load balancer automatically load balances across all of them.

Speaker 4: Thank you. Also, if there is no question, I have another question. Go for it. Just curious, uh how how uh big of a team behind uh um Rollsform?

Speaker 1: The the team kind of flexed here and there, depending on what stage of development we're in. Um but generally it's just a couple developers. It's really not a huge I mean we when we first built it, it was a few more developers we probably had two front end people, two back end people in a DevOps, but in the current state where it's more kind of steady state bug fixing. It's it's not even a full-time developer at this point.

Speaker 4: Oh, great. Thanks.

Speaker 1: I guess we owe that to uh the awesomeness of Django in uh in React.

Speaker 4: No, I think it's it's still challenging, but uh you're doing a good job uh apparently with a small team.

Speaker 1: Well and and to be honest, like our the six feet up team is pretty senior. Uh we don't have a lot of junior developers on on the projects, any projects. So that I think that helps. I hope. Any other questions? Well, if not, I can let you get back to the rest of the conference. I appreciate you all stopping in and uh saying hi. It's nice to see some faces uh during a conference.

Speaker 4: Yeah, thank you. Thanks, Castle.

Speaker 1: You're very welcome.

Speaker 2: Cheers.

Speaker 1: And I'll be and I'll be around today in Slack and I'm I'm here all day tomorrow too. So if you want to catch me tomorrow in Slack. Feel free to ping me or find me and gather. Uh we can chit-chat. Happy to do that. Awesome.

Speaker 4: Do you have a boot

Speaker 1: We do. We do. There's there's a six feet up booth and there's also a loud swarm booth.

Speaker 4: Okay, I'll drop my later. Thank you.

Speaker 1: Yeah, or you can find me on Twitter or after the conference. I'm not hard to find. I'm the only Calvin Hendrix Parker there is in the world. Cool. Well, I will talk to you all soon.