Predict Lightning Strikes using Django and AWS

Hello DjangoCon 2022. My name is Calvin Hendricks Parker. I am co-founder and CTO at Sixfeet Up and I'm excited to be here to talk to you today about how we used Django and Amazon Web Services to help predict lightning strikes. Really interesting and actually one of our impactful projects. And so if you've heard my intros about being impactful and in and in intentional about the kind of work you want to do. This has been a really awesome project that we worked on with a company that came to us with basically a a unique problem. Let's just we'll dive in and I'll describe some of the problem to you so you can kind of get a feel for what's going on here. The initial problem basically is we don't know where lightning is going to be,

at least not until recently. All of the predictions that you currently get about weather and thunderstorms and things like that are based on where lightning is right now. We have lots of sensors throughout the world that can tell us exactly where lightning is striking right now at the exact moment. So actually if you look at this diagram right here, this picture. You can see the the algorithm is showing uh lightning, potential lightning uh in a specific area. Uh that's actually predicted lightning. So lightning predicted. like 25 minutes before uh it's actually going to strike in that spot. So this was actually a situation where there was a jet skier. in Tampa Bay who was out on the water and actually got struck by lightning

uh at 3 37 p. m. Now if you look at the algorithm here, you can see that the prediction algorithm is actually showing that the lightning will strike at 310 in those this this kind of area leading up to where the lightning or where the the jet skier is. And by 313 the algorithm is telling us, actually if I move out of the way over here, you can see the lightning is going to be right on top of that jet skier right here. uh which if you look at the time that the the lightning struck at 337, you know we're getting a full twenty-five minute ahead of time uh warning for that jet skier that they should have gotten off the water, which was Sure, it's certainly more than enough time to ensure that that person was gonna be safe. The initial 911 call again three minutes later

at 340 for that person who got struck by lightning there in Tampa Bay. So you can see that being able to predict this with enough time to actually act on it can save human lives, can do lots of interesting things. the the algorithm has a lot more um applications other than just you know saving human lives on uh jet ski or you know little leaguers playing um baseball Really the applications extend way further. It's what's more interesting is you think about uh things like airlines when they have to hold up air traffic because there's a storm coming through. The smaller they can make that window that they have to hold up to air traffic, the less revenue they lose

and the less cancellations that kind of ripple through the whole system. So if they can do anything possible to know when and where lightning's going to be, they could do things like minimize those windows, minimize cancellations, minimize all the delays and all that kind of ripple effect that actually happens. as part of scheduling all the airline flights. Or another example of using this is is gonna be the insurance industry, where if they knew lightning was going to be in a specific area, they could de-energize uh equipment and saving you know m potentially millions of dollars a year in uh claims when they could have actually prevented the damage from actually happening. uh instead of having to file a claim and and pay for that out of the pockets of their uh the people paying the premiums. So

again, really interesting that so what was the real problem? Why why are we involved and why is Django even involved in in all this as well. So the customer, uh in this case Jason Deese is a research meteorologist. uh for the National Weather Service in the Atlanta area. He he came to us with a problem where he had built this algorithm in a Jupyter notebook and so it ran on a single laptop obviously not optimal for deploying and running in production, but that laptop and the Jupyter Notebook weren't exactly fast for doing these kinds of inferences for the prediction of lightning I think it was taking around 90 seconds to get a single inference based on a GPS location. And also the the notebook wasn't able to in real time

ingest a lot of the data, which we'll talk about here in a minute that is required to be able to predict where light when and where lightning will be. So we basically got tasked with implementing a cloud native solution capable of this low latency request and response lifecycle. to productionize this idea. Take something from completely a scientific idea and make it run for real up in the cloud. The real challenges though come in around the data. When looking at the kinds of data sets that are going to be needed to ingest and the frequency with which you have to ingest that data to make sure you can make these kinds of accurate predictions. we have to be able to have a s some kind of a pipeline or system for that. And this is where the initial ideas came from to really make this very cloud native

so that we only paid for what we're using. we're having to run a lot of machines full time because some of the data comes in pretty sporadically. Things like the NextRad data, which is the radar data you typically see when you're watching the weather um online or on the TV That kind of data comes in right now. We get archived data every five minutes from uh all the US uh NextRad stations. So there's about what 159 NextRad stations and 49 uh TDWR stations, so all feeding into an S3 bucket. Uh Amazon Web Services has some of this NextRad data available for you to use as part of the some open data initiatives that they have. But you've got about 500 gigabytes of data per day that you ingest, and we're ingesting it on every five-minute

basis. Another dataset used in this application is called Wrap Data. Um I can't remember what all the acronyms mean necessarily, but that data is about 450 gigabytes of data per day. And that's generally been ingested about once an hour, or whenever that becomes available to our system to use. And so this is also not including that we're adding in new data sources continually as they start expanding into other models that they're going to be doing for prediction. uh things like tornadoes and other other areas for weather prediction that we can now be predicted faster and more accurately and more ahead of time. Oh yeah, rad route uh rad data. Rap data is the rapid refresh uh weather model data, so that's what that was coming from. But basically we had to get the performance down to a level that would be acceptable for people to subscribe to this service.

Um something like below uh 500 milliseconds for a response would be adequate for people to make decisions uh based on this and do it in a way that It was portable. So in this case we chose the GeoJSON format for being able to process and store and transfer and cache a lot of that data back and forth between the application and its users of that data. So again, the decision was made to really focus on cloud native from the start. And I know this talk is about using Django to predict weather, and it's actually possible to deploy Django in a very cloud native way. We really heavily leveraged some technologies in the cloud like Lambda for serverless, Elastacache for caching. So we could offload those pieces from the Django application and start leveraging the

cloud native tooling that's available. In this case, we're using Amazon Web Services for that. So as well, talk about a lot of AWS specific technologies, but in a lot of the other public clouds the same same kind of tooling would really apply as well And we really wanted to make sure that this could scale, be extendable, using full containers for local development and for deployment. We want to make sure that the we can you know basically expand as the needs expand for the um the project itself as well. And so CloudNative makes a lot of sense because we can basically piece together solutions and use these various building blocks and you only pay as they're basically being used. But before we get into the actual cloud, let's talk about the developer experience.

because a lot of the the speed of getting new features into a platform are really going to come together based on the ease of onboarding new developers and how fast developers can iterate through these various uh pieces of the application or building new features and functionalities into the application itself. So if we start on the developer experience right here , we've got the local machine, so you can kind of see that right here, the the Docker container, Docker Compose bit. In this case, the customer is using GitLab for their repositories and their CI pipelines. So code is committed into the GitLab repositories. Infrastructure is code. In this case we're using Terraform to build out and maintain all the infrastructure pieces that are made up of the AWS platform and also the GitLab platform.

So not only can we terraform the AWS, we can also Terraform GitLab and any of the other tools that are in between so that these can be easily redeployable. So to kind of start off with Terraform, typically we would you know, start with the console or the CLI, build up some infrastructure, and then we will start building out the Terraform configuration files and then apply those to the infrastructure. So in the AWS cloud side for development right here, we're gonna have uh like our container registry where we actually deploy the the built images for um In this case, we're doing images for Lambda and for our Fargate instances. So we're fully serverless on the compute side for the Django application. So instead of running an EC2 instance and deploying Django into that EC2

instance. We run Fargate tasks, which basically mean we aren't responsible or we don't deal with the underlying compute that's just managed by AWS and all of its services underneath. So kind of shifting left some of that responsibility for the infrastructure underlying the application itself. And again, we'll also deploy all the median things into S3 buckets. And we're going to deploy those containers into the container registries for Lambda and for the Fargate instances. And that's kind of how the developer instant developer portion of this all works. Again, really wanting to be cloud native and now that we had Lambda supporting containers, we wanted to use the containers everywhere we possibly could.

uh just made for a kind of a one way for building the application, whether we were deploying the Django part of it into Fargate or for deploying the Lambdas which are doing a lot of the data ingest for the meteorological data, we could use the same technology and the same build process for doing all that. And so the there's actually a blog post Right here I'll link to it. I'll throw a link into the Slack as part of the application, but you can actually just go to our blog and search for the secrets of building Python containers for AWS Lambda And but I'll go over really quick, kind of briefly how that's done. And it's going to rely on multi-stage builds in Docker. And so if you're not familiar with that, it's a This is kind of a great primer for it. And this is actually a more advanced use case, but you'll see that the code for it's relatively simple.

So we're looking at a Docker file here for a typical Lambda. There's a base image provided by Amazon, in this case we're using Python 3. 8 for this one. So if you base your Docker image on that specific base image, and uh at this point copy your code into the image and then run that container. You can now upload this into Lambda and your Lambdas will execute with the uh handler kind of the similar you know API that you're used to for doing Lambda, but you basically packaged everything into a single uh container image and deployed all the requirements all as one nice bundle, basically as a Docker image. That works great if you've got a pure Python application, but if your application is doing any kind of scientific

computing typically they're wrapping around some kind of C dependencies to be able to do that scientific piece for existing libraries. So how do you do that is really to do a multi-stage build. And the the key to this multi-stage build is going to be to use the Amazon Linux AMI. They have a an image that you can use as a base image. For building um Lambda Docker images, but in this case we want to compile our C dependency. with its Python bindings in that first stage that's built on top of the Amazon Linux image. And to do that we had to install our developer tools. And the Python 3. 8, whichever depending on which version of Python you're deploying for which version of Lambda, in this case

3. 8, this is the example you would use. And then you can now do your pip install. of your dependencies, including any dependencies that would have C portions of it that are compiled code. Now the key bit here is you got two you see there's like two two pip installs uh right here on the screen. Uh First, actually not a pip install. One's a pip install and one we're going to create the wheels. And so if we create these Python wheels, we can now use those in the second stage of our build to just install native Python wheels that have been compiled for our specific platform and architecture. So any of those C dependencies that get compiled, packaged into a wheel, now can be installed into the second stage where we'll again use that standard Python

base image But in our case we're going to copy all those wheels that we just compiled from the previous stage into this image. And now we just pip install straight from that directory. So you can see that the key option right here is We're going to tell Pip we're going to use this app slash wheels directory. And we've got our requirements. and file. So we use pip tools for all of our install and so the IN file contains all of the base level dependencies for our application. And we're just saying install every wheel that is sitting in that specific directory into our image, and then we publish that image into our ECR or image our container repository for use with

Lambda. So again, the the Python part of this is really similar. We copy our app, but we have our app. handlers as the entry point for the Lambda itself, but now we can leverage uh SciPy, NumPy, uh other scientific tools that may only be written in C and have Python bindings can now be used inside of uh regular old lambdas without having to do any kind of special gymnastics other than we've got a single Docker file that now understands how to build and deploy our Python dependencies for our application, includes C bits in there too So now once that's all deployed, this is kind of the cloud native deployment of our application. You can see right here in the middle are the Fargate pieces. Those are the ones where we have our Django containers deployed.

The upper part of this diagram, right up here, where you see the Nixrad, the NOAA bucket, those that's where NOAA is actually deposited once a minute or whenever the radars do their sweeps is depositing those next rad data into an S3 bucket. We use the events coming off of those buckets. For example, you can detect when a new key has been written into an S3 bucket. We use that to know when we can go and grab that data and process it by our lambdas. So we've got our lambdas right here that are processing the data and shoving that into Elasticash. So down here we've got Elastichash and our Postgres RDS instance. So we use those basically as a shared data layer between the Lambdas and the Django

instances running in Fargate themselves. Uh the SNS topics was mentioning coming from the the buckets let us know when those things are all ready to go. All the containers whether they're running in Lambda or running in Fargate as Django, are using a common library between them that understand how to process and and use that weather data to start building the model and building those predictions. Redis is heavily leveraged in this case to cache the delivery of those results from the NextRad data, the wrap data, and the other data sources that are coming in. But you have to this one caveat though, if you're using Redis as a cache between Django and Django Redis and regular Redis, uh you're gonna have to be careful of how you use that data outside of Django.

Uh so if you're using uh Django Redis Cache. It makes some expectations about how Django will consume that data and how it'll serialize data back into Redis. So you need to make sure you look at the alternative cache classes and serializers that are provided. The stock one is really aimed toward just Django But if you're going to consume and use this data outside of Django, you're going to need to use one of the alternate ones that is a little more generic and meant for use across other applications here. So there's lots of options there, but just be careful. We stubbed our toe in this one spot and I think it's important to point that out for everyone else to make sure that they don't do the same thing.

There's other bits actually in the diagram, kind of S3 buckets for common media, things that Django uses, like the media directory and the uploads directory and content, things like that. Um cloud native tooling, uh actually right over here you can see them the Route 53 for a lot of DNS. We use the SNS or SES for simple email service for sending out any email notifications. Cloud Watch for monitoring the full stack of applications. So that way you have observability across all the various cloud pieces. As you start distributing your application across multiple components like this using Lambda and and Fargate and now you've got Lambdas and running flour and you've got your celery workers doing scheduled tasks. It can be tricky to kind of bring all that back into a common spot

and CloudWatch is built into the Amazon platform. It's probably not the best solution for this kind of thing, but it's the easiest one because it's sitting right there and included in the platform. I mean you obviously pay for what you consume. One last bit over here is we still we use um actually oops no here right here the um SSM parameter store, so there's a way for us inside of Amazon to store secrets that can be consumed by the various images as they start up. So things like uh database connection strings or secrets, the Django secret. You want to keep those separate per environment. So if you've got a staging environment or a QA environment, it's really important to have different secrets for each of those

um pieces that way your your developers don't have to necessarily have specific access to production secrets, kind of a plausible deniability, and just a good practice to kind of give that least privileged access to the platform itself. Parameter Store makes it very easy to do because the the Fargate tasks or any of the if you're using EKS for Kubernetes. can read those secrets and they are encrypted inside the AWS platform in parameter store. And the last one over is over here, which is this icon right here is uh guard duty. So that's just basically a checkbox you can turn on is supported by the Terraform, but that kind of does a little bit of extra layer of security on top of all this, just to alert you if there's any abnormalities or weirdness going on inside the platform.

It's always nice to have uh someone else watching. We're also using Sentry on top of this, so you can see Sentry up there. Um it's nice for being able to trace you know application errors or even performance um errors. We're using the performance and their issues part of the platform. And that has definitely saved us a few times from potentially uh before we get to production releasing a change that might induce uh a performance issue on the platform itself. So the next level of this, like where do we go from here as we build this platform to do more and more weather predictions, is going to be speeding up the developer experience again. When you're dealing with all these various um platforms and all their various like cloud native tools, sometimes it can be tricky to test your work

before you actually get it deployed up into the public cloud, because you'll have dependencies on things like SNS notifications or S3 buckets, you know, there's various pieces that are kind of out there that are hard to more difficult to simulate on a local development environment. So that CICD uh deployment methodology, you know, take adds additional overhead as far as time to getting something deployed from the time that I write the code to it gets deployed into that Lambda container can be a matter of you know many minutes. because of the the image build, the packaging, the deploy, and then the instantiation of like the lambdas. To help speed that up, one area we've been looking into and started to use is a tool called LocalStack. uh which actually allows you to emulate uh

much of the AWS infrastructure cloud pieces at a local level instead of uh having to rely on deploying into the cloud to test any of your changes because that can be very tedious if you need to make a one-line change and then wait five minutes for your change to deploy into the cloud before you can test it to see if it was actually working or not. We really want to be able to speed up the container build process as well. There are additional parts of Docker now that can enable a new caching. If you're not familiar with it, uh there's call it's called BuildKit built into Docker now. Uh if you're on it's relatively modern, but I think most most CI tools are now supporting BuildKit, which will allow you to build images and then use those images as a cache.

locally on your own machine or even in your CI pipeline to speed up those builds so that the build process doesn't have to go through and rebuild the whole image over again. It's also we were interacting with Code Artifact, which is basically a AWS's private version of a PyPy uh wheel or egg repository. And so being able to build those artifacts um and have them available to your local uh This system is actually kind of tricky because if there's a private repository, there are tokens that you need to generate to be able to pip install those wheels onto your local instance. those tokens can expire uh if they're baked into an image and you're deploying that image at some future date and you have to deal with that.

So there there's some new ways around that with um some environment variables and so leveraging again the parameter store for storing environment specific access to the code artifact. It helps basically streamline and smooth out a lot of the developer environment issues that you're going to run into when you start doing cloud native development and you're doing this all locally. trying to simulate some of those pieces. What's nice is the the container image for Lambdas, the that Python Amazon image specifier or provided by Amazon. allows you to run those lambdas locally and so uh the local stack basically leverages that allowing you to run and simulate Lambda locally, simulate um SNS notifications locally, simulate S3

buckets locally, so that you can pretty much run your whole stack of software and test everything end-to-end before you actually get this up into uh a production or staging environment and where it takes more time to debug and and and make a change. The next steps for the whole platform, you know, after building this initial platform to predict lightning Now we've got this fully cloud native product based on the NextRad uh radar data. It's actually uh allowed the platform to kind of go next step. Uh the the next places we're going with this platform are going to be taking in and actually being able to stream that NextRad data into the platform. Right now we're using the archive data, which is about five to six minutes old, but it gives you A whole set of data all at once that you can consume so it takes less time.

Basically do it kind of in batch instead of as you get a new radar sweep. being able to to process that data. The next step is going to be can we take the radar sweeps as they come in across all what is 159 different NextRat stations. and being able to bring in and validate quicker and basically reduce that gap between the time we process the data coming in from the radar or the weather data. to the time that we actually give inferences for uh when uh lightning's going to strike. And then the last bit is going to be around use implementing some new ML ops platforms The team over at Flash Technologies is actually doing some machine learning forecasts now. The initial lightning one wasn't necessarily a machine learning model, but it was a different kind of a weather model

that predicted the lightning, but applying the same techniques that we did for the lightning model to other types of models and allow them to scale that really is an interesting opportunity now due to the fact that we don't have to pay for giant GPUs on large machines to be running full time. A lot of that a lot of that can be done in uh Fargate instances, ad hoc, using Lambdas. Now one one story that might be of interest to folks who are trying to attempt this. Be careful what you choose to put in a Lambda as opposed to what you may put into a Fargate or if you just it makes more financial sense to really spin up an EC2 instance based on your use case. You make sure you do your math and do some of those calculations. You may find that that lambda might be more expensive than an EC2 instance if you are running really long running processes.

Last year or about a year and a half ago, Amazon upped the maximum runtime for a specific Lambda instances. And it can be quite expensive if you start going to that 15-minute territory. So if you got short very bursty, uh intermittent things, land was a good choice for that. Um this has all been awesome. I'm super excited again to been a part of this project. It's really helped us get that next level for Six Feet Up and their Impactful Project campaign of getting 10 impactful projects in 10 years. And working with the Flash group was really a joy. They were really forward-thinking and open to us bringing a lot of this cloud native technology and Django to help solve hard problems like this. If you've got any questions, I'll be around in Slack.

Feel free to drop them in there and I look forward to talking to you all. Thanks.