What in the World? Determining and demonstrating meaning in geospatial data

Speaker 1: Hello everyone, welcome to What in the World, an intro to Geojango. My name's Tyrell Dennison I'm director of engineering at Lofty. Lofty is a software consultancy in Fayetteville, Arkansas. I'm also a regular contributor to the Friday Afternoon Deploy podcast. uh a nice place to rant about all things software that we've created as a byproduct of our time working together at Lofty. You can email me at Tyrell at HigherLofty. com or find me anywhere socially most of the time at at Tyrrell Dinison. We're going to start off by talking about maps. Because who doesn't love maps? They're great. We have them in everything. We have them on our phone. We used to have them on paper. And now they're one of many powerful ways that we can visually represent information.

Speaker 1: as data continues to be the driving force of all things business and the value of different aspects of data. has become more and more known. Geospatial data is something that is really growing and people are really excited to be exploring. And so in that regard we use tools like GeoDjango Thankfully for those of us in the Django world, we have Geojo to help us with that kind of information. To go ahead and read straight from their docs, GeoDjango intends to be a world-class geographic web framework. His goal is to make it as easy as possible to build GIS web applications and harness the power of spatially enabled data. It does that. with Django model fields for OGC geometries and raster data, with extensions to Django's ORM for querying and manipulating spatial data.

Speaker 1: by loosely coupling high-level Python interfaces for GIS geometry and raster operations and data manipulation in different formats, and by editing geometry fields from the admin. That's a whole lot going on in there. A whole lot of acronyms, a whole lot of terms you may not have heard of. Had you just landed on the Geojo page in the Django project docs, you might be confused by all of it And I can't blame you. And so we're gonna unpack all of that in an attempt to make something very complex, very simple. The Django Project has done the best they can. There's plenty of other material out there to get you up to speed I'm glad to contribute this talk to help you do that as well. So let's dive in. First, we have the Open Geospatial Consortium.

Speaker 1: That's OGC. They're basically the governing body that helps determine how we standardize geospatial data, how it should be represented, and what contextual information, such as SRID, will be helpful to make sure this all stays consistent. Secondly, GIS stands for Geographic Information Systems. And if you're like me, maybe the first time you heard GIS was post -GIS. And I assumed it was basically derivative of Postgres. But GIS is is far beyond that. GIS software and tools. are far outside of Postgres and Python and everywhere else, but they are a system designed to to capture, store, manipulate, analyze, and manage geographic data.

Speaker 1: So, there are several components to a Geo Django app. First, the PostGIS database that we talked about. There's a process of installing some extensions and things of that nature, which are highlighted in places like the Django docs for GeoDjango that will help walk you through that, some scripts as well And then beyond that, there's a basic app configuration component. So we'll take just a few minutes, dive into an example of what that configuration will look like So looking here at an example configuration file for a Django project, you can see that the database engine is different we use the Django Contrib GISDB backend PostGIS because thankfully Geo Django is a first-class fully supported uh

Speaker 1: system within the Django ecosystem So you get that just by making that kind of change. Also in the installed apps, we need to be sure to include Django Contrib GIS. And with that, you have the foundations for a Geo Django app. Again, your Postgres database will need to have the appropriate extensions installed to make it a PostGIS database. But once you've done that, your app is basically ready to go and start coding on and running. So let's talk about geospatial data. The data that you get from your clients or from other data sources. Well could come in a variety of formats. So we want to look at those.

Speaker 1: First and foremost, according to me is GeoJSON, mostly because JSON is everywhere these days. All SPA apps interact with it, all REST APIs interact with it. And so most of the modern web is ready to interact with JSON. So GeoJSON is just valid JSON that is geospatially um aware. And beyond that you have geo packages, which are actually SQLite databases and they are stored as a file. The great thing about geo packages is you can actually do queries on them as they sit. So Geo packages are less common, but they do have the advantage of being able to be manipulated some at rest.

Speaker 1: So that's a lot of fun. And then Unfortunately for most folks, we have shapefiles, which I like to call the Internet Explorer of Geospatial Data File formats. They've been around forever, they are ubiquitous, they are used, loved, and hated by everyone who's done much work with geospatial data. Because fun fact, shapefiles are not actually one file. They are one, two, three, four, and potentially five files all in one. You've your dot shape, your. shx, your.dbf, your. prg, and potentially an XML. Thankfully all you have to do is point to the . shape file. uh which references all of the other necessary information and

Speaker 1: the any any tool that you're using to parse that information will be able to uh extract everything else. So point to your shapefile. Ideally they are in the same directory or they're all zipped up and ready to go and can be extracted from that So let's all just hope the industry continues to move away from shapefiles into things that are a little more fun. Let's get in to layers. Part of how we explore layers is with tools that are already out there, mostly uh in C. like GDAL, Geospatial Data Abstraction Library. That is commonly used and thankfully we have a Python wrapper for it. So you're not having to write

Speaker 1: C code to use the powerful tools that we have there. Same with OGR, which fun fact stands for Simple Features Library. OGR used to have a meeting, but that was lost, but apparently the acronym was exciting enough to keep So we still get to call it OGR for no reason. And that's for use with vector libraries as well. Lastly, uh we'll be diving into layer mapping, which is how geospatial data is stored, particularly in things like shapefiles. And on that note, let's have a demo. So here in my terminal, you can see that I am SSH'd into a Docker container that stood up on my local machine that has a fully running Geojle instance in it.

Speaker 1: We're in a directory called COVID demo because this app is built around the Kaggle COVID dataset for their COVID challenge. So let's look at what they gave me data-wise and no surprise, it was a shapefile. We have all the various pieces of the shapefile in our data directory and we're able to parse that just by once again pointing at the dot shapefile. So let's do that and we're going to use the OGR info tool for that. So we can see here when we open this file that we have one layer in the file and that is called unique geocoded matches. Let's see what is in that layer.

Speaker 1: So as we dig deeper into that layer, we can see a host of information. The geometry type is listed here as point. The total feature count is 94,000. The extent of all of those points is listed as well. And then a whole lot of other information that gives you basically everything you need to make a model out of. Each one of those features that is listed is basically the equivalent of a row in a database. So we know that there's 94,000 entries and we're going to make a model to pull those into. So as you look here, you'll see that I've created a case class and

Speaker 1: uh it's inheriting from models. model, but I've imported my model from Django Contrib GISDB as opposed to the traditional models import. So that's key because it gives us this all-important point field which is our geometrically aware field. We have a host of fields readily available through Geojle. This one, because we saw within our shapefile is point data, we've created a point field on our model. And I've gone in and used the rest of the information I got from OGR to create a model. This is really good to do by hand.

Speaker 1: You can see I've even set max lengths and things like that based on that because it will make you a little more familiar with their data. If you're in a hurry, there's also a tool called OGR Inspect that will auto-generate a Django model for you based on the shapefile that you parse with it. This is the output of one of those that I've preserved here. You can see the actual model itself and what the mapping that they used is so that you can confirm that Once you've created a model file that is uh uh a mapping of your geospatial data. You want to ingest that. I've created a load script to do that here, something that you can do as well, in which I go in and give the mapping.

Speaker 1: point to the file and consume it using layer mapping which is once again coming from our Geojango GIS utilities tools. Now that we know how to get our data in and what format it will come in, let's talk about geometry. So there's as I mentioned quite a few out-of-the-box geometry types you can use with Geojo. We've already talked about point data. Point data is simply a lat and a long that designate one point in space. If you have several of those, you can use a multi-point field, which basically is the equivalent of creating a cluster of points that aren't connected.

Speaker 1: When you start to want to connect those points, you get into more advanced features like polygons. A polygon being a square, a rhombus, you name it. I'm not gonna throw out rhombus I'm not gonna try and be a geometry guy. I'm not a geometry guy. Quadrilateral isosceles triangle.

Speaker 2: Do it.

Speaker 1: You can create that Hexateragram.

Speaker 2: You could?

Speaker 1: You could create a

Speaker 2: you could, but shouldn't you? That's the one.

Speaker 1: What's the one? What's the one?

Speaker 2: There's so many.

Speaker 1: What's the Tess the Tesseract. You gonna make a Tesseract. Make a Tesseract in here. Alright. So really When you decide to start connecting the dots, so to speak, of point and multi-point features, we get into polygon creation, which is simply a series of points that have uh a sequence that follows to draw a shape with uh a the same start and end point. Um and then if you want to have negative space within a polygon, you create a multi polygon. So that would be something that would transform a circle into a donut in case you like polygons within your polygons.

Speaker 1: At that point you also can create line strings, which would be what we would create roads with. or things of that nature or skeletal examinations of the interior of objects like polygons. All of that is given to you by Geojango right out of the box. stored and queryable through the ORM features we get with Geojo as well. Now it's time for some fun facts with geometries. SRIDs, those are spatial reference system identifiers, and they're a way to give highly contextual and potentially highly localized information around uh how things should be represented and understood geospatially. So for example The the data set that we are using uses, I believe, for

Speaker 1: three, two, six, that's correct. That's one of the most common ones and most generally utilized. However, say you have decided that your region needs a more specific representation of that data due to the geographic features of your region and they're not well represented by uh 4326. You can have your own SRID or people who generate those things more accurately would create that for the a region And you can specify that so that you can have a highly localized representation of that data to account for the geography of that region Beyond that, we do need to remember that there is a difference between geometry and geography.

Speaker 1: This information is stored inherently as geometries, but we can make it geographically aware. And uh we can also call methods on geometries to convert them to geographies. So uh one way to think about it is All geometric measurements are done on a Cartesian plane, whereas geographic measurements will be done on a sphere because we're on a globe. And so that's one thing to keep in mind as you're writing queries, as you're calculating things. And you'll have a ton of tools available to do that properly. We have a ton of return formats as well. Once that data has been parsed and stored within the app, we can extract it as well-known text, which is a string representation.

Speaker 1: of the information. There's well-known binary, which is the binary representation of that same information. There's extended well-known text, which includes the SRID, same with extended well-known binary, and of course our good friend GeoJason. So let's have ourselves another demo. One of the great things about using Geo Django is We get to leverage all of the power of Django's built-in admin and uh Geo Django has built a lot of stuff on top of that. It's super easy to implement that. All we have to do is go into our admin and register the admin as a geo model admin. Let's see if we can import that right there. And now let's go to admin and see what it gives us

Speaker 1: So on my back end here, I have a list of cases that have been imported using my import script. And we can see each case is within this information here. Let's dive into one. These models rep uh are basically a one-to-one representation of what came out of the shape file. And you'll notice one really neat thing is we have a built-in map right here with a point on it. Let's back it out and see we're exactly in Wisconsin, we are. Excuse me, Minnesota, we are. And we can also, thanks to GeoJango's admin

Speaker 1: tools Go ahead and edit that. Let's say it's actually right here instead. And we can save that. Add a little extra info And there we go. We're already editing geospatial data. We can also use those tools to draw polygons and save them and edit existing polygons and save them. Beyond the great things we get with admin, we also get to use DRF in a geospatially aware format. So you can look here, I have a serializer file pulled up. We're importing REST Framework GIS. and using a geo feature model serializer. To do that, we simply have to specify the geometry field within the model

Speaker 1: that we are serializing. List all the fields we want to include, very standard format, and that will give you GeoJSON straight out of your database. One word of warning. One word of caution, marshalling all of that information into JSON is very expensive within Python. It will take a lot of time. So consider if you have large queries using something else outside of the context of the ORM or the serializer, that's where PostGIS has a lot of really powerful tools for really complicated things. that can be written right into your Python code. So if you see performance start to decrease in

Speaker 1: marshalling large amounts of geospatial data into JSON that's when you should explore different options like the tools you get with PostGIS out of the box. Another thing to make you aware of is how you modify your view file. One thing to point out in this view file is that we actually have a bounding box filter that I've implemented from the GIS aware DRF. And that'll allow us to pass uh points to create a bounding box and query against that. So your JavaScript front end can take that information, someone can click around on a map. Send it back and bring you back all the data that fits within those coordinates.

Speaker 1: So let's take a look at our REST API Here is our case list. I have modified that view file to limit it briefly. You can see over here that I've done a slice on this one We'll look at the full dataset later with the bounding box in place. But you can see that this has returned information invalid GeoJSON. Geometry is added to GeoJSON right here. Explaining the type of feature, the coordinates, all of that good stuff. So this is perfectly formatted, easy to read, and you get to use all the DRF tools you know and love. Over in this tab, we have the unsliced version of our dataset. However, we have a bounding box being implemented in the query.

Speaker 1: You can see the results differ because Here we are in Mississippi and in this other tab, we in various places in China as far as the data that's being pulled back, the only difference being the coordinates in the bounding box So we know that's working and that's the first of many things we can do with the geospatially aware REST framework. There are a lot more complicated queries we can do using the ORM syntax we're familiar with here. Here are some examples. So here I've created a point. and simply pass in the coordinates. I can then use that in a query for multi-polygon contains. And

Speaker 1: in this case I've created what I will call an infection point and said Use that to determine if uh what the impacted city would be. So I am querying to see what cities contain that point We can also do a lot of really powerful queries with the built-in tools we get from the Geo Django's extension of Django's ORM. So here's some pseudocode examples that we'll look at that kind of demonstrate that for you. I've created an impact point, which is just a point object with a Latin long. I've also created a potentially impacted city. polygon which will represent uh the overall space that that city contains. And so I can query and

Speaker 1: create impact cities by saying what Cities contain this impact point. I I have a city query here and I say multi-polygon contains impact point and if that point exists within that polygon it will be returned and we'll know that that is the city that has been impacted by that impact point. Conversely we can look and see if there are other cases within that potentially impacted city by querying to see what other points exist within that polygon. And that's within that's what's in this second line here. Beyond that we can also query for distance and things of that nature, nearby cases. Just straight out of the gate I've specified 7,000 as the distance from the point.

Speaker 1: Unless you specify otherwise it it uses kilometers. Or in the the one below that I've specified miles and shrunk it down to seven miles. There's again a huge amount of queries you can do through the ORM that are given to you straight out of the box. A great place to learn about that once again is in the GIS section of the Django project docs, which is all about Geo Django. So now we want to render this information because it's been proving valuable. So one great way to render that information uh that's Really straightforward is using Leaflet. Leaflet plays really nicely with GeoDjango. You can see here I've created a template.

Speaker 1: I've pip installed it. I've specified the different JavaScript components that are necessary I've got a method to implement it that I call and specify specifically some information that I want displayed on the map with the points. In this case I've displayed place name. So let's take a look at that. Alright, here's our map and you can see if we click we see data That is the data that I specified. Zooming out. And again, this is the sliced data for sake of performance. But we'll use a tool here in a moment to explore even deeper.

Speaker 1: But you can see that that's a very simple template to put in place. That took me no time at all and instantly I've got data on a map which will excite anybody with geospatial data And you can find out all of the things you can do with Leaflet. It's a powerful tool. We at Lofty use another tool called DeckGL in conjunction with Mapbox to do really powerful visualizations. And there is a wide open version of that called Kepler GL that we'll use now to really look at all of those data points, manipulate some of the visualizations to really make it stand out.

Speaker 1: So let's go look here at Kepler. I'm going to go ahead and point it to my API and have it consume it.

Speaker 1: You can do it Alright, this is Kepler consuming the geospatial data in its entirety from the dataset. You can see we've got all the data loaded here on the map, but it's pretty dense. You can't really make much meaning of it, just a bunch of dots. So let's change the way that's visually represented and get a little more meaning out of it So we're going to go into Kepler and we're going to select fill color here and we're going to change how that color is determined to be colored by rank. That's pretty good, but let's get something with just a little more contrast.

Speaker 1: I like this one. Okay. So now you can see that things are colored differently based on their ranking and we end up with almost the heat map that we're familiar with And Kepler has tools for doing those kinds of heat maps. They also allow for 3D visualizations, things of that nature. DeckGL is the tool that powers all of that So there's a host of ways to represent that data once you have it stored, once you've queried to find meaning in it, and once you want to present something. Using these tools, you can make really compelling websites, graphs, things of that nature

Speaker 1: that can demonstrate stuff that It's hard to convey with just some words. They say a picture is worth a thousand words, and showing this data on a map is a great example of that. So as you can see, there is a wealth of tooling around geospatial data within the ecosystem that we all know and love of Django. Hopefully your time watching this video has made you more aware of it, more proficient in it, or more excited to go explore. Thanks for your time. I appreciate you listening.