Automated Data Loader: Wagtail for Weather Data Integration

Speaker 1: Right, let's go. So this next session features two speakers from the World Meteorological Organization, Eric and Grace. And they'll discuss their automated data loader, which solves the challenge of unifying differently formatted weather data from stations across Africa. Over to you!

Speaker 2: Thank you so much, Lisa, and it's a pleasure to have you all join our session. Welcome to WordTail Meets Weather. And when you talk about weather, let's think about it for a bit. When cyclones threaten Mozambique, and when heavy rains threaten Malawi and Togos lowlands, and when torrential rains soil up the Volta Basin in Ghana. And when swamps flood up the Ghana River. And when rivers rise in Zambia And when power outages threaten weather stations, the automated data loader ensures that weather observation data reaches regional forecasters in real time. Our talk today will focus on how we used

Speaker 2: Wagtail to unify the diverse gaps in weather data transmission in Africa. We developed a tool called the Automated Data Loader. Next, please. My name is Grace Amondi. I'm a WebJS developer together with my colleague here, Eric Oteño, also a WebJS developer. We have a team of five experts. And we are deployed by NOCAP, the Norwegian Refugee Council, to the World Meteorological Organization to strengthen the digital transformation of the national meteorological and hydrological services in Africa. Next. So while we talk about Africa, I think there's a slide missing.

Speaker 2: Maybe go back one step. All right. So while we talk about Africa, currently there is a large number of vendors Setting up stations within the continent. We have a number of manual and automatic weather stations. And while there are many vendors and it looks like an advantage, it also comes as a challenge These vendors transmit this data in different formats and they also store them in different ways. So with this situation, we see a Different update frequencies, different formats like CSV, buffer, GeoJSON, text, different units of measurements for each of the stations

Speaker 2: from Kelvins to degrees Celsius for temperature, or even meters per second to north for wind, or even different time zones. For example, this is a REST API for TAMO. TAMO is one of the vendors supplying stations to Africa. There's another example of the FTP server for CAPS and Micros And you have a third example for the CSV files by Campbell. And this is just a few of the vendors and how they structure their data. Next, please So the challenge comes in here. Meteorologists need to send this data globally, but with the varying formats and the cocktail of vendors, they have to manually wrangle this data.

Speaker 2: They have to convert manually, do time zone transformations, and at the same time, the national systems are isolated. So this makes it difficult to share data globally to global forecasting centers. And with this, we see delays in weather warnings. So with this small 15-window delay that is given to share data globally, it makes having a manual world practically impossible. Thanks, please. So the burden lies at the country level, more so the countries with limited c technical capacity. So as I said, they have to create custom scripts for each vendor to retrieve data from the vendors and send them globally.

Speaker 2: They also have to do the conversions of time zones manually and also the units, and this is all prone to errors So where there is no automation, there is no real-time data. And in the long run, we see a potential risk in delivering early warning services and in the end delaying saving lives. Next, please. So we come with a solution. It's called the automated data loader. It's divided into various segments. At the top, we have the data sources coming in from the different weather stations on the ground. From manual or automatic weather stations, transmitted via different protocols like FTP, directly accessing the databases, or even API formats And then in the middle

Speaker 2: we have the ADL architecture. And the ADL allows you to develop plugins That can now pull data from the different vendors. So for each vendor, you develop a separate plugin for it that is able to understand how the data is structured and push it to the core. So the core of ADL is built on Django WagTe, and its function is basically to schedule and do the orchestration, data normalization, unit conversions. handle time zones, so it does not have to worry about how this data is structured by the vendor. This is already done by the plugin itself. And then while we also have plugins that pull data from vendors, ADL also allows you to develop plugins that push data to receiving or dispatch channels.

Speaker 2: So you can have a plugin to push data to an FTP server or an S3 bucket or even to a WIST toolbox which is WMO. information system for data transmission or even to a climate data management system or even to a mobile application. And behind the scenes we use the timescale DB to manage the time series data. Next, please. So a little bit about the plugins and how they work. Each plugin knows its vendor. On the right, you have an example of the TAMO plugin. So it listens to the TAMO API It checks the format, it's able to map the various data parameters to the ADL standard format within its database. So whenever there is a new vendor within the country, we develop a separate plugin

Speaker 2: for it. And then the core knows all of these plugins through a registry. So the core does not need to understand how the data is structured. This is all done by the plugin itself. So Eric, where does Rugtail come in in all of this?

Speaker 3: Yes, thank you. Thank you very much, Grace, for that very good introduction. So yes, so for the usual uh typical uh work tail uh setups, usually have the pages as the central sort of uh focus. So you work with pages, you add content, you allow your editors to update, fill um by using the forms and so on. But uh looking at this uh page, this is the ADL interface. It looks a little bit different from the typical uh work tail in the sense that on the left side here we have custom menus. So we have sort of uh remove the default uh menus you are using uh you are used to seeing the pages, seeing the site uh reports and so on But here we have put the domain

Speaker 3: specific uh models and added the menus. Looking at the summary items, you'll see it looks uh different. So this is also custom And then the home page panels, you realize that they are custom. So usually you see the history of maybe the workflow, pages awaiting moderation and so on. But here is different in a way that we have taken Wagtail. We are no longer focusing on the content, but using the admin interface. and all the good things that come with Wagtail to be able to build custom dashboard that address a specific use case that is different from developing typical uh websites So the first thing we did is to look at the menu and how we can customize it to fit our case. So

Speaker 3: we use hooks a lot. So looking at the constructing the main the construct main menu we're able to hide the default um menu items and then we bring our own using uh Model U sets as we'll see in the next examples. But this is what you get. So usually you have these default menus. Now by bringing our domain-specific uh entities, we're able to Create a custom menu. The next thing is to create the summary items. So by using the construct home page summary items hook, we're able to customize So the first thing of course is to hide the functionalities we are not using, like the pages. Usually you have the page, images, documents.

Speaker 3: So in this case, we are not using that. So we hide the default ones and then we bring in our own specific entities so we give like a summary of the things that uh ADL focuses on. So uh we needed uh when we were looking around for a framework we could use to be able to develop uh this kind of specific use case. Waxcel provided most of the things. It checked most of the functionalities. We were looking at an admin interface. So we wanted really to focus on uh building fast because we needed to hit the ground running we did to test this very fast we had options we could have developed uh API endpoints, develop a single page application, but that will have taken time and we wanted to use some of the things that come are better

Speaker 3: included from Django and Octale So the very nice admin, that was the first uh attraction, the permission systems for the admin, the hooks, you can you are you can already see we're able to use hooks to customize most of the sections uh we want. The model admin, we wanted a way we could uh quickly register our models to the admin. WorkTell provides various ways of doing that. The form framework. We do a lot of uh filling in of uh information to customize to configure the stations, sending of data and so on. So the same way we were making use of the pages that was giving you all this nice validation and the nice features that comes with work

Speaker 3: tail forms, we wanted to be able to use those ones to continue or rather to improve. Provide the functionalities that we are used to. And of course, once it's being very a very mature framework up to now, we thought this is a very good time to test and see what we can do beyond just the normal content. So the first thing was of course to bring in our models for our specific entities. So usually you use snippets if you have like custom models that you want to use Around Wagtail. But there's also you can bring your own models, you can make it very custom, you can use viewsets. Initially we were using the the separate package, Wagtail Model Admin But after introduction of view sets, we quickly migrated and started using view sets and we are using them all over uh ADL.

Speaker 3: So for example, this is a station view set. Some model uh we extend the model view set and then you are able to do All of this customization. Another nice feature is having the views and the chooser viewsets. So this way we could have these nice choosers as you can see from here All over when you're using foreign keys, we are able to use the same way you can pick snippets in your pages. We're able to pick our custom models when uh administrators are choosing and linking uh tables across the system. Uh so how do you bring plugins to the system? So we have seen plugins are the modules that allow you to connect to stations. And bring the data to the system. So the first thing, of course, is to extend. So we provide like a base, a base of uh base

Speaker 3: classes that you can extend. So if you extend these base classes, then we have a function Uh we have a view that will look into all the subclasses and automatically generate view sets for these uh models. So this way The plugin doesn't need to add anything to the admin. As long as they are extending the base class, which is a polymorphic model, we use Django Polymorphic, they are automatically able to create dynamic uh view sets and add this uh uh add the interfaces that the plugin exposes in terms of the model. And then the plugin also registers itself. So we have a custom registry inspired by Wagtailux that keeps record of the functionality that actually does the work.

Speaker 3: So this is how we are able to integrate. So let's say you have a new type of vendor, you want to add it to ADL. So we develop a plugin following this contract that we have. Extend the base class. You don't need to add anything to the admin. Install your plugin to the core and then automatically it's added to the admin using these dynamic viewsets. So, this is how an example will look. So, if you add a connection from Tamil, uh you will have it added to the system. Using Django Polymorphic, able to allow you to when you add a connection, you are you asked which one do you want to create depending on the plugin So all of this gets nicely added to the admin interface. This list will continue extending as more plugins are added. And then the form.

Speaker 3: itself that allows you to configure the station and its connection and its configuration. It's nicely rendered using the field panels. If you have custom widgets, all of this is done automatically when the model viewset is registered. Um then you link the station so the same concept here we are using uh inline panels this is this have been very useful for us This is a widget that you can use to so it remotely picks uh the station. So it's a custom widget in in short that is able to pick the station so that you have this drop down that you can select from a remote API. So this one you implement it in your plugin but it's automatically added to the admin.

Speaker 3: I think that's the main uh the main note here. Uh we also have a custom data viewer. So A menu is added on the admin side. This gives you a very nice way to interact with the data that has been uploaded. And this is built on top of View, Vue. js. So we are also able to um bring in these React uh or JavaScript frameworks like UJS, React React. js, and all these others. So by using some uh Uh some packages, uh this one called Django View. js utils. We are able to plug in into Actel, extend the templates, uh the generic templates a little bit, and then we have uh View. js components added to the page without uh

Speaker 3: you know findling so much with uh the package management for node and all that setup So it's a very interesting uh setup that we are using here. Uh so beyond bringing data in, we also have a way of pushing data out. So ADL, its main purpose is to pull data from all these varied sources. but also give you a way to push data out. So by using uh same approach, polymorphic models that you can extend, we give you a base class. If you extend it, you implement the contract. of sending station data. So here you can implement uh whatever uh uh channel that you want to send to FTP S3 and so on and this will automatically be added to the admin Uh another interesting feature we have is we are using Streamfield

Speaker 3: beyond content. So usually when you use stream fields you use them to uh to make these small things like tables, uh, you know, like uh charts and so on. But here we are using Stringfield to store the configuration for quality control of of of uh uh this data. So for example, here we have a parameter temperature. So you know this data set sometimes you have to have a limit of how low or how high it can go. So for example, yeah in this case, we are saying that maybe temperature should not go be beyond zero. We know that this area should not go beyond zero, like the minimum and the maximum should not go About 45. So by using a swing fields, we're able to put this configuration in. It's an example of the structure block we are using.

Speaker 3: And then we have we group them all here together. And then uh so we have we limit how many you can have. So you can have a minimum of zero. So that means you don't want to have a quality control for this uh particular data parameter but if you put a mark uh we put one which is the maximum allowed it means you have enabled This particular uh check that you added. So you can add all this, and this gives us the flexibility to bring in new one. When it comes to the actual implementation, so we use the block type to filter the different uh um. You can see how we check which type we have, then we pick the actual validator, then we pass it. We pass it the parameters that it expects for each type of block.

Speaker 3: So this way Instead of just using the string field for content, we are using it to pass around configuration for that control some sections or some components of the system Uh so uh in summary we were picking uh some of the things that WagTail provides and some of them which were very important to us. especially the hooks. So this is really uh allows us to plug in into different extension points of worktail. We use the model cluster as you saw to build uh the inline bar uh inline panels this is very important when you want to have this nice UI that works with foreign uh tables and so on. The permission system important

Speaker 3: control who accesses this data string field for QC configuration and so on So what we brought in uh timescale db an extension to Postgres that allows us to control the time series data, celery for ex orchestration. Uh we do a lot of um scheduling. So um uh how frequently should we check for new data from a source and so on? And then pint for conversion and handling all this data conversion from one unit to another one. So in short, Wagtail gives you 80% of what you need especially when it comes to interface and quickly prototyping and building uh this kind of system, but you build your own domain specific functionalities.

Speaker 3: So I'll invite Grace to talk about uh where we are currently the deployment.

Speaker 2: So within the short period that we have already been working with the countries just this year, we have seen quite tremendous results. We have over 10 AWS vendors, automatic weather session vendors that have been integrated. We have over 500 stations being actively monitored. Millions of observations processed monthly and 99. 9% uptime for operational deployments. And for the first time, a number of countries are sharing data globally, like Chad and Togo So the map on the right shows the progress we are with, the blue in development, dark blue operational, and some of the countries that we plan to work with. So the software is installed at each country level

Speaker 2: and then we guide them on how to customize and also set up the vendor-based plugins for their specific ADL instances. So before ADN, meteorologists had to manually download CSVs or buff formats or GeoJSONs, convert the units upload the data to the WIST tool which is WMO's information system hourly and this was days of work. But now with ADL the process is automated And the data reaches the WIST too and globally within a short period of time. So we have learned a number of lessons, among them being that uh it's important to have a contract, especially for the plugins. So

Speaker 2: we notice that if plugins are developed by let's say 10 different developers, 10 plugins developed by 10 different developers, it reaches a point where things start to break very fast. So it's important when contracts are developed that everything is kept minimal and strict and that plugins are basically meant for fetching and returning the data to the core. And then the core does the smart work of orchestration, data transformation, and pushing the data to dispatch channels. And the result is that the plugins developed in the past will continue to work in the future. And then at the same time, observability is not optional. Whenever a station does not send data It will mean that either the station, the plugins

Speaker 2: failed to fetch the data from the station, or there's a network timeout due to slow internet, or there's an issue with unit conversion, or the station is offline. So the beauty of Wagtail Admin is that you are able to now build a dashboard which Eric showed that allows you to see a summary of the 24-hour activity per station. So this allows operators to spot any issues that come before even users are able to report about any missing data. So the key takeaways that you should learn from our presentation is that one, the Wagdail Admin is not just a CMS for building websites It has more beyond. It has power permission systems, view sets, model admins, hooks, and it's powerful with these blocks.

Speaker 2: You can customize Wagtail for almost anything. We didn't use the pages for WebTech, but we still got 80% of what we needed. Also, the plugin architecture allows for community and contribution. We developed the core and countries can now develop plugins that extend the core. And at the same time, ADL is not just a side project, it's a critical infrastructure. such that it ensures that observations reaches regional forecast in near real time and thus work tail powering life-saving work So with the Django and Wagtail, we can handle mission critical systems such as scale. Don't underestimate what Wagtail can do. Thank you. So we do have resources.

Speaker 2: We have the open source code for Wagtail, for ADL, sorry, and also the documentation. Feel free to scan the QR code. It will lead you there. And then you also have the open source code for a few of the plugins that we develop. So just feel free to have a look at them and feel free to also install and test them out. Thank you.

Speaker 3: Yeah. Thank you very much.

Speaker 1: Absolutely brilliant. Thank you both. There were such amazing quotes. Say just pulled one out in the chat as well. And also the life-saving work of Wagtail. I think that's another amazing quote that we can have. We haven't got any specific oh we might have a specific question now, but I just wanted to tell you how many positive comments there have been as well in the chat about your creative use of Wagtail but also your fancy customization that everyone is extremely impressed with. So thank you very much for sharing that with us. Sebastian has asked, what are you missing from Rack Wagtail as a framework or what would you be changing to facilitate your tool?

Speaker 3: Um I'll say most of the things that we really wanted to implement, we have added them. Uh maybe the one that took time was bringing in like this uh uh uh frameworks for JavaScript and tooling. I know this is not a working problem, but uh More of like integrating these frameworks with JavaScript. There's no one single way of doing it. There are multiple ways of uh Doing it and depending on your level of expertise, this could take you time, but nevertheless, we managed to do it. But so far, so good. Um I think the the the thing is also that we have uh an experience with work sale. We have used it uh extensively for other projects which are now website-based. So We

Speaker 3: can find our way by digging into the code and you know looking at the documentation. But yeah, so far so good. I think we are comfortable with what Wacta provides us. What now?

Speaker 1: It'd be good to have you back for another event as well to hear how it's going in six months or a year as well. I think that's all the questions, but will you um please be able to share your slides? in the lobby so that people can have them and then they'll be able to find those QR codes and resources that you shared at the end as well. Thank you very much everyone for coming. You've got a bit of time to be able to make it to your next session. Thanks, Eric. Thanks

Speaker 3: Grace And K run by

Speaker 2: button.