100 Million Parking Transactions Per Year with Django

Speaker 1: And so for our next talk, uh let me introduce uh Wootur with 100 million parking transactions per year with Django.

Speaker 2: Thank you. Yes, I'm gonna talk about parking transactions. And actually uh we've been doing that a few years at the DjangoCon during coffee breaks. We always ask like how do you use Django and we get that question back and we noticed certain uh people are interested in how we deal parking transactions with Django. So why not take half an hour for everybody Um what I'm gonna talk about is um A short introduction to the company I work for, Monet, because that's the company that deals with the parking transactions. And I think you need some background in parking data. And then I'm going to show you two apps that we built in Django.

Speaker 2: One we call the ETL process extract, transform, and load. To get all that data into our database. And we heavily rely on the Django admin for that. So you're gonna recognize some screenshots. And the other one is the dashboard that our clients go to and they can actually have their view on the data. I'm gonna tell you a bit about some packages that we embrace and some uh examples as well. Uh first a little bit about me. Uh I studied civil engineering at uh faculty uh faculty of civil engineering in Delft I still live there with this happy family and I uh like to go for a run

Speaker 2: and also like my job so a bit more about that Monad was founded around 2018. Me myself, I joined 2015, but my colleagues were already active in parking data way before that. So say around 2007 there's this parking consulting company in the Netherlands. And you might think why parking consultancy? But um Dutch cities are not built for cars uh and yet they're there. Um so many municipalities uh want to mitigate pollution, uh limit search traffic, you know, to find a parking spot. Uh when building new houses you have to build new parking spaces.

Speaker 2: Maybe you could use existing existing car parks. So many of those questions well consultants deal with those questions but they need data. So that's why my colleagues came in and they did data analysis analytics on parking data And in 2012, they made the wise decision to use open source software, specifically Python, Django, and PostgreSQL. So around 2021 we sponsored the Django Con and Django Software Foundation. So yeah, we've been around a few years and then we expanded abroad. Here you see the map of the Netherlands where we're active, but it's 2025 now

Speaker 2: and we're also active in Germany and some other projects. So that's a bit where we are as a company, but what about the parking data, right? Well, probably many of you parked a car somewhere at some point. And um You see some images of uh situations we're dealing with. So on-street parking or curbside parking, I think is the correct term. Is where you just park your car along the street, you go to this pay-in-display machine, you get a ticket that you can put behind your window shield. And enforcement knows that you have a valid parking right. More and more, this is a digitalized uh flow.

Speaker 2: So next to that you see the mobile phone where you can start an app uh start an app with a parking transaction in a certain area. At least in the Netherlands there is this national database that has all these parking rights available, so also enforcement can check if a registered license place license plate is actually valid. Uh has a valid parking right. That's why this car with the cameras comes in. It can easily scan all those license plates and check for uh valid parking rights Currently in the Netherlands it's around 70% of the on-street parking transactions are done by mobile phone. I think here it might be a little less, but it's coming.

Speaker 2: You can also park your car in a car park, or as we say, off-street behind the gate barrier. We get into the difference a little bit more in the next slide. Um but I also want to show you that even bicycle parking is more and more digitalized and a problem in the Netherlands. And EV charging is also becoming bigger and bigger in data terms. That's all that we're dealing with, but for now we focus on just parking your car for simplicity in this half hour So uh like I said, you can park your car on the street and either by mobile phone or at a on a machine. You get a valid parking right, and basically it says you

Speaker 2: the start date time of parking your car, uh the amount you paid, and the duration you paid for. all at once. So say you park for two hours. We're not really sure if you actually use those two hours. Maybe you are done shopping earlier or you take the risk and stay a little bit longer. That's slightly different for off-street parking. In terms of data that's more rich, so we have more detailed timestamp of entering the car park and pay timestamp and leaving the car park, but also more information about the customer. Uh you could just get your ticket from the uh the gate barrier, or maybe you have a subscription, um your license plate might be recognized, or you have a card.

Speaker 2: Uh so we can distinguish um The type of customer. So looking at that last part, the off-street, and to get a bit more down to word or down to what we're familiar with, I'm gonna show you the model. This is the off-street parking transaction model and I'm gonna go through a few things. F say this was before we took the class from uh uh about uh foreign keys so gonna change that later um But one thing I want to highlight here is the Postgres partition model. It's from Django Postgres Extra. And you can see in the post

Speaker 2: partitioning meta that we partition on account ID. All our clients have an account ID and they're only supposed to check their own data and many processes are done on account level. So it makes sense to partition on account ID and Well we've uh heard a lot about indexing uh in the previous talks. And you know if an index is a bookmarker on which page it should be. I would say that partitioning is like a section of your library where all your clients' books are stored. So you immediately know which section to go to. And if you want, you can take the whole section out without. interference of other clients sections. There is this thing though, a

Speaker 2: a bit of a downfall we had. Uh in earlier versions of Postgres and Django, we could not partition on a foreign key. So uh also related to one of the presentations of yesterday we had a shadow uh shadow column called account partition id to be able to do that and we still have a limitation um And you can see that in the mount details, that's a JSON field. Because imagine you go to a car park, you do shopping, at some point you're done, you pay for your ticket and you want to leave, but you realize you forgot the axe. So you go back to the supermarket, buy the X, but now you have to pay again because you stayed a little longer. That's two payment timestamps for one entry-exit

Speaker 2: transaction. And we used to have a separate off-street parking payment transaction model with a foreign key, but that doesn't work anymore since we have a partition table. Although I think in Django 5. 2 we might be able to fix that. So we have this entry, exit, and last payment timestamps. And like I said, they could be no. All of them could be no. It could be that the gate barrier is open by the time you're leaving. for some reason might be uh malfunctioning or what we actually experienced that somebody just drove through it. Um It might be that you don't have a payment timestamp because maybe you drove in and out within 15 minutes

Speaker 2: or you're a subscription member and you don't have to pay. And even the entry timestamp might not be there, although that's a rare case So we have this transaction timestamp that is at least one of the three. And it's necessary for also for indexing, but that's where we all do all our queries. So there's different facilities of course, and I left some things out. Oh, but I do want to highlight the ID. It's a UID field, because we learned the hard way that an integer is not enough. Also a presentation of yesterday that I wish I'd seen earlier. So those um uh parking transactions both on and off street

Speaker 2: Uh say ten years ago we were dealing with uh five hundred thousand a month of those in those two models. Um And there are more other data sources that count up, but that was roughly what we're dealing with. And even without partitioning was doable. But as the time grew we gained more clients and the clients gained more paid parking area. You can imagine that if you have regulated parking in the city center, well people tend to park the cars just outside that border. So you increase your paid parking area, you get the same effect. That's we call the the waterbed effect. So yeah, more clients and more paid parking. Uh it leads to uh yeah, more and more data in our database.

Speaker 2: Except for those COVID years where we have uh big dips. But now we're roughly at um 8 million transactions a month, so yeah, hence the title under the year. How do we deal with that? Is the actual question. I'm gonna tell you a little bit about the ATL process. Again, that's extract, transform, and load. So we extract the data from different data sources. It's not only different data sources, it's also different file formats. So maybe you might say 100 million a year. That's not that much. I guess probably a lot of you dealt with bigger numbers, but the issue here is that it's from so many different sources

Speaker 2: different file formats, different versions of their software. And we're just, I didn't even mention that, but we're with four developers. Still. So we have um I'm gonna show you already we have this uh ETL tool, or you might just refer to it as our admin That deals with all these processes. So you see here some icons, it's uh some machines basically throw an email uh every day once they're done with their data We have database connections, APIs, S3 buckets, or our own S3 bucket that clients can upload the data to. File

Speaker 2: servers, uh, IoT protocols, uh, and even the last icon is showing you that clients can just upload their Excel file. It still happens. So that's how we collect all the data. And then the second icon there is basically showing you that we have some checks and balances applied to the data before we load it into our vendor-independent data model. Sounds fancy, but it's the model I just showed you. And from there we can upgrade aggregate tables are loaded into external database so the client can do uh their own analytics on click view, tableau, or whatever. But we also provide our own dashboard, which I'm going to show you in a bit.

Speaker 2: So, in terms of the admin, you see here a reordered version of our models and tasks. I'm not sure if you can read it, but it's basically all those steps. So it's Extracting the data tasks, get file from remote file servers, HTTP calls, etc. Pre-processing, loading it, transform it with SQL. Uh do some tests and then data output related tasks. What does that look like if we see a process flow here? Uh you recognize the admin, uh but it's heavily customized in uh in a way.

Speaker 2: Actually it's it it's so familiar to me that sometimes I'm not sure if it's uh a feature that my colleague built or if we're on a new version of Django. Um but you see here a process flow uh for demonstration purposes. So it's uh an account of Amsterdam where we Extract open data every fifteen minutes. Um it ran forty seconds ago, um and it failed. So it has uh two uh Task to extract the data if we would click on the edit button we can edit the actual Python script using the request library to get the data Then we load it into the database and in the end we do some checks on that open data. Well the first step already filled, so we get an alarm bell there.

Speaker 2: And the status of the process flow is filled. We can notify somebody by email. We can add comments to this process flow. And more recently we can set the status to in alert. I'm gonna show you on the next slide. It's a bit Sentry like because we had to deal with uh a lot of emails if something would fail every fifteen minutes and we weren't sure who picked it up or who dealt with it. Uh so we Got this alert page where you can see the actual process flows that are in alert as in they failed and there's a rule that every two successful runs it would be automatically closed alert. But it can also assign it to a user,

Speaker 2: which would be one of us. And you see the last output of the task that failed. So this is really helpful for like scaling our company and still be able to maintain all those process flows with just a few of us. And I'm gonna show you a bit about the dashboard as well. Um and like I said, uh those are two Django apps So you saw the ETL process that is actually dealing with all the data, and then we have the dashboard displaying the data to the client. I always have to show you the iceberg because obviously most of the work is done under the water, but

Speaker 2: top of the iceberg is a really nice top. I can show you. This is what the dashboard looks like currently for our clients. They can log in, they have uh certain filters, daytime filters, select many areas, facilities. and then show the transactions the uh the amount or the park duration for example. That's the very basic thing. Export to Excel, uh show diagrams in the dashboard itself. Um And well that grew over time. I think uh we're on uh Amazon Cloud Service now for seven years. Before that it was hosted in uh my colleagues

Speaker 2: I don't know the word. Metacast where your router is. Um so A few packages that we embraced uh along many more um but I wanna highlight them because uh you know it it it referenced the the The talk that we just heard about batteries included and how easy it is to extend other packages. You saw already the Postgres extra for the partitioning model Uh and some custom fields. Uh one I personally really like is Django all-out, uh, because well, we were in a situation where a new client demanded us to authenticate through uh

Speaker 2: single sign-on. And they were on um Microsoft Intra ID and we were like, oh man, that's gonna take us some work. Because not only we have to build that for that client, but we have all these other clients still uh authenticating in another way And we got away with it for some time if we were able to show them that we use rate limiting on false authentication. uh and a few other things. And for those things, we turned to Django Allab. And then we realized how easy it was to implement single sign-on from there. especially with the use case of having multiple clients in your application. So that saved us a lot of time

Speaker 2: and We also got the feedback from the client that we were pretty fast in implementing this compared to other uh other software suppliers We also really like Django Ninja. We use it for most of our internal API calls. You are gonna see some geographic representation that deals with a lot of those internal APIs. A Django RQ for scheduling all the process and offload the uh heavy uh reports that are being built to a background task. And what the last two model translation and a CK editor 5. I'm not saying that these are the best packages for your needs, but I do like to showcase them because combined they can be really handy.

Speaker 2: Back to uh model again. Um so we have here our uh our our a model for our uh user manual uh to be able to have other colleagues also update the user manual uh and not have it like in the project itself Sort of CMS. So you see here the CK editor 5 field, and that's where you can edit the content. You can give the title as part of a section But before we save it , since we use model translations, model translation basically gives you other fields in different languages that you provide

Speaker 2: And yeah, we don't want to make sure we have to translate everything before we can write the model. So before we save it, all the non-populated fields are being translated from the populated fields. And of course afterwards you could check it. And you could do that with HTML as well. So you don't need to worry about the formatting just in the language you are familiar with. The admin from model translation has a really nice tapped translation admin. I wish I found out a little earlier because Otherwise you have like all these models for your languages. And now you have these nice tabs. So this is what it looks like in the admin. We're on English. We fill

Speaker 2: a paragraph uh in English highlighted it as we want and before we save it it's uh translated into different languages. There's also this history button. I didn't even mention it, but that's from Django Revision Reversion. That's very handy if you want to see who updated what and when. And in the front page it looks like this, so the user gets a nice uh instruction manual. But the instruction manual is not the best part of our dashboard, so I'm gonna show you a little bit more if I have some time still. Familiar to what you just saw, this is a chart um uh in our dashboard. It's the

Speaker 2: I think the most used module uh in the dashboard because you saw the off-street parking transaction model. Uh you either enter a facility, leave it at some point, or at least have a less payment timestamp. And if we look at all these transactions, you could basically every hour check how many are there in there at this point. That's how we basically build up the occupancy chart. So for every hour of a facility uh we have a record of the the uh occupancy rate. Um and you see here a plot of uh I think last year to date. So on average, this parking facility is pretty full on uh Friday night and Saturday.

Speaker 2: Yeah, you see my mouse, right? So here. And actually one of those Saturdays it hit the total max. That's this black line there. Coming back to the earlier question, like do we have extra space? Like if we would build more houses in this area, could the uh should we also build more parking spaces or could we use this facility? Well, you cannot use this facility. But there are other facilities where there's plenty of space. You also see this distinction here by the way in blue that's the typical subscription member so either somebody who works there during weekdays or stays overnight as uh people who live there And in green that's what we call short-term parkers.

Speaker 2: The geographical representation is a bit uh Google Maps style. We actually have a Google Maps image there. Interactive, so You have uh the parking spaces, you have the points of sale, like uh the pay and display machine, and when you click on it, you get some feedback, uh some info uh of the total transactions, the average parking duration, and that if you would click on it you go to these reports that I just showed you. Now, with this scan car that I mentioned in the beginning, we also have a lot of data from enforcement that basically checks if you have a valid parking ride. If you would do that

Speaker 2: well two, three times a day, it gives you also an indication that at least a car parked there. Which could help us build up these occupancy rates also on street. So that's what you see over here. You click on a certain street, and you would see that it had There are like 15 parking spaces available and over the measured time based on 12 measuring moments you'd see that the occupancy rate is 88%. Um and this is real uh really um uh handy information I would say. Really It shows like municipalities like uh what happens where and when in terms of accompancy

Speaker 2: rate uh and Do our policy making decisions actually affect what we want them to do Or should we change some things? Especially if you show it uh like over a day. This is one of the biggest big five cities in the Netherlands It's pretty green if you look at the Aquancy rate over a whole city in the morning, but then in the afternoon, as people get home from work, it gets busier and busier in those streets. Wrapping up Why we use Django

Speaker 2: and well continue doing so? Um it's very fast and has a steep learning curve. So uh we're just with the four of us and we are still able to maintain uh everything uh Also referencing what was said before, I mean Django doesn't break things. We heavily rely on that. It's flexible and customizable , and we have many documents and examples available and all you guys to listen to what we should do with foreign keys. And other things that we really appreciate. So yeah, that's basically what I was gonna tell you.

Speaker 1: Thank you for your talk. Um before answering the on-site uh questions we have one uh remote audience question. Um brilliant talk, thanks. What was the primary driving factor as to why you built this in-house using an edited admin rather than using a third-party tool? Or even just developing a separate tool that wouldn't require editing the admin so heavily.

Speaker 2: Uh so why we use the admin instead of a third-party tool or yeah. Well, first of all, it was really easy to start with. Without a lot of work, it got everything we need. And It's fun also. And we yeah, we were able to maintain all the needs that we got through the through time by customizing it and or waiting for a new version that actually had what we need. So I remember my colleague told me at one point that uh he said I really extended like how much we could do with customizing it. Uh so yeah we had to look around at some point but uh well as Django

Speaker 2: grows and uh and we grow we I think we grow along so there's no need to go to something else

Speaker 3: Thank you for the excellent presentation and the system looks great and I believe we can all of us can learn something from it. Uh I have two questions. Uh one of them is why uh ninja uh Django Ninja instead of Django framework. And don't take me wrong, I'm not disagreeing, just curious about your reasoning. And the second one goes in line with the previous question, why using the Django admin? Because eventually when Django upgrades it can break our customizations in admin. Uh my idea is that normally using Django Admin as an application is not a very good practice.

Speaker 2: Well I disagree on the last part. But the uh well why we uh uh you had two questions. The first question was Oh yeah, Django Ninja, yeah. Uh we used both at some point, uh because uh the Django Ninja came later and uh Well, my personal short answer would be it it's it's way easier and and faster and uh easier to maintain But uh I would have my go have to ask my colleagues for more specific answers if you're uh after that. Um and Uh why still using the admin and and customizing it so much?

Speaker 2: I think you have to, I mean you probably know that the Django admin allows you to customize many things as well. So it's not customizing it is not necessarily meaning that it's gonna break in the next version. Uh and as we found over the years like the the customizations some at some point might be integrated in uh the Django admin itself or in the third party app so we could move it out again but some customizations still live there for a long time Like I said, something sometimes I don't even know if it's a customization or if it's part of Django. So yeah, we we don't see Many problems in that actually.

Speaker 4: Thank you. I I actually quite like the idea to customize the Django admin. Um I was wondering whether you uh did a study or evaluation when you were doing that uh comparing it to for example Django CMS or Wachttail. Django CMS or Wachttail or the Django admin and why would you go this way or the other for the use cases you had.

Speaker 2: Um yeah well by the time we started customizing the admin uh we have never heard of Wagtail. It was only recent for us. Uh and I think at at this point I'm a bit like my previous answer, like at this stage, there's for us no need to change. Uh I'm not sure back then if we did a really hard study on what would be all the options, it was just very easy to start with the admin. Uh and we do ask ourselves every now and then, is it still a good idea? But uh Yeah, like I said, it's it's still makes that the four of us can maintain this whole project. So we like to keep it that way. Oh, we're always happy to have more employees of course, but still maintain the project uh easily.

Speaker 2: Thank you. Thank you.