Building maintainable Django projects: the difficult teenage years with Alex Henman

Right, uh morning everyone. Thanks for thanks for having me here. Um as you might be able to guess from my accent, I've uh come a little later to be here, so happy to be speaking here and Yes, going to be talking a bit about kind of my experiences working on kind of Django projects over the years. A little about me first though. So I'm the head of product engineering at Bohurst. We're a company based in the UK, based in London, providing data on high-growth companies in the UK and Germany as well recently. We sort of expanded our markets a little bit. We use Django really heavily as part of kind of everything that we do. So it's the core platform that we have. It's also used quite extensively for like our data engineering pipelines. So we sort of go quite hard on the ORM in particular. Um, this is my first DjangoCon

US, but I have been to a few Django Cons before, so it's my seventh DjangoCon in total. My second time speaking at one. Um I do it mostly because it scares me rather than 'cause I enjoy it. So um I'm finding it, you know, sufficiently terrifying right now Um for a bit of um something else about me as well. Um just you know bit of bit of character reference. I very much enjoy a nice beer at the end of a long walk. Here's a picture of a very delicious beer that I had earlier this year in Brussels. Um would recommend. Bit of context though for the talk and why I'm here in my I think I maybe have a perspective that you might be interested to hear something about. Not right now, actually.

I've um been working for um something like 12 years now for my current employer. It was my very first job straight out of university, and I've been there ever since. And so I've been working on Django projects for pretty much the extent of that. And sort of one major part of that, the sort of core platform that we sell to our clients, I've been working on for something like 10 years now. So I've seen a code base develop quite a bit over those years. And so this is probably a collection of some kind of hopefully interesting tips and tricks, probably to get around some uh things that you know maybe develop in a code base as it ages which are not necessarily uh The a sign of the like the neatest things that you could be doing. So hopefully they're slightly horror stories, but maybe there's some useful tips that some of you will find find helpful.

Um as evidence that um Of of this. Here's a picture of two people. This is the same person. So this was me, the very first conference I went to with work. This was uh PyCon UK 2014. Uh and then on the right that's just a picture of me enjoying another beer. Um I think that's the last image we get of beers. I think we're in the alcohol section of the uh talks at the moment. So yes, I think not unique insights, but hopefully, yeah, there's there's some things that I can tell you from having kind of worked on one project for a long time. So a couple of key themes that I would like to talk about. First big one is sort of about interoperability. This actually overlaps quite a lot with some of the discussion that um has been going around a lot of the other talks. It's interesting how Thoughts seem to align sometimes on this.

I'm going to talk particularly about kind of front-end framework interoperability stuff, which has sort of come up a few times already today and also yesterday quite a bit as well. Then the other thing that I want to talk about a little bit is taming complexity. So as a code base gets bigger and you have sort of more legacy features, legacy APIs and things that you need to maintain. How do you handle some of the problems that come with that? In particular, I'm really going to focus on what do you do when you've got slow pages, slow queries? How do you kind of handle that, even if you're not necessarily able to do something about Impacting the sort of underlying performances of those things, how can we kind of work around some of those performance issues? So yes, there's at least one story from each of these. I realized as I was preparing slides that 25 minutes is really not very much time at all. So won't be able to do everything that I might have promised in the abstract.

Hopefully the story that interested you will be in here somewhere. So let's start on interoperability and uh yes, talk on that uh front-end framework stuff for a bit So a little bit of storytime first. The year was 2010. The number one single in the UK was The Club Is Alive by JLS. That might be a reference, which I realise doesn't necessarily transfer across the pond very well. Don't know whether JLS really made it international. So the UK's number two single was California Girls by Katie Perry, so maybe that's a a better reference. I don't know if this just makes me feel old though, that that can possibly be fifteen years ago, but anyway. Um other things that happened in 2010. Um the iPad and Instagram were uh first launched.

And crucially Little video. The company that I work for now was founded. So it's gone through a few rebrands over the years. First off that was UK Funders. And so This was the only evidence I could find of our logo when the company was first founded in 2010. So you got a little bonus video there. Um why am I telling you all this? Um so I guess basically just a reminder of like what the state of the front end world was like. back in 2010. I don't know if it was just us, but um I think frameworks were very much less of a an obvious thing to be doing with your front end code at that point in time. So AngularJS was very much in its infancy in 2010, otherwise React views, all these kind of other frameworks that we sort of talk about today were not even apples of their creator's eyes yet. Uh and so

what did our kind of front end code base look at this look like at this point in time? So obviously we have lots of kind of vanilla. js stuff. Um of our big complicated features, which is sort of writing in in JavaScript. But obviously JavaScript at that point in time as well was not a sp especially well-developed language for doing a lot of the kind of complicated interactivity stuff that we want to do now. So alongside that you kind of had to use jQuery to do anything a little bit more complicated. This was sort of where we started out A few years later, this is probably around about the time that I actually started at the company, we're sort of looking into uh what kind of front end frameworks that might be interesting, what's out there. Um Did we go with React? Did we go with Angular? Did we go with View? No, we went with Backbone. js, obviously. Everyone remembers Backbone JS, right?

Um Okay, at least one person remembers Backbone So it actually has a lot of really neat ideas. And I think when I sort of uh look back at how a lot of that kind of stuff works, there's a lot of overlap with some of the ways that HTMX works as well now. So I guess what's what's old is new again. So we kind of work on this kind of infrastructure for a while, and then a few more years pass, and we sort of think, oh, we maybe want to move on to a new front-end framework. And so we sort of look at what's out there, look at React and Angular, and Vue is the horse that we back for like what our new sort of front-end framework is going to be. So that's what we go with. And so then obviously the next step is we migrate all of our legacy code over to Vue, and so we have one beautiful, elegant Vue front-end

system. Or perhaps not. Instead, what we end up with is, and the year is 2025, and it still looks a bit like this. Maybe we I want to shrink the jQuery and backbone um logos a little bit on this. But this is kind of still what our stack looks like today. Maybe this isn't the right answer for you. If you can fully migrate over to your the new system that you want to go to, then I would suggest that you do. But For us, that was looking like a huge, complex project. It's probably going to be like six months of everyone just like dedicating their time to this. What are all the like valuable things that we could deliver as a business that we weren't going to do? uh if we decide to just like migrate everything over to our new framework. So instead we had to kind of go with a bit of a hybrid situation. But suppose for you you're able to get that migration done a lot quicker.

Maybe you can do it in a month or something and you're able to allocate the resources for it. Um you're still gonna probably want to have some kind of transition period as you move from one to another. Uh you don't necessarily want to have you know, D-Day or whatever, where you're suddenly switching from one framework to another and all of a sudden all of the bugs are introduced simultaneously. So maybe it'd be useful to have a way of kind of working with two front-end frameworks simultaneously without sort of fully, fully backing one So here's um a little pattern that we've uh kind of used fairly extensively to sort of work between um kind of the different frameworks that we have as as part of our stack. And so the key thing here is using this kind of custom event thing from JavaScript. This kind of allows you to define a custom event and trigger it, and that's something that can then be caught somewhere else in your code base

So the idea here is that if you look at the sort of first snippet um up there, this would be in your legacy framework of some kind. You create one of these custom events, you give it a name. So here we're calling it load view component. You can send some detail to this event. That can be kind of any kind of data that you might want to pass to your new framework. And so here what we're doing is we're kind of saying what the name of the new component in our new framework is that we want to now render as part of our like old legacy system. We give it an ID of kind of a div that we're going to be like creating in the page in our in our legacy framework for where we want to inject that new component. Uh and then we dispatch this event. And then in our new framework, we add an event listener for that event, catch it,

and then When we catch that event, we are kind of collecting up all of those components that we now want to render from other parts of the platform. Here we're using this teleport feature of view. React has a very similar thing called portals. I imagine it's a thing in kind of lots of other frameworks, to basically inject this component now into the right part of the page. So this is a way that we can sort of start to add progressive enhancement to the legacy parts of our front-end frameworks. Another little tip that I'm not going to talk about for too long because I think lots of people have talked about it before, is the JSON script uh filter that's available in the Django templating language And so we use this as kind of a neat way of sending data between kind of maybe sort of more legacy Django views that we have

and sort of modern parts of our front end framework. So this is in the case where maybe you don't want to like build a whole new API to send some data through to uh some part of your your front end platform. And so we sort of have established a pattern where we always use this like view data name in the context when we're passing something through. uh from the uh in the Django template. Uh so that's the first snippet. And the second little snippet there is what we would have in like our base template. And so that means that anytime this view data thing is found in the context, it automatically gets passed into this JSON script thing. And the idea with JSON script is that it will create an element on the page. It will serialize the data that you're passing in from the context and stick it into that element on the page. And then in your front end tool, you can access that

through this sort of get element by id thing and then we call JSON pass on that in order to then turn that into a JavaScript object. So it's sort of a a relatively like neat API that you can give yourself in order to sort of pass data between those more legacy Django views and your sort of more modern front-end components. That's the last bit of front-end code I'm going to show you, I promise. Moving on to uh sort of taming some of the complexity. Um as I say, really just going to talk about slowness here. Um complexity, I think Over you know, 10 years or so is going to creep into your project. You're going to have some legacy features which your CEO tells you you really can't kill. Um or you're gonna have uh sort of legacy APIs that you've built for particular clients and that sort of thing.

Uh and one of the kind of key outcomes of that I think is that like some of those things are likely to not be performing particularly well, but also you're not necessarily going to want to invest that much extra time in working on top of those because they're not delivering that much new extra value for for the business. So how do we kind of uh look after those those things. One idea is leveraging the sort of statement timeout of uh Postgres databases. If you're using Postgres, I'm sure there's equivalence in other databases. Interestingly, there's like no default statement timeouts when you sort of have Postgres kind of configured with its default settings. And if you look at the Postgres documentation, they also tell you not to set a default statement timeout everywhere. and rather handle it at the application level.

As far as I know, there's not really an easy way of then just setting that statement timeout within Django on uh on all queries. So what we do is something that looks a bit like this. So we extend the Django Postgres backend. So we extend the database wrapper. um within that. And then there's this init connection state method uh which initializes those connections and all we do is just sort of As a post step after that, we check for this statement timeout within the settings that you've configured within your Django settings. And if that's there, then we uh execute this query to set the statement timeout on any kind of uh subsequent queries that happen on that. that connection. So this can kind of ensure that you always have some default statement timeout. You might want to set that sort of relatively low so that oh if a particular query on a random page for some reason ends up taking too long.

just let it die, you'll either get a 500 error or you can then catch that uh exception a bit lower down in the in the application and show a more friendly uh error to the user But then what you might want to do as well is suppose there's certain pages which you know are particularly slow. So then you want to tell them, well, you're allowed to run a little bit longer. Because we know this page isn't accessed too often or something like that. And so we can use something like a context manager, which looks a bit like this, in order to set a different statement timeout just for all the queries that might be executed within that context. So what we do here is we uh get the current statement timeout so that we can reset it back again at the end basically, then set it to whatever the new value is, uh, and then finally at the end, yes, we uh we set it back to what it was before on the connection.

Another idea. Well maintained. Um had to AI generate a logo for PG Cat though, by the way, because they they don't seem to have come up with one yet. Um it's a really useful tool. Largely the idea for uh things like PG Bouncer is this idea of connection pooling. So I think the reason that um people generally use things like PG Bouncer is that the kind of connection overhead when you're connecting to a database is actually quite high within Postgres. So it actually costs quite a lot of memory, I believe, to have like lots and lots of connections open with your database. So that was why I think PG Bouncer is uh is generally used, but we can actually use it for a slightly different purpose in that case.

Um what PG Bouncer allows you to do it, it allows you to have um a certain number of connections which you define that PGBouncer will open to your database and it will never open any more than that. And it also says how many uh simultaneous queries it can hold on to before it starts just dropping queries altogether. So you can give it a really low number. So say you tell it, let's just have, you're only allowed to open two connections to the real database. And in total, you're going to handle four queries and anything after that starts getting dropped. And so we can then use that as a wrapper for those particularly slow uh pages that we have, that those always get kind of executed through PGBouncer instead of directly to the database. And you can then um kind of uh ensure that the load on that doesn't sort of too badly impact. the rest of the platform.

So let me show you what a little bit of what that looks like with a quick diagram. So this is the situation where you don't have PG bouncer between you and the database. First query to your really slow page comes in, uh, everything's fine for now. Another one comes in, and those are still being served simultaneously, and now the load on the database is starting to get a little bit heavy. Someone else hits it again, someone just mashing that refresh button on some reporting endpoint that you've got, uh, even though you've told them please don't do that. Uh and then the fourth one comes in and all of a sudden The database is now completely locked up. All the CPUs are busy doing something. They're serving those four really long running requests. Don't know how long this is going to take. Now no one can log into the platform. Bad things are happening. Instead, in this case where we have sort of PG bouncer in the way, realize its dioram just says database, but imagine that there is a PG bouncer which kind of acts like a database just in the way between you and the database itself.

This first two queries come in again and the load is starting to get a little bit heavier on the on the server. But now instead the next one queues up behind the previous one that's already there. So the database connection won't get dropped, but it's just going to wait until that first query is actually finished operating. And so the load isn't going to get too heavy on the database itself. Other people are still able to access the platform, do other things there. If a fifth one comes through, that just immediately gets cancelled. And again, that's an error that you can kind of uh catch a little bit higher up in order to Give a nice friendly message to the user saying, sorry, we're a bit busy doing other stuff here. You have to come back and try that again later. And then one last topic. Again in the sort of load balancing

kind of area. I've got a massive Envoy logo just because that's the tool that we happen to use for this. This is kind of how we're load balancing against our uh web workers. Um so one of the things is that like for most load balancing tools that you're using. You'll probably find that the standard load balancing pattern that you're using is round robin load balancing, which is mostly fine if you have a again, for all of your views take a similar amount of time to load. Um, that will probably work absolutely fine for you. But suppose you're in this situation where certain pages maybe take like 100 milliseconds to load. Other ones take something more like 30 seconds. You have this kind of really imbalanced load that comes into your platform.

So what do you do when one of those reporting endpoints gets hit and it's taking 30 seconds to load? Why is that a problem? Well, let me show you what a situation might look like here. So suppose we have three web workers and kind of our requests are coming in. First request that comes in is a really quick one. This is one of those ones that's only going to take 100 milliseconds to process. But then one of those slow requests come in. We don't get too many of those. So they're not sort of showing up too regularly, but one of those slow ones comes in. And then a third uh quick request comes in. Workers one and three uh go away and do the work that they need to do, serve those two requests, and they finish off now. Then the fourth one comes in, and so with round robin load balancing, it's been through workers one, two, and three, so now it goes back to one again.

Request for is served by worker one. But the problem is that request 5 now gets stuck behind that still slow running request that you've got worker 2 running. So even though there's a completely free worker that's available with round roll and load load balancing, it won't automatically kind of use that freely available worker. Depending on the tooling that you're using, this might already be set up nicely for you. But if it's not, it might be something to look at. And so an alternative strategy is least request load balancing. Which kind of works in a fairly obvious way, as you might imagine, from the from the name. So those first three requests come through in the same way. Those uh requests one and three are um handles uh as normal. Then now when request four comes through it has a look at what workers um

what requests they're currently serving. So it can see that oh worker two is serving request two, so I won't allocate to that one. But workers one and three are free, so I can give one to to worker one. And then the next one that comes in, again, it checks for what's available and that that comes through onto that third worker. So again, really useful, I think, in these situations where you have kind of very um variable loading times for for different parts of your platform. Um actually got through that rather quicker than I thought I was going to. But yeah, reminder of the things that we talked about. I've showed you kind of a pattern for sort of defining some clean interfaces between your different front-end tools that you might have by using this kind of custom event thing. And also talk briefly about using JSON scripts to kind of send data to frontend.

And then this, this, and this. I had absolutely no time to talk about, or I thought I had no time to talk about. Please come and find me afterwards if you're interested in hearing a bit more about those Uh and then the final topic I sort of went through there was yeah, talking about how do we handle those slow pages using a combination of uh database statement timeouts, uh using kind of connection queuing through PG Bouncer in order to uh queue up those queries and then also talking about least requests load balancing. Um yeah that was it from me. Uh but yeah any thoughts or questions please come and follow me throughout the remainder of the conference. If you do have any questions now, happy to to discuss those. But otherwise yeah. Thank you very much for your time.