Django Admin at Scale: From Milliseconds to Microseconds 🚀

Speaker 1: Uh thank you for that introduction. Uh well uh congratulations everyone. You're finally uh made it to the last talk of Jangukon So yeah. I know I know you all need to prepare for uh after party, but yeah, we'll quickly uh wrap this up. So yeah, uh Jen Weadmint scale Let's start. Uh first let's talk about Django admin, right? Because it's basically a dream, right? Because without Much uh without doing anything you get to so much, right? Like based in cred operations without writing any code, just you know initializer models And uh form validation and generation, that too, without anything. And authentication and permissions out of the box. That's really cool, right?

Speaker 1: And also you can easily customize it with a couple of lines of code. Yeah, basically a dream Uh the I believe the best thing uh since sliced bread, right, for rapid development. Now Let's talk about a real story. Like to be honest, is the big reason I'm here. So last year I was consulting a streaming company where we had a critical table called notification log. This table uh tracked email templates, like uh when emails were opened, clicked campaign performance data and user engagement metrics. It was very critical for the business because our product and

Speaker 1: marketing team relied on the data. Because this data helped to identify what our user likes or dislikes so that uh upcoming campaigns can be designed uh basis on that actually. But one day a disaster struck. So have you been you know trying to open a page and waiting, waiting, waiting, waiting like Just to, you know, some data, some metrics. But after so long, uh you get this right. So yeah, so that's what our product manager saw. So, our product manager opens admin to check the performance of uh latest campaigns that were recently deployed. But after endless loading, uh what she sees?

Speaker 1: This is the good old I think sorry, sorry, sorry, sorry. Yeah, the good old 504 error. So uh the reality check here is when our database grows, so the Django admin can become very slow. But uh and on Slack all I could hear is that we need these pages working again, right? And they need to be fast. So uh the business impact like because of uh these five fours or the managers were not able to see the metrics They were not able to verify campaign performance. Marketing decisions were delayed. New

Speaker 1: features were put on hold while we fixed the admin, made it fast. And also the team credibility was uh damaged because uh we had uh legacy systems already which was working without issues and But with uh modern frameworks and this latest, you know, framework, you're seeing fireworks, right? Uh that should not be the case, right? I mean Django is fast, right? I mean we can definitely scale, right? So the turnout. So we implemented couple of uh we implemented a turnover to and fro, complete couple of techniques or like what to do and what's not to do in our code base. Which I'll show you today.

Speaker 1: And so the before what was happening is timeout errors greater than 10 seconds, basically fire four, first rate in teams, teams are not able to see your data and it also impacted business. But after Skoval techniques or you know, after implementing these things So the response dropped down to less than 200 milliseconds. But uh I know I know the title says microseconds, but yeah, you all were click-baited, so sorry. But yeah, so uh after our implementations, products managers were happy and able to uh quickly see the data and uh act on that. Then uh they were able to uh take data driven um uh marketing decisions actually.

Speaker 1: So yeah. Now let's talk about uh today's journey. So like but we'll be going through. So so first will be diagnosis like how to identify common performance bottlenecks like Uh what can be you know minor uh issues about Lex that you can easily identify. Uh then we'll talk about solutions like battle uh what we have uh implemented and what we have changed so that techniques I'll be showing you and then maintenance like how we can keep uh these performance over time like your code base grows and your database grows so that Now let's talk about like diagnosis, like what makes our Django in slow. I believe you

Speaker 1: all have probably faced this by now, I guess. The simple N plus one query problem, right? For example, this order admin, we are just showing the ID, order ID, total and status, and the customer name here, right? But behind the scenes, what it will do is it will make one query to fetch orders, but n number of queries to a fetch-related customer for each order, right? So basically n plus one queries. And when you go use Django debug toolbar, you'll see something like that, right? On uh on top you'll see some meta queries And the count query, which is from default page needed. But in the drop-down, we'll see uh lots of uh this customer queries, which is happening because of our customer name, right?

Speaker 1: So next one will be inefficient filtering, right? So we we have we can have lots of filters right in our uh uh models and table For example, in our product admin we have something like this category tags and stop, right? But your when your database grows, right It can contain millions of records, right? So for this filters will basically make full table scans and if you have multiple filters that you know You see where I'm going. So we'll talk more about this at the end because this is the uh main reason I'm presenting here because the filters were the main reason And uh another one is uh memory intensive admin actions, right? It's like we for example we need to uh

Speaker 1: bark uh product active it uh featured or not. So we can something like that we just loop over the query set and uh product is just set it true and save it. But uh let's say we have selected lots of uh objects at a single time. It can lead to out of memory errors with thousands of so and individual saves so and queries so that is uh one uh common uh bottleneck The next will be change forms, right? Uh In for example, in product, we can have you know different uh foreign keys which can contain thousands of you know options for like categories, tags, related to products, right?

Speaker 1: And many to many keys as well. So Django will load all options for each field, like all categories could be thousands, right? and uh all tags could be tens of thousands and uh other for related portals so yeah that can lead to very slow page loading and it will basically uh lead to five core. Next. So these were the common uh border legs you will probably see Now how to diagnose your own admin, right? Like these were the some issues which I faced. There were some others, but yeah, those were the very common So how will you diagnose? So first thing is officially Django debug toolbar.

Speaker 1: Please use that. If you are not using, please do use that Really helps a lot. You have seen uh the screenshot before that how the queries will be visible from that uh uh screenshot then uh you can uh write your loggers loggers that basically logs your database query logs like how much time is taking how what is happening behind the things you can use those And uh Django Silk Profiler is also a very nice tool if you're not using please do check it out. Then you can utilize queryset. explain like uh index is being used or not, what's going uh in the query. So you can utilize that as well Other than that, the uh different tools like New Relic, DillDog, and Century Apim is very nice tool. You can probably utilize that to monitor

Speaker 1: How let's come to uh solutions like how uh we started to improve our uh jangment to make it a bit snappy rather than throwing fire fours So let's talk about the N plus one problem, right? It can be easily solved by strategic uh related field loading. For example, we are here showing the ID, customer details, total and status. So before uh using that list select it, we were causing n plus one queries. Now if we just use that and it'll just One query it basically creates a join in SQL and it will fetch the customer with the orders. You can also write custom

Speaker 1: display metal just like we are doing here. If you have any custom thing you want to appear on that admin page, you can utilize this or something like that Also you can use uh preloaded annotations which basically uh helpful for calculated fields like do you don't want to do uh every time to uh for every object you can just On query set level you can just annotate this and it will just utilize that. Next is uh pagination. Before I talk about this, in Jenguen 2022, Tarun gave a very nice talk which covered different pagination techniques This can be a very good starting point for implementing customer pagination. So if you have not checked it out, please do.

Speaker 1: So here uh let's just say we have lots of records in our uh a table, for example. By default a list per page is 100. So if let's just say we have millions for record like 100 is very small, right? We can probably increase it So a better approach is by default it uses offset limit pagination to make it uh Optimal. We could use key set pagination or some might of you know as this cursor pagination, I believe. So this is a custom implement implementation which I'll be showing you, like how we can utilize. So it basically uses the rather than using uh limit offset.

Speaker 1: We use the primary key to fetch the ID from the previous page and use that ID to Filter the records for the current page, right? So it will basically perform better to our traditional limit offset page initiation on a larger dataset With uh custom pagination like this, we can easily paginate through millions of millions of records. Here we as I told, like using the last ID from a previous page and then filters the query set Major benefits in this approach are that filter can use the index, right? Index directly because usually we'll be filtering out the primary key, right? So already indexed. So it just Help whereas the offset limit offset uh pagination

Speaker 1: would need to scan and discard the earlier rows first and then we'll fetch others So let's uh this brings us to optimize uh admin action, right? Uh previously we were looping around just to you know Save our products just to market feature. But we utilize we can utilize the bulk operations without uh loading any of that. Like just use the query set update. It will basically update all the records in a single query. Super efficient. For simple operations rather than using loop, you know, very efficient. But now you all can say like what about the complex operations, right? We need to do something before saving the object, right? And now that takes us to

Speaker 1: batch processing. So what we are doing here we are basically creating batches so that Uh it doesn't take lots of memory at a single time so that it basically leads to OM. So just create a batch of you know, let's just say for example thousand objects or five hundred. It's basically on your like how much you can how your server can handle create a batch like this and this process it batch wise it will be very efficient rather than uh by going oh looping over so it will basically help you Now uh change forms. So

Speaker 1: for foreign keys when you have multiple uh options or categories you do when you uh when you load that page it will basically start to uh time out cause let's just say a forum key has Thousands of records, right? So it will basically take some time. So you can replace it with uh raw ID fields. It basically just shows you the ID of that object and you can just go there And it will take you so it will save some save you some time. For many too many, we have filter horizontal. It basically helps you to load many too many keys, uh records easily without you know uh much taking much time. Now you can uh force your uh query set size as well using the overriding get form.

Speaker 1: You here we are I'm uh for leaded projects I'm using I'm Putting in a limit for how much products I need to load. So that you can also do. For expensive field like which needs some processing. So you can utilize this field sys. Which collapses classes to uh defer loading expense expensive form section. So it's basically helps you save some time Now uh interesting stuff, targeted caching. So anyone here have used uh Django Cash Ops? If not, uh that's great. If not, please do check it out. I highly recommend it

Speaker 1: Very nice library, handles most of the things automatically like cache invalidation, query state invalidation, everything. So and it is very stable and uh production ready. So please do check it out So here's an example of uh custom uh model admin which we can utilize. Here we are just caching our query set for uh the timestamp we can provide like how it timestamp will be uh the timeout will depend on how frequently our data is being changed You can uh uh put your as per your need, you can put anything like that. But the cache S is provided by cache ops. Basically it will uh create uh quiz caching in uh your whatever uh cache you are using, memcache, redis, or it works with.

Speaker 1: all of those so it basically uh caches your query set and it will it will be utilizing every every time you uh fetch it Now, how we can use this? So just uh inhale it from our uh cache source metal admin and just uh sh uh set your uh timeout for breads and it will basically uh start using our cache for uh fetching records. Next list filter optimization type. So standard it basically loads for uh every request like it will basically query uh your table to fetch the uh categories. But uh what we can use is uh override this admin simple list filter and uh just override the lookups method

Speaker 1: And by caching, you can just limit our options by using a query set and it will basically use the caching option. So it will improve the your list filter optimizations. And next is yeah uh when uh everything is not working for you, then probably You can utilize raw SQL, but I uh don't highly recommend it. But use sparingly with Constitution because soon your code will contain more SQL and less Python. So yeah, I have seen it. So yeah So that were some solutions that uh we have implemented and improved our uh performance of our admin pages Now, uh maintenance, like how we can uh keep these performance gains over time, right?

Speaker 1: So monitoring strategy, it basically depends on your use case, like how you're growing and how your code base is growing, but these are some. uh techniques which I have seen or have implemented. Basically you can create your uh custom middleware to track uh admin performance uh setup alerts as well like for slow as let's just say it's taking more than uh 200 milliseconds so you can just log it somewhere and you can just you know work on that like what it's uh how why it is taking so much time Then you can also have uh regular profiling sessions. So uh we used to have bi-weekly uh Sessions for our API uh like what uh last two weeks why there's a spike so then we started to uh talk also about uh like why this Django pages uh spike

Speaker 1: why what was the reason like why this went slow so we had to we uh started talking about that as well in our sessions Then uh last thing is you can also use uh monitor your index uses because it can also uh become slow when you when it grows, right? So a sample performance middleware can be something like this. You basically in your middleware you can just basically log your like It uh went below the threshold. So you can just log it and you know see what uh is going on. You can just improve it later on. Like also you can do performance testing as well, you know you add add uh admin specific performance tests

Speaker 1: Try to measure with uh realistic data volumes. I know it's a bit difficult to uh replicate your production uh load on local, but it's manageable somehow Also uh you can simulate uh real user interactions, not just page loads So a simple uh performance test can look like this. Like we created uh 10,000 test records just to test like how much time it will take. So it basically depends on your needs, so you can just utilize that. So uh real world example like what we uh what I or my team did So this is uh what my

Speaker 1: log admin looked like. So it has you as you can see it has six filters We uh for pagination we were already using no count paginator, so no uh count query. So yeah And uh I believe uh we are already using listed list selected, but still it was throwing five course. So after uh digging uh checking out like what is taking some time using uh debug toolbar so the major problem was the lookups qs here right It was uh I believe at that point of time it was maybe overly optimized or uh uh it might have been assumed that these the categories which we are uh

Speaker 1: Fetching might be dynamic at some point of time, but the uh the values were static and came from constants, right? And it uh they were defined by analytics team so so that they can easily query. So uh they these categories can be returned without utilizing DB, right? We can just return these constant without uh db users. So what we did something like this. We let's just say we had categories like this. So we just created this value pairs and just returned it in the lookups rather than making query for that, right? So

Speaker 1: It uh basically takes uh us back to the gold old principle. Kiss, keep it simple, stupid, right? You don't need to query if these are in our constant right. So uh sometimes we try to make things complex which may not be required but sometimes it's required so basically on your you know need So uh the next thing what we did was started utilizing uh the only and defer fields. So uh we did uh so uh Since it's not a big table, we needed lots of table data, but we knew that we don't need this data. So we utilize the deferred fields or defer

Speaker 1: so that it can basically Just uh remove these fields from the query set. So it basically it uh improved the performance a lot. Also, we are using here if you can see the dot cache here, it's coming from uh Django cache ops So we can just utilize like what ops like get count and we can you can define your uh method what uh you uh what oh I'm spoken the term but In ops, you can just say dot git dot get. count, you can define that and it will basically cache that. And obviously timeout will use that for validation and everything. So uh the key takeaways was uh so uh in my local I was not able to replicate like why this is happening.

Speaker 1: So always try to diagnose with real data and to So what I did was uh installed a Django debug toolbar on my QA environment. It uh uh the QA team was not very happy, but yeah I had to do it And also like next thing is think about uh query accounts, not just speed. Because major of the uh as in previous talks I've seen that major of the Slowness comes from database, so that that too. So when optimizing, try to use uh built-in tools first like Slack dated only, defer, etc. It really helps a lot because Django is supposed to be faster, right? Then use cache strategically because uh in uh read-heavy applications

Speaker 1: the data might not be changing that much Also the last thing is monitor and uh maintain performance. So yeah. I believe thanks for listening. That is it.

Speaker 2: Congratulations on making your product manager happy. But in your case, and also in many other cases, I think you only run into these performance problems after they already hit production. Do you have any tips for us on like preventing common mistakes during for instance code review?

Speaker 1: Uh just like a tool like try to write a you know uh test performance related tests so that you know you have an idea like what it how much load your uh code can uh take at this point of time so Probably adding performance ready tests will help you. Also that admin that middleware I showed it basically help you to know identify problems earlier. So that might be useful for you

Speaker 2: Yes, thanks.

Speaker 3: We have one question online. Will this also improve crude query set operations that are carried out in the front end or not in the admin? For example, what we are using the Earth?

Speaker 1: Uh can you please repeat?

Speaker 3: Using DRF. Uh

Speaker 1: using DRF.

Speaker 3: Okay, I'll repeat the question. Will this also improve crude query set operations that are carried out in the front end and not in the admin? For example, what if we are using DERV?

Speaker 1: Yes, I believe using Slack traited only defer will definitely improve your uh API performance. And yes, definitely.

Speaker 3: Thank you.

Speaker 1: Thank you.

Speaker 4: Yeah, thank you for the talk and the overview of cache ops. And I was wondering if there are any other uh caching strategies that you tried and experimented with um before arriving at this um that you could talk about?

Speaker 1: Uh previously we were using normal uh Radio session. So it had lots of problem like invalidation and everything. So that's why we that's when we got to know about this library that it handles automatically most of the things for us. It uh caches the query set directly. So I don't need to you know convert it to a list and string format so that uh it's just add some more time. So that's why we got to know that Try to use Django GashOps will basically help us sometime.

Speaker 4: Thanks.

Speaker 3: No more question Well then thank you again for your talk.