Optimal Performance Over Basic as a Perfectionist with Deadlines - Velda Kiara

Speaker 1: Um with Velda Chiara. Here she is. She has, I've just learned, flown all the way for this conference from London, stopping off in Nairobi to see her folks and to pick up some things because she wanted to stand in the purple light. I asked I asked Velda, I said, what would you like me to tell them? And she said, I like purple And I I didn't realize she really likes purple. She came for the purple here. Oh, and by the way, um I'm a software engineer, which is a vast undersell of what Velda does. She is a Django contributor, working with the Django Conference nationally, also as a contributor in their ongoing work. for the community at large. She is also recognized by Microsoft as a valuable contributor. And today she'll be talking about not only how to optimize

Speaker 1: your work in your Django apps. but also how to minimize your memory usage. So Velda, um thank you for coming all the way for us. Floor's yours.

Speaker 2: Ooh, hi My name is Velda. I am a software engineer. I'm also a Microsoft most valuable professional for web and Python development. I contribute to open source and I currently help maintain the Django debug toolbar if you've used it. I'm also part of the Django Not Space as a star, which is a program for Django people. And I run the newsletter called The Storytellers by Tales and Substack, which is under veldekiara. substack. com. So I am honored to be part of the conference because

Speaker 2: as you heard, I love purple and I love to be part of a conference that loves or appreciates purple as much as I do. And today I'm going to be speaking on optimal performance over basic as a perfectionist with deadlines, which is a beautiful way of saying Django optimization. So before we talk about performance optimization, we need to understand why it actually matters. So in 2016, Google Consumer Insights found that 53% of mobile site visitors would leave a page if it takes more than

Speaker 2: three seconds to load and which means that you no longer like the longer your page takes to load the less conversion to user buy-in you have which is not great when you want to increase sales and make a profit. And optimized pages reduce cost because you use fewer resources in terms of servers and it's also able to scale because then you use the same resources to solve High traffic. I'm gonna adjust this just a second.

Speaker 2: Oh, sorry. Yeah. Just a second, my display was acting weird.

Speaker 2: No Yeah, I think that's a good idea.

Speaker 2: Hi there. Hello. Yeah, let's see. So I wanna give it a call. This Okay, yeah. So that was fun. So We have talked about why performance matters, but in this case why you should be excited for this talk despite the technical difficulty is we're going to be focusing on database optimization, view rendering

Speaker 2: optimization, efficient serialization, and concurrency. I'm excited also because one of the speakers also talked about Async IO, which is Part of what I'm going to talk about in just a few minutes. And when we talk about database optimization, it's important because it allows you to get fast and efficient data retrieval, especially in high traffic applications. And there are different ways you could optimize. So first things first we're going to look at indexing in databases. Yeah. So an index is a data structure that improves the speed of data retrieval operations on a database. So indexing helps in faster accessibility of your database.

Speaker 2: So in an indexed database it searching for data in columns takes a bit long And a quick check-in with the code you're seeing right now, it's Django Code, and Django uses the MVT architecture where your models handle the data logic and structure of your databases, which is what we're looking at now. Your views handle the application's logic and functionality and templates handle the layout and structure of your application. So, how indexing works is when you create an index on a column, the database builds a different, a separate data structure, which tends to be A B tree or a hash table that contains the values of the index columns

Speaker 2: along with pointers corresponding to the rows in the table. So when a query is executed, the indexed column, the database can use the index to quickly locate the rows that match the query criteria instead of searching the entire table. In our superhero model, we have indexed the name of our superhero and the real name. So every time you run queries like superhero. objects dot filter then pass in the name like if you're a fan of Spider-Man or Superman the the index will help the database find the relevant rules much faster. So this goes for the real name as well So, as much as indexing is helpful and helps you retrieve data quickly, there are some trade-offs to consider.

Speaker 2: Like as I said, it takes on like it creates a separate data structure for you, so it's going to consume additional disk space because it creates a different data structure. And then when you insert, update, or delete rows in a table with indexes, the database must also update the indexes, which slows down the operation. So it's essential to balance the need for fast feed operations with a potential impact on right performance. And not all columns should be indexed, like you can pass in all your fields there The other thing we're going to look at is query optimization with select related.

Speaker 2: Select related is a method provided by Django ORM that optimizes database queries for models with foreign key and one -to-one relationships. So it performs an SQL join operation to retrieve related objects in a single query rather than executing separate queries for each related object. So our publishers in this case, I think yeah, you can see that. So our publishers publishers sorry in this case are the movie houses that actually brought the superheroes to life on the TV so it can be Marvel, DC, Pixar, whatever excites you

Speaker 2: And when you're using select related, it performs the SQL join as I said, and it's really efficient when you're dealing with a large number of records. So ideally how what would happen here is it would Fetch all your superheroes along with your the publishers obviously of the Said Heroes So without select related, you'd run into the N plus one query problem because and being the number of superheroes because you'd have to fetch All the superheroes then s then fetch all the publishers of each hero. The other thing we're going to look at is query optimization with prefetch

Speaker 2: related. So prefetch is also provide is also a method provided by the Django ORM that Optimizes database queries for many-to-many and reverse foreign key relationships. So unlike SelectRelated, Preferred Related executes separate queries for each relationship and then combines the results in Python. It's this method is actually useful when you're dealing with like large data sets which have Complex relationships. So to show this, we are going to update our models with like a review and establish the many-to-many relationships with our superhero model.

Speaker 2: And what what happens is It's a many-to-many relationship because a superhero can have as many reviews as possible and reviews different reviews could also include as many superheroes as possible. So this is how you'd evidently run the query, and what happens is it's going to fetch all the superheroes in one query, and then it's going to fetch all the reviews related to those superheroes and then combine them in Python. So without this Uh without running prefetch related, you'd have to execute one query to get all the superheroes, and then a separate query for each superhero to get their own reviews, which still resolves to the N plus

Speaker 2: one query that problem that we talked about. And this is also going to result to a significant performance hit because it's still going to hit your d your database every single time. So Use profetch related if you have complex relationships. So we're going to look at sharding which is the process of distributing data across multiple databases to improve performance. And it Typically helps avoid overloading a single database and ensures that traffic can be evenly distributed across different servers. And how it works is it can be split up into different criteria

Speaker 2: So different users' data can be stored in different shards depending on their user ID. Say we have user ID one all the way to 100, could use probably shard one. 101 to 200 could use shard too. You could also distribute data based on geographical location of your said users, which can help reduce latency. Another criteria you could use is one that is in line with your business logic or whatever works for your business, whether it's splitting the data into Continent separations like having eaten in Asia, Europe, Australia or Africa

Speaker 2: So in Django, you can implement shirting using third-party packages like Django DB Shard, or you can manage it manually by creating a custom database router. So here is a simple illustration of how you would define your custom database router. And this particular function is for a read operation and it directs the read operation to the appropriate shard based on the primary key of the model. So if the primary key is even, it routes it to shore dB1. And if it's odd, it routes it to shard db2. This the same thing goes for the right

Speaker 2: operation. So if the primary key is even, shard db1, and if it's odd, shard db2. So uh when when I forgot to mention that this is a separate file that you need to include in your models or in your apps. And you also need to update your settings because if you're not telling Django what you're doing, it's not going to be able to know what it's supposed to do. So you need to add your routing information to your database routers and tell it that oh when I say shard DB1, this is what I mean that you should use, right? So, with this setup, your Django application will use the custom database router

Speaker 2: to manage hurting for the superhero, the publisher, and the review models. And it's going to direct the read and write operations to the write charts. You also need to update your databases because you also need to define this Whether you're using Postgres, MySQL, or whatever database you're using, need to still define this here. So benefits of sharding include that It distributes the queries across multiple databases, hence improving speed. And horizontal scaling is also achieved, which allows your app to handle increased data volumes. Next up, we're going to look at view rendering

Speaker 2: optimization So views in Django are responsible for rendering HTML and handling HTTP requests. They play a crucial role in your overall performance of your application. So if views are not optimized, they can become a bottleneck, which leads to slow response times and poor user experience. And if users are not having a good experience, then that means you're not actually getting to make money because if users don't like it, then they're not gonna buy whatever you're selling, right? So ways to optimize include reducing unnecessary computations and also limiting data that you're sending to your client by using pagination. You could say that On the first page, you're gonna have like

Speaker 2: portray results to like 50 results and then later on show 100 results and all that You could also use caching that is cache the output of views or specific parts of the views that are frequently accessed by your users. For input and output bound operations such as making external API calls or query databases, you could consider using asynchronous views. And since optimization is just not a one-time fix and it's also not a one size fits all, you should potentially profile and monitor your application to see ways that you could actually

Speaker 2: optimize. So sometimes caching may work and other times it might not work because then you're having the stillness of data. So you need to actually look at what your application needs and what you could do to improve over time. One of the optimization techniques for templates is by using tags. So Tags are special elements within templates that allow you to add logic or dynamic functionality to the HTML. So we use in this case we're using the include tag that has a component slash navbar dot HTML. And the include tag In this case, it breaks the templates into reusable parts, thus reducing duplication and simplifying maintenance.

Speaker 2: So in my case, this particular file you're looking at is the base. html file that's going to be reused in different pages because I want to have the navigation bar in all the pages. I want to have my header, I want to have my foot or two and the title in different pages. So this is how the nov bar. html file actually looks like. So the include tag basically makes it easier for you to use different templates and reuse them. as often as your application needs. It also has role because for accessibility enhancement. So every time you're building something, please consider accessibility because we're all different.

Speaker 2: The other template technique for optimization is the use of static As you can see, static with the CSS files and the JS files. So static files are files that do not change dynamically and are served as is to the client. So common examples include the CSS files, JS, images, and your little extra fonts. So the static tag is provided by Django to generate the correct URL for static files It's important because the URL for static files can change depending on the deployment environment you have, whether it's deployment or production.

Speaker 2: And by using this static tag, you ensure that your application can find and serve the static files correctly regardless of the environment. So disclaimer is when you're using Django, please do not forget to configure your static files and add them to your installed apps as well as your static URL and directory as well. Uh right now we're going to look at fragment caching, which I didn't know was called fragment caching. I just assumed it's just something called part caching or something So it's basically caching individual fragments of your view that do not change, hence reducing rendering

Speaker 2: times. So in this case, you could cache whether it's a sidebar which acts as a nav bar because it's like it contains. information that doesn't change frequently like contact information about us and blogs probably if your blogs don't change as frequent as well. So in this case we're trying to cache it for 600 seconds which is like 10 minutes you could also go on full page caching where you cache the entire view For most of your static pages that don't don't change. So it's beneficial for pages that change that do not change. So if your pages are going to change frequently, you should consider using something else that's not caching

Speaker 2: Next up, serialization. So serialization is the process of Of converting data into a format that can be easily transmitted over a network like JSON or XML. So it's important to Ensure that you optimize your serializers because it reduces the size of the data payload and it improves response time and bandwidth usage So, to achieve efficient serialization, you could try to include only the necessary fields that the user needs to see. Let's say if you have a book serializer where the important information that the user needs is the title of the book

Speaker 2: and the author of the book. But now for us we need to also identify what the book is and we do that using unique identifiers like ID. So you could have the three fields. So this helps because once you reduce the amount of data being transmitted, it also improves the set performance of that data. You could also use custom serializers because if you want to be in charge to see which data structure you want to use, how you want to modify the data, if you want direct access, then you should use the custom serializers. And this helps you to optimize because now if you're using a data structure that makes it easier to transmit the information, then a win for you. The other thing could be using Gzip

Speaker 2: that is compression in serialization using GZIP. So gzip is a file format designed to compress HTTP content before it is delivered to the client. So uh it reduces the size of the payload, hence making it faster to transmit and if you want to use gzip in Django you need to add it as a middleware because it reduces the network overhead and speeds up responses. The other thing we're going to look at is asynchronous views. So asynchronous views enable Django to handle non-blocking operations like input and output bound operations which are database queries or external API calls

Speaker 2: without blocking the entire server process, which leads to better scalability and throughput. So a couple of benefits other than non-blocking and improved through boots include it's suitable for applications with like high input and output demand like API integrations. So when you're trying to add asynchronous views in your Django application, you need to import the JSON response which is used to return JSON data in your HTTP responses and you also need to import async. io Which is the model the module that supports asynchronous programming in Python?

Speaker 2: So in the first function We have also used the async keyword because that's the only way we're going to know that it's an asynchronous view. And the async um We use the await to call other asynchronous functions. So inside the view of the first function, we have the fetch data from API function, which waits for it to complete without bloc blocking the server and then returns the JSON response. And the second function is more like a a mock of the API call. And what this does, it's defines another synchronous function that simulates fetching data from an API and uses await

Speaker 2: async iode. sleep2, which is like a line simulating a non-blocking operation, which is For an API call, but then it pauses for two seconds, which allows other tasks to run during this time which makes it which makes the server to be more responsive. And then after it returns a dictionary with a success message that hey your data is fetched successfully. So our disclaimer for this or considerations you need to think about when you're using asynchronous views in Django is that It was asynchronized views when introduced in Django 3. 1, so you just need to ensure that you're using a later version that is more compatible for this. And then

Speaker 2: to fully utilize asynchronous views, you should run your Django application on an ESGI server like UV UVCorn instead of the traditional WSGI server. And then Jing is not fully asynchronized, so you could use other libraries or channels or databases to perform the same functions. So Celery is a powerful synchronized task or job queue that allows you to run tasks in the background. So it's great for offloading time-consuming operations like sending emails, processing large files, or handling other long-running tasks.

Speaker 2: So to create a task you need to add the Add Shared Task Decorator and you need to also install Celery and configure a message broker like RabbitMQ or Redis, whatever your preference is. And you also need to do this in your settings. py file so that again Django understands what you're trying to do here. And this particular function initiates the call the it calls a salary task from a Django view to send an email whenever A user uses the signup page, like, hey, thanks for signing up. It's like acknowledging that the user has signed up So um benefits of

Speaker 2: using celery is that it improves application's responsiveness because Users do not have to wait for this task to complete before receiving a response. It's also scalable because this the System is more robust under a huge load and it allows it to handle a large number of tasks concurrently. It also provides support for periodic tasks. scheduling or allowing you to run tasks at intervals like daily reports, cleanups, or any other tasks that you have. It also provides built-in mechanisms for error handling and task retries in case one of your tasks fails.

Speaker 2: So that's it for today, and you can reach out to me basically Velda Chara. I might be in different platforms, but that's the name I use And if you'd like to have access to this particular presentation, it's on this QR. You could scroll down to Pie Cascades, so that's where the link is going to be. I'm also going to upload the YouTube channel there once I get the link. And Thank you so much for taking your Sunday time and sharing it with me. I am honored and hopefully see you soon maybe in a different city. So thank you

Speaker 2: I don't know.