Django REST Framework: Taking your API to the next level by Carlos Martinez

Thank you. Yes. Thank you everyone for joining to this talk. Okay, let's begin our talk regarding um Django Rest framework and how to make it uh and you go to uh the next level. So first of all, um who are are used uh Django Rest framework or no Django Rest framework? So great. I don't need to to make a review. So who am I? I'm Carlos Martinez. I'm from Colombia. I'm backend developer at URIT. You can follow me on Twitter or GitHub and I'll have also I have um on a small blog.

Um in this is Colombia, there's the place that I came from. Um I'm run out the Python Bogota um group. Um we have a meetup for about Um 2,500 uh members right now. Um I also do photography time to time. I have a daughter and I do travel photography. Um so let's begin. We're going to start with a five minute um API. And we are going to do something like have users um create events, have promoters for events and to create tickets. So um

While we are doing this one, we're starting to think about these five topics. How to display different data based on context. For instance, I don't want to say to get the same feels for an event with inside a ticket for instance. Uh how to get better performance, how to filter information. Inside the API, um configure permissions and how to render results in different formats. Um let me get out of here and Let's take a look how our first version of the API is going to look like. So um this is our tickets endpoint, as you can see. Um Sending out

a nested object for event and also for user on each ticket that I get. Uh user has only um fields doesn't have any nested object in particular and um for promoter um for event sorry I got the same but only promoter is the nested object So um that's it, that's everything. The API is working. Um but no. Uh the problem is here that we have here is if I Um got only ten items, I guess I have right now. Yeah, it's not quite much. Uh it works fine. I get responses in about

um Let's take a look how much time is taking. It's about 100 milliseconds. But let's start to to create um a few more tickets. Always happens these kind of things. Tickets. Uh, come on. Um Give me one second please. Maybe I'm in the wrong branch. Yep, that was the reason. Okay. Sometimes happens

Okay. Right now I'm creating five thousand um new tickets for this API, including promoters and users. So um it is continue going to to create a few more. Um what we're going to see in our API is that all this response times is going to grow bigger and bigger and bigger and bigger every time So um our API is not done. So first of all, um something that we can um start to thinking about is when we have nested objects we can reuse our serializers. So instead of have different serializers for what is a representation of an event be uh and inside of

tickets and create a new serializer class, we can start to to use uh something called dynamic serializers so we can reuse our logic in a one ser in one serializer and use it for different purposes and different sections of the API. So this is um a dynamic serializer class that you can uh use Um is uh is based on Model serializer uh from Django Rest framework. But uh the difference is that it has um uh different attribute to get fields and that's what does really the trick. Then you will

can uh create, for instance in this case an an event serializer with uh an space uh to get the representation that you expect to get for this particular nested object. So um you can define um static methods to get those fields in um this this particular serializer. So if everything works fine you continue screening work more things. Um let me continue there. Maybe I just create too much. Um So that's that's the main thing that you can do with dynamics. So you can, for instance, create a static method to get uh location fields.

But you can get only another one method to return uh ID and name, for instance, if you needed to. So that's the way you can create those particular methods. That way you can save time, reuse logic within the serializer that you already create, and there is no code duplication. So what about getting better performance? As you can imagine, you we started pretty good, we started pretty fast, there's no issues, but the things uh started to get worse and worse when we get more users involved in our in our application.

So um let's see if that already finishes Come on, well, I'm going to stop this. Um and let's take a look to uh how it's it is it is working now. Um I'm in in the write one branch, so let's Let me switch back to the worst um scenario right now. Um previously I just read about um fifteen thousand elements um is trying to to render that information and you can see here the um how the How Django is making too many queries to the API, to the sorry, to the database

to get all the information that I am asking in this particular endpoint. So until Uh Django doesn't get all that information is not going to respond. So that's the reason we should start to to use uh prefetch. Um and prefetch what really does, um it it will create for us a different query. to a database to try to get all data that we need to make a representation um for a serializer really easy and in just one query if that's that the case For instance, um let's take a look to what

happens when you create the when you get ticket objects. But when you add select related What it's going to do is going to get all the fields uh related to that model in particular that you have relation. in in that model in particular. So here is the same query, but it now gets not only the information about ticket but also the information about user. So it's not going to go again to a database and make a hundred thousand of you uh of queries, but instead it's going to be just one.

So that small change is going to uh save you a lot of time. It's going to um save you a lot of head edges. So that's the important thing that uh that you need to take a look. How to get everything in in just one query. Perfetch related is going to be only one that which is useful, but um keep in mind that it's uh great for uh many-to-many relationships. And the previous one, as you we can see, we get where user ID equals two users. id. So that way it's making the relationship between them. Um on prefetch related is going to be a little bit different, but is

even though it's going to be better than not making any prefetch related. So uh in this case in particular it's going to do two queries. The first one to get the information about the ticket, and the second one is going to get the information about the events. But is going to select only the IDs that they need to get that to get the information that they expect So for instance if you make a filter for tickets and you are getting just a set of tickets that is going to create a different set of IDs that is going to query for the table event. So it's going to be way smaller. Also, something that you can do and you can improve

is to use the prefetch object. What uh really help you out is to get the information that you really need for that particular representation on the API. So if you get uh if you see here um inside the prefecture related, instead of having the string of the related field I'm getting a prefecture object inside a set uh string of the um attribute or the external or or the foreign key. and I can create a new query set and select the things that I really need. So with that dot only, I am asking for in this case ID and name. So the query is going to be

a little bit more smaller. And when you have a lot of data to to get in, it's going to to make the change. So let's take a look if that's already finishes. Yep, it takes two minutes and five seconds uh to complete it. So our user is gone from the application, it's not going to uh continue with us. Um so prefetch and related and select related is a must to our our when our applications are growing. Next that we can do is to start to make and filter. How to get our information in a better way.

One package that is really useful to get this is to use Django URL filter. Um you can set up um Two ways. The easy way is this one. You only add Django filter backend for um URL filter integrations DRF. And you can select the filters that you expect to get filters on. And with this in particular you will be able to get to do something like this So you can go for the endpoint events and you can filter to get a set of IDs if you need to, or you can get the name and make sure it contains any particular string that you expect to get.

Or if a related object has some particular name in this case, and and you can start to filter even more complicated things So it's going to help you out to get complex queries inside the the URL. But you can also make it with a class and use a model filter set that you can reuse on different endpoints. It's the same thing, the same logic. Um, it's pretty similar to to make a serializer, a model serializer. All right. So um but we already make prefetch, we already make a few things regarding filtering.

What about cache? Uh we can start to use uh the Django powerful cache that we um have out of the box. So to do it, you need to activate those middlewares with the red arrow arrow and make sure that the common field common middleware is in the middle Somewhere in the middle. Um and to make it work uh using Django Rest framework, you need to call these two decorators. Um variant cookie with the method decorator variant cookie and the um decorator to get uh catch a page. and you can set up the time if uh that you want and if you need to uh and I recommend it

to to make it uh select and create a K-prefix So that way you in you won't forget how to uh invalidate your catch your your cache at this point. So um Uh if you set up this um and it's working, it's going to give you a better performance, please don't not forget to invalidate catching. Cash in some way. So drop these lines of of code somewhere in your code to make sure that when something change the cache is invalidated uh at the right time and not after the time expire. Um but

There's something else that we can do to improve our performance using cache. One is a really good package called Django CacheOps. And what is going to help you out is to cache all ORRM transactions that you are doing and you can cache multiple query sets So um to use cache ops you need you need to to start to work in with using Redis. So to to use it you add to install apps as Many packages already, set up your Redis cache, and you can use these configurations to set up a few things. regarding what you expect to

cash ops to start to to make cash for you. And you can set up your own models and your applications to be cash. So you can define what operation is going to be cache by default, what is going to be the timeout by default on different applications and different models if you need to. And um you can define to um to create uh a cache for everything, but it's not quite recommended because you made up having cache on different things that you really don't want to. So if you need to you can also make cash.

You may you can use cache ops to to make it by yourself. So um as you can see we have um this get query set from uh MoldView set and you can define the your query as as expected, but at the end you append a dot cache and open a close in parentheses And with that, it's going to be cached in in Redis, and you will get the results, not for from your database, but instead from Redis So it is going to help you out with performance. But what about permissions? What we can do to improve our permissions. Actually, Django

uh already has a very great um permission system but it we can try to use something called dry permissions and it's going to uh improve our experience So um has any other package we installed with pip dry rest for um dry rest permissions uh added to our stallid apps And what I'm what we are going to do is to in in our view sets, we're going to add permission classes. This this line over here, and you can define if that's on a public endpoint or a pre um is required authentication but also add drive permissions

and now you can start to doing something like this So in your models you can define um a set of global permissions. In this case is these examples are It has permission to read, has permission to write, or it has permission to uh create. So you get access to their request. Up this point you can say you can get the user and you can make some validations. You can get uh require a permission, require an attribute inside the the the user or anything from their request. So um you can define uh different one different sets of rules for each and

one. And you can also have object permissions. So it's not only global, but also you can have access to the instance itself and make some validations The first example is going to make like the user is related to that particular object. And also you can require, as I mentioned before, I require a particular permission already granted for that user that is requesting access. Um but even though you we you can add also permission not to only the object but also to a field or an action. So if you

have a field and you want to protect to um get rights to a particular field, you can set up this method and you can protect this particular change. And if you define in a view set on different action, you can use it in uh with the name and you can make a restriction over there. And so all the logic regarding permission is going to be only on each model. So it's easier to find where the the actual logic is is there. But you can customize a little bit more. Maybe you want to share

uh a set of permissions uh within uh different uh models. One way to to do it is to create a custom permission. So this is um based on REST framework permission itself and you can um use it for different uh you you can you only need to implement the implement the method has permission and has some utilities. For instance, safe methods are going to be um get, meta, and options. So you are giving in this case access to make quer make requests to that methods, but it's not one of these methods. is going to require a permission before you can grant it access for

patch, boot, post. So um that that can help you out to add This permission to a different view sets and then uh you can protect in a different way that particular uh view set that you that you add that permission. But how to use then drive permissions when you are outside of a model view set? If you create an API view, you're going to see that it will throw you an error because there is not a key request. So important in those cases, you will need to create your serializer. uh uh with context is going to require to g provide a request otherwise it's going to throw you an error

because uh the model has not uh way to to get the the request and make um uh and run all your logic that you are defining there. And um Last one that I want to bring to you guys to to take your your API to another level is to uh use some renders. So um yesterday uh we saw uh this package working when uh using aut Automagic. So this is going to be another example. It's very very useful to and very easy to implement. So on your model view set, you only need to add

this particular class uh XLS file mixing and inside renderer classes um you need to add Xl renderer So with that in with that in mind, you can will y you can create Excel files just calling the um um the the API by creating by adding a new header or making explicitly on the URL. Don't forget uh if you need to, if you want to to get uh to continue having the API HTML view from your from Django Rest framework, don't forget to add browseable API renderer, otherwise it's going to be just JSON or

or Excel. All this code is going to be available on this repo. I'll give you a couple seconds you can take a picture. And let's take a look to how it works after we do all that things to to the SAPI. So let's take a look. So I'm going to change the branch to master right now. And um our previous response took two minutes and a little more. And now it's only one hundred and twenty-four mil milliseconds. Of course, it's paginated, but that's the idea to get quick responses.

Um and uh uh we are getting uh the event and the user here And uh you can see here, remember last time we try I tried to to do this. These are the logs to all the queries that the Django was doing to get that inform that representation And now uh this is it. This is what it 's really doing right now to get all that information. So it's a very small query. Um our DevOps team is going to uh Love us to do the kind of thing that that kind of stuff. But of course you can get even a a little bit more. It's going to take um

Extra time to to render. It depends on how many data you want to get. In this case, I'm getting All almost all that that's it? No. Ten thousand items, uh, but it's getting seven megabytes of data at this point and and it took about uh three minutes. So maybe that's not what you're expecting to to to get. That's not exactly maybe what you need, but you can define the limits using the Django REST framework pagination. So um well that's that's it for now.

Um Uh I I want to uh thank you to my team at UIT and also for the client that I work with, uh Building Engines. Um that's That's where all the magic happens and I learned that all these things. Um thank you.