Prefetching for Fun and Profit by Mike Hansen

So a little bit about myself. I'm a software engineer at Rover. For those of you who are not familiar with Rover, it's an online marketplace for pet services like dog walking or dog boarding, grooming. Things like that. And Rover actually got started about eight years ago at a um startup weekend as a Django project. And so we've been with Django for many, many years. Um and it has continued to um service well. Um so today I'm gonna be talking a bit about uh prefetching. Um and over the last couple of years, There's been a number of times where we've had uh we've encountered some uh things in our code where it was maybe not necessarily

uh the best, uh but Digging down into Django's internals and learning about how the prefetch related implementation works, we were able to uh Uh come up with um I think good solutions to the problems we were having. So um today I'll give you a brief overview of what we're gonna be talking about. Um so the very first section will be talking about um N plus one query problems. Um this is sort of the um The main thing that prefetch related is trying to solve, so it would be good to just make sure that we're familiar with that so we know what we're trying to avoid. The next section is sort of a

prefetching under the hood section. And so the things that I um Hope to cover here are things which A are not sort of covered by the Django documentation and B uh focused on the things uh where the Django's prefetch related code interacts with um sort of external uh code. Um so these are the types of things that you'd want to know about if you were um designing uh some code to interact with Django's prefetch uh related system. And then uh given that, um oh, one thing I want to say is um that section will be probably fairly brisk, and it is not necessarily my intention for um

For you to sort of internalize every kind of line of code that we go over and present. The main thing that I want to accomplish there is just to um sort of call out what the main sort of moving parts are uh with prefetch related and how they interact. So even if you like don't quite understand a section of code or you know, forgot what uh covered earlier, like that's not a big deal. Um and so then given that, um knowing about how uh prefetch related works Um I'm gonna go over a series of what I'm calling case studies, uh but these are scenarios where um what Django provides out of the box Maybe doesn't isn't necessarily a good solution.

And so with a little bit of work, we can uh use um Django's internals to say come up with something better. So yeah, with that being said, talks are for the audience. And so I sort of wanted to cover the beginning, like what do I want you to get out of this talk? Because I think knowing that maybe will make this talk more effective for you. So given that this is a talk about pre-fetching, I want you to come away with a understanding of the key points. with respect to how Django implements prefetching. But I'd also like you to sort of think about uh your own like the

code that you work with on a day-to-day basis. And where what sort of problems or things you've incur uh encountered with respect to, say, data fetching or M plus one uh query problems. And then finally, um, I'd like you to see how you can use, say, knowledge of Django's internals to improve your own code. Um, and I think more generally, uh I think uh one way that we can sort of grow as engineers and become better software developers is to kind of dig into the sort of libraries or frameworks that we depend on every day, especially if um we encounter something where oh it's not behaving as you are expecting um then like that's a great time to say

hey maybe I'm gonna set aside some time to really dig in to see what's going on there, even though it may be quicker to say check Stack Overflow and find some you know quick solution. But I think being able to sort of dive down into the code that you work with is a great way that we can become better software engineers. So um first thing, like I said, we'll talk about uh M plus one query problem. And so what is this? Um so the M plus one query problem is what I'm calling a data access anti-pattern. Um and this commonly occurs uh when using ORMs. And so how it works is um an initial database query is done and it fetches multiple rows of data.

This is the plus one in the n plus one query problem. And then for each of those rows, an additional query or queries are done to fetch more data related to that particular row. And so this is the n queries in the n plus one problem. So for example, if in Django we uh run a query to fitch a list of 100 dogs and then for each of those dogs we want to say display the name of their favorite toy. Well if we sort of do it naively then you may you know, get into a situation where you run a hundred queries um to fetch all of that data and if each database query say takes ten milliseconds,

then you're in a situation where like you've already are spending you know one second of time during the request response cycle just to execute all of these database queries. queries. So that is what we are trying to avoid. So what does uh what do n plus one query problems look like? Um so I'm going to go over a couple examples of how these commonly come up in our code. So let's say we have a Django template here passed in as context to the template. We'll say dogs is equal to all of our dog objects. And then we do the initial query. So when we iterate over this query set, uh Django calls the database, fetches all of the dog objects.

And then as we go through and print out their name and the name of their favorite toy, when we access dog. favorite toy , behind the scenes, Django does an additional query. So the green, the top one is our plus one in the M plus one problem, and then the dog. favorite toy gives us the N different queries. Another example that where this comes up a lot for us in particular is using Django Rest framework. So many of you uh may have used this before. Um so this is a pretty simple uh serializer. Um f for right now we're just Serializing say the ID of a dog, and then we have a list view that uses that serializer.

And for the query set, we say get all of the dogs that belong to a particular owner. Great. So this doesn't have an M plus one query problem. Here we have uh Our sort of plus one query which fetches all of the dogs. Uh, but our dog serializer doesn't do any additional queries, so we're in the clear. But maybe let's say a month later you say, hey, our dog serializer needs to, like, we want to know about the name of that dog's favorite toy. And so we had say something like this. Um and here Uh when you specify the uh the source, we're saying, hey, that's uh the favorite toys name. And behind the scenes, Django will do a query there, and with this sort of setup

um you run into an M plus one query problem. And maybe some of these are harder to see initially because maybe you're you're uh defying your views in different files from your serializers. And so you don't necessarily know when you go and change the dog serializer that you have to change all of the views that use it to make sure that you fetch uh the dog's favorite toy. Um cool. So how does Django uh solve this problem? Um so we use prefetch related. And so instead of uh passing in like dog. objects. all to the context, we can call prefetch related and give the name and say, hey, we want to prefetch our favorite toy.

And then when we iterate uh through all of the dogs and print their favorite toy's name, uh Django doesn't actually do uh an additional query for each dog. So how how does that work? Well what Django does do is it does one additional query for each uh prefetch um that you specify. For in this example, it was the favorite toy. So kind of how it would work behind the scenes is that it would go through, fetch all of the dogs then iterate through that list, collect up the IDs of their favorite toys, and then execute a query something like this, where you select all of the toys where that uh toys ID is in the list of favorite toy IDs that you just just collected.

Great. So how how does Django do this? Like what is it actually doing under the hood? So that's what we'll talk about next. So we'll start at the beginning for how we specify prefetches. So every time uh prefetch related is called on a query set, uh what Django does is it returns a new query set. And then there's a attribute on the query set called underscore prefetch related lookups. And basically what it does is it takes everything that you passed into prefetch related and just appends it to the end of that. So for example, if we define our query set like dog. objects. prefetch related favorite toy. Um, we can look at this prefetch related lookups and see that, oh, favorite toy is there.

If we were to then go and add in say an additional uh prefetch related lookup to that, we can see, oh, the favorite toys manufacturer is now part of that prefetch related lookups attribute. Okay, so that's that's sort of where Django is storing um the prefetches that you want on a given query set. So then when the query set gets evaluated, say like you iterate over all of the values, um somewhere along the line, fetch all is called. It's an internal method. The first part here where it sets result cache, let's say if we were to iterate over

all of the dogs, result cache would be a list of dog instances. Then after it's fetched all of the dogs, uh it checks to see if you've um specified any pre-fetch related lookups. And if you have and it hasn't run the prefetching process already, then uh it calls self. underscore prefetch related lookups. So we sort of trace that through. Um and here, uh this internal method calls prefetch related objects. And it passes in the result cache, which is the list of, say, in our case, the dog um instances that we've fetched. And then

uh the it passes in the prefetch related lookups that we've sort of specified along the way. So we'll continue to drill down and take a look at prefetch related objects. So this is the primary function that uh Django uses to perform prefetching. So you can find this function in Django. db. models. query. And I have gone through the slides are available afterwards. I have uh sort of GitHub links to all the places in the the source code where these things are happening. Um but we're not so this function handles a lot of the kind of the bookkeeping uh process uh during the prefetch related process. Um

and so for example if you have you know like favorite toy double underscore manufacturer The preventulated objects function keeps track of, okay, these are all the favorite toys you've um You fetched and then now you have to get all the manufacturers for those. Um so we're not gonna be digging too much into that. Uh but one thing that uh prefetch related objects does do Is it calls um this get pre -fetcher function? Um and so the primary uh purpose of this function is to get a prefetcher. So that makes you wonder, okay, what is a prefetcher?

Well, a prefetcher is just any object which defines a get prefetch query set method. That's all. So for example, let's say we have our dog class and we access favorite toy from the class as opposed to an instance of dog. Then we get something Django calls a forward one-to-one descriptor. Okay, that's some object. But you can look and that object defines a getPrefetch query set method. So dog. favorite toy is a prefetcher. And so I think if there's one thing to take away from this section is this get prefetch query set

method. Like this is sort of the most important thing in defining how uh pre-fetching is going to work. Um so Yeah, if you can remember one thing, it's this get prefetch query set method. If you, you know, just grep through Django's code base, you'll can find all of the instances where Um Django site is able to or define some prefetching behavior. Um for when talking about uh Prefetch related, you'll see these things called descriptors coming up. And it's kind of hard to sort of work with um uh say defining new objects which are uh

defining new prefetchers without knowing about descriptors. And so a descriptor for our purpose is an object which implements double underscore get , or there are a couple other methods here that an object can implement to become a descriptor. And what they do is they customize what's returned when the object is accessed from an attribute on a class instance. So that's kind of a lot to parse. But like if we use Django, like we're using descriptors all of the time. Descriptors are what makes Django ORM work. So for example, when we have

a instance of the dog class and we access favorite toy, we get a toy model instance And in order to get that, you know, Django can go behind the scenes and fetch that from the database, instantiate it, and return it. But that's different than here when we accessed favorite toy from the class. And so how we get this difference in behavior is through this descriptor protocol. And you know, like descriptors are just everywhere in Python. For example, like this is how methods work in Python is If you have some function

and you call double underscore get and pass in an object, then what you get is a bound method. So that's like if you notice, you know, accessing a method from the class, you get a different type of thing when than when you access the that same attribute from an instance of that class. Okay. So going back um to prefetch related objects. So Now we have a prefetcher. So that is one of the jobs of prefetch-related objects is to find some object which implements get prefetch query set. Um and then uh prefetch related object calls a

like helper function called prefetch one level. And this is really where all of sort of the work that we care about for the purposes of this talk are is done. And so uh this function looks basically has this outline. Um so So what we're passed in is a list of instances. So again, this is in our example, this would be a list of dog instances. And then we get a prefetcher , which is the thing that we uh just just talked about. Um, and so that implements this get prefetch query set method. Um so the first thing it does is it calls that uh method on the prefetcher.

And so This is kind of the interface that get prefetch query set has. This is basically where all the sort of interesting stuff happens with respect to prefetching. And so what get prefetch query set is supposed to return is a six tuple with these sort of six pieces of information. And so sort of talk about uh what they are. Um so the first uh the first thing in the tuple is uh basically an iterable of uh related objects. So when we are doing dog. objects. prefetch related favorite toy, uh

this rel qs will be a query set of all of those favorite toys. Then the next two uh the next two things in the tuple are functions which take one argument. And I like to think of these as uh taking um basically returning something like I think of as like a join value. And so they return some value which is used to associate the instances, which are say the list of dogs, and the related objects, which are the toys. So the first one takes in one of the related objects, a toy, and returns the join value.

And so here our join value is going to be the primary key of the toy. So given a toy, we just return its ID. Then the next one takes one of the instances, so in this case a dog, and returns the primary key of the favorite toy. So these values , using these values is how Django is going to know, oh, I should associate this toy with this dog. And it's based on these like join values. The next three things, um that are returned, basically tell uh prefetch one

level or the prefetch related subsystem, where do I now that I know like how to associate, uh say a toy with a dog, like where do I stick that data? Um and so uh the first one is Whether or not there is a single toy object associated with a dog. And so in this case, uh each dog has one favorite toy, so single is true. The next attribute is a cache name. And so this is a name related to the prefetcher. And we'll see sort of how this gets used when we take these values and And store them on the instance. And then finally, uh

there is a Boolean called isDescriptor. Um and this uh applies in the case where uh single is true. Um we we'll see how this gets used, uh but it just changes the way the place that uh the related object is stored. stored on the instance. Okay. Um so this is sort of what get prefetch query set returns, and this defines really the interesting behavior. um for prefetch related. So we'll go through like later in the talk and see lots of other examples of you know this six tuple returned and how you can uh get different get different behavior in Django.

uh by customizing the values returned there. Okay. So we've called get prefetch query set. We get this six tuple. And then what Prefetch One level is going to do, it's going to go through all of the related objects and figure out which instance they're related to. So it's gonna take the uh so the very first thing we we do is we get all related objects, which we just call list on the first thing that was returned from get prefetch. uh get prefetch query set. And so basically this is the only thing that the uh prefetch one-level code does with um this rel

qs variable. And so it doesn't even have to be a query set, it just has to be some iterable. Um it Django doesn't rely on, you know uh specific properties of the thing that's returned there. It really just calls list on it and then uh goes from there. So in our example, this rel QS was going to be all of those favorite toys, right? We took toys and filtered them such that their ID was in those list of favorite toy IDs that we collected. So the next thing we're going to do, we're going to define a cache dictionary. And we're going to iterate over all of the uh related objects that we've gotten.

And we're going to get this join value. So here remember uh this rel obj adder was a function which took in a one of these related objects, a toy, and produced the join value. And so in this particular case, that was going to be the primary key for the toy. And then once we have that, we use that value as a key in this cache dictionary. And uh the value associated with that key in the dictionary is going to be a list. And it's going to be a list of all of the um Related objects, so the toys that have that join value. And so because uh Because in this instance the toy ID is a primary key.

You're at most gonna you're not gonna have lists of say more than two um more than two objects because only uh There is only one say toy per um ID. Okay, so we've gone through and we've built up this cache which maps our join values to list of related objects. And then the next thing that Prefetch1L does is it stores the objects on the instances. So uh Once we've fetched all the related objects, like I said, we need to store them somewhere. Um so it iterates over all of our instances. So these this is the list of dogs again. Right, here

you know we fetch that on our initial query, and we have that. And then here this instance adder is this uh function. um that we talked about earlier, which uh takes the dog and produces the join value. So here, like I said, this is the primary key for the toy. And what we do is we uh define get um this variable called vals. So we take our cache, look up um the see if the join value is in there. Um if it is we return that list of related objects. Um if it's not then we get the empty list. So now for every object we have a list of related objects.

So for every dog, we have a list of favorite toys. And here is where uh like I said those sort of final three values from that six tuple uh come into play. And so if single is true was returned from get prefetch query set, then we uh sort of fall into a conditional block of code that looks like this. So uh we know there's only going to be at most uh one uh r related object. So here we get that as a um singular vowel. If there's no uh objects in that list, we get none. And so

then we fall into three cases. The first one is if in the uh when you specified the prefetch, you sp specified a two adder. Um that is an optional argument that you can specify and you say, hey, once you fetch this. Stick it in this attribute. So we call set at adder on our object and to the two adder and we get our val, which is a toy. Otherwise, if this descriptor uh Boolean returned was true, uh then you basically set the cache name attribute on the object to be this singular value Finally, if none of those are the case, there's this fields cache dictionary which lives on the object.

And we put the value in that dictionary associated to the key cache name. Cool. So this is what happens if uh you return single equals true. You can control where prefetch related uh puts your um related object. Uh if single is false uh was returned from get prefetch query set, then we fall under two cases. Again, if you've specified a specific to adder when you specified the prefetch, Then you're going to set that attribute on the object to be that list of related objects Otherwise, if that's not the case, then we'd have to do a little more work here. But at the end of the day,

you can see on the final line You have a prefetched objects cache attribute on say our instance, like a dog. And in that you have a key as a cache name, and you set that to be a query set. And two lines above that query set you've set the result cache. on it, which says, hey, I've already basically fetched that this data and this is what those values are. Um cool. So that was sort of a lot. But if we go kind of over Uh everything that was done in prefetch one level is we called get prefetch query set.

It got this big s uh six tuple which defines what it's gonna do. We we took that data and went through and associated all of the instances with their related objects and then finally we stored those related objects on the instances. And how we did that depended on those values returned from get prefetch query set. Great. So once those are there, how do we make use of them? How does Django make use of of them. And so this depends on uh which particular prefetcher um you used. But in the case where we are we had this dog. favorite toy, this descriptor will call self.

field. getcached value, and this looks in the fields cache dictionary. To see whether or not the favorite toy has already been fetched. So the code in this descriptor sort of knows where the prefetch related code is going to put the related object. Similarly, managers on instances will check this prefetched objects cache within their get query set method. So you know if you were to do something like Toy. dogset. all, right? This is all of the dogs which have this particular toy as their favorite. That will check on the

toys prefetched objects cache to see if this query set that already has all the dogs there has been already fetched. So that was kind of the main um I don't know kind of landmarks in the in the pre-fetching process. One you specify uh the Uh prefetch lookups, those get stored on the uh prefetch related lookups on the query set. When it's evaluated, it calls prefetch related objects. This function goes through, finds a prefetcher. That's one of these things which defines get prefetch query set. And then the prefetch one level takes the data there. and sort of joins the initial objects you have with the related objects that it just fetched, combines those together.

and then finishes its job. And then when there's additional code, say in these descriptors or these managers that know sort of the right places to check to to get access to these prefetches. values. Okay, so that was um a bit of a whirlwind tour. Um but now I sort of want to talk about these sort of case studies. and like how how we can actually like make use of this information in our code. So the very first one I want to talk about is Like, let's say suppose we store our user model in a different database than the dog model. So here we have in database one

our Django user model. And over in database two, we have our dog model, and it has a user ID uh field, uh which is a positive integer. So we can't use a um foreign key because you can't have foreign keys across databases. Um Django like doesn't like that. Um and so but we would still like to be able to use this in sort of a a A way that feels um the like it's actually a uh actually a foreign key. So in particular, we'd like to be able to do things like, hey, for all of the dogs in user.

dogs. all, we'd like to print the dog and the dog's favorite toys name. But because we want to be able to do things like that, we also want to be able to be able to do things like user. objects. prefetch related and we want to fetch all of the dogs and all of their favorite toys. Great. So this is the um this is sort of our scenario. And so how do we do this? There's a couple of uh steps. The first thing we have to do is define a manager which implements this get prefetch query set. So this is our prefetcher. And this is the thing you'll get when you access user. dogs, right?

You'll get one of these managers so that you can say. count or dot all and do all that with it. And sort of the two functions we'll be interested in are, like I said, get prefetch query set to define the prefetching behavior, and then also get query set. So that we know like, okay, once the prefetching is done, how do we actually get that in use by the dog 's manager? The next piece we have sort of in this puzzle is a descriptor. This, like I said, implements this get uh double underscore get method. And when it is accessed from a class, so capitaluser. dogs, uh it will return the descriptor, but if it's accessed from a

uh say user instance, like lowercase, user. dogs, it returns one of these dog managers. And we'll pass in that instance so it knows which particular user you're talking about. And then finally, we set this uh dogs to be our dogs descriptor on the user. So going back and filling in the miss missing pieces. We have to define this get prefetch query set. So uh it takes in a list of instances. In this case, it will be a list of users. an optional query set. But the main thing we do is we go through all of the user IDs and collect them into a list.

And so here this uh list is called owner IDs. It's a list of integers. And we return the six tuple. And so the very first thing is going to be Basically like dog dot objects. filter owner ID is in this list of integers. So we don't even need to know about Like we don't have to have a foreign key there, right? This query executes on database two. You're just getting all of the dogs that have um their owner ID in this list of integers. Like Django can do that no problem. And now we have to come up with the join values. And we're gonna basically join on the primary key for the user So when we get a related object, like a dog, we return the owner ID, and we get a user, we return the user ID.

A person can have multiple dogs. So we'll return false for single, we'll cache it under the dog's name, and is descriptor doesn't really apply in the case when uh single is false, so we'll return false here. And this basically is everything you kind of have to specify to the prefetch related system in order to get the things that we want to do to work. And then finally, we also need to implement this get query set method, which the first thing it tries to do is access the Dog 's key in the user's prefetch related prefetch objects cache attribute.

So when we perform the prefetching, we'll get a list of uh dogs and the prefetch related system will put it in the uh for each user it'll put it in uh the dogs key in their prefetched object objects cache attribute. If it's not there, then we sort of apply the default behavior of the manager. And so now with those sort of things in place, you can do uh things like this, uh where this is only gonna uh do three queries, one to fetch the users, one to fetch those users' dogs, and then one to fetch those. uh dogs favorite toys. And so that's going to um basically this looks exactly like it would

if uh the dog objects and the user objects were living in the same database and you add a foreign key um for this particular uh pattern. So the next sort of scenario I want to look at is that, hey, there's really nothing special about those integers we used as a join key in the previous example. And so you can basically uh sort of join any two models as long as you can get values that are equal between them. So specifically, say, suppose we're able to define a dog's classmates as all of the dogs that were born in the same year as that dog.

Okay? And so in our code, like with this, maybe we want to iterate through all of a dog's classmates and print that classmates' favorite toys. uh name. And so here, uh if you aren't able to do any prefetching, you get a M plus one query problem. So I'm gonna sort of skip all of the part uh with the manager and descriptor before and just sort of go into uh the sort of main piece, the get prefetch query set method. And so how's this work? So we get a list of dogs and then we collect all of the years of their birth dates. So here, years will be a list of integers.

And then we define the data. So the first bit will be all of the dogs which have a birth date in one of the years that we've collected. And then we have to, how are we going to associate a dog with its classmates? Well, we're just going to use the birth dates year. And so here, because the the instance and the related objects are both dogs, we have the same um this same these same functions to get these joined values. Um and again, you can have multiple classmates. We're gonna cache it under the classmate's name, and uh is the descriptor doesn't apply. So, you know, other than all of the sort of um

kind of boilerplate to get the managers in place, uh this is really all you need to do to be able to pre-fetch all of the dogs that have the same. um birthday uh as uh your particular um dog that you're interested in. Okay, uh next uh one we kind of want to look at. Um so we recently had to add support for adding translations to fields in Django, uh Django model fields. Um Rover kind of exp expanded into Europe and we needed to support a lot more languages. And we have some models which store uh have fields which store English text and we want to be able to present that text to users in their preferred

language. So we went with a approach that uses one table to store all of them. So the main things in this table are a generic foreign key to the object that we're translating, a language code to specify which language it is, a field name to specify which field on the uh content object that we're translating and then value which is uh uh the translation itself. So one particular example of that in our system is a dog breed. So a dog breed has a name, and we want to be able to translate that. And so we can we can use kind of the reverse relationship between the generic foreign key using generic relation

and say we could prefetch all of those. But it does have the disadvantage that you're prefetching every language, uh the translations for every language, even though you may only need one. So we'd like to be able to have something called like active translations, where we only want to pre-fetch the subset of translations which correspond to the active language in in the sort of request response cycle. So if a user's preferred language which is French, we only want to uh fetch the tr French translations. Additionally, since a large percentage of our traffic uses English as a primary uh language, we don't want to say necessarily take this uh additional query overhead

if um if we're not actually going to use any of the translations. We don't actually want to do those queries. Cool. So how do we how do we get prefetch related to um do this? Uh So in our sort of the relevant get prefetch query set method, we do something like check if the active language is equal to the uh default language. And that's what we'll assume that the uh the language that the uh The values in the table are stored in. And if you're equal to that, then we're just gonna bail out early and we're gonna return an empty list here. And it doesn't matter really what we uh return for our join values because there's nothing to join. And in this case

we don't we can pre-fetch the active translations, but Django won't do any database queries. We don't have to go the database database and come back. And then additionally for that sort of manager, we get the query set and then we just have to take that query set and filter it by the active language. for um during the request response cycle. So we're able to sort of transparently get um Get field translations without having to do a whole lot of extra work and extra queries. And the final one I want to talk about is mainly more to uh showcase kind of the flexibility within the prefetch related

system. as opposed to like this being a pattern that I would recommend in your code. But let's say there's a third-party uh dog date service. Which provides a HTTP API that lets you fetch uh possible play dates for your dog. So a request looks like this. We make uh get requests and we pass our dog ID as a query parameter and we get back a list of dictionaries basically. Um and we have the dog ID as a Um key in there and then playdates, which is our list of URLs. Um and their HTTP API also happens to support uh multiple dogs per call. So if you pass multiple

dog IDs, you get multiple dictionaries back with their IDs and their playdates. And so suppose we have code which looks like this. In our dog model, we have uh uh dog dates external ID. So this is the for a dog in our system, what is the ID in their system? Um and then we have a uh play date available playdates property. Which when you access it, it goes through, makes a GET request to this URL, and then returns basically the playdates field from that response. And so and so now you have something like an M plus one like request problem. So you know you're not doing

queries to a database, you're making HTTP requests. And so if you wanted to go through all of the user's dogs and print the available playdates, then you get one HTTP call per dog. Um so hm, this looks like something that we can solve with prefetch related. Um and so we can actually do that. So in our get prefetch query set. Here we have we get passed in a list of dog instances. And so we can build up one of these URLs that has all of the um That has all of the dog IDs as query parameters. And then here in uh we return, well the first thing we return is uh we make the HTTP

call. Get the JSON response, that's a list of dictionaries. And now we have to find some way to associate that list of dictionaries with the dogs. And so we'll use that dog ID data in there. And so So the first one takes one of these dictionaries, gets the dog ID, turns it into an integer. And the second one takes one of our dogs and use uh returns the dog date's external ID. Um and we know that there's uh exactly one of these dictionaries per dog. So uh is single is true. We'll store it on a in say underscore available playdates. And now you can do something like this where you do dog. objects. prefetch related available play

dates and behind the scenes it will make a batch request. to this API. And then if uh and then you no longer have the M plus one problem that you would have uh if you didn't do something like this. So that was mostly what I wanted to go over today. This week I'm going to be working on getting some of this boilerplate code that you need to uh write in order to get your own sort of custom prefetchers into a third party package called Django Prefetch Utils. So I'll be working on that this week and at the sprints. And I'll be around if you are interested in things related to data fetching or prefetch related,

anything like that, I'm happy to talk on that. So thank you.