Django, what the JOIN? with Simon Charette
Published October 23, 2025
This video features Simon Charette at DjangoCon US 2022 in San Diego, California, USA.
This talk was presented at: https://2022.djangocon.us/talks/keynote-state-of-orm/
LINKS:
Follow Simon Charette 👇
On GitHub: https://github.com/charettes
Follow DjangCon US 👇
https://twitter.com/djangocon
Follow DEFNA 👇
https://twitter.com/defnado
https://www.defna.org/
Simon Charette reviews the Django ORM features that most changed how developers work, including `prefetch_related`, transaction management, custom lookups and expressions, subqueries, and constraints. He then examines persistent problems: duplicate rows and incorrect aggregates across multi-valued relationships, implicit versus deferred field loading and N+1 queries, and the limited ability to define reusable, application-level relationships. He argues that progress depends on creating better ORM extension points, documenting the internals, adding typing, mentoring contributors, and making incremental improvements that broaden the small group able to review ORM changes.
Summarised automatically from the transcript.
Automatically transcribed, so expect mistakes in names and technical terms.
All right. Thank you again for the introduction and thank you for everyone being here either virtually or in person. I feel very blessed to be able to be here today and celebrate the Django community with all of you. So before starting the presentation, a bit about me. I am from Montreal, Canada, and I've been contributing to Django for about 10 years. You might know me better, not necessarily by my name, but by my Charit S. And all on GitHub and the uh avatar that you see on the right. Uh I started contributing uh Django ten years ago, but uh when I started, I mostly focused on the French translation issue that we were hitting, and over time I slowly um dip my toes into the RM and migration stuff. As a day-to-day job, I work at Zapier
as a staff engineer. Not sure if you know about us. We're a no-code automation tool and uh the mission of the product is to make automation work for everyone. We're about 700 people working across uh 40 countries and we're doing so all remotely. Zapier um uses Python and Django internally a lot. We have a large Django monlet that we're trying to break into service that some of them are using Django as well. And uh we do make an extensive usage of the RRM internally against MySQL and PostgreSQL. So uh what exactly is uh ORAM? What does the acronym stand for? It means object relational mapping and it's the layer of Python. uh that Django provides that allows you to map entities to uh database tables.
So if you used um you've used it a bit in the past you you might be familiar with that. things it allows you to do, retrieve data, persist it, filter it, aggregate it, name it. DRM nowadays is quite powerful in terms of what it allows you to do. It is the abstraction layer as well that allows you to not too painfully switch between backends. So if you're trans trying to transition from one database to the other. uh the um abstraction layers that aren't put in place should allow you to do that a bit more easily than if you're writing raw SQL So today we're going to review uh what I believe were some of the most influence changes, uh features introduced in the DRM the past 10 years. We're going to focus on that and time period because that's the period where I've used Django
the most. We're then going to do a technical dive into some of the issues that are still remaining that I believe would be beneficial to solve and we're going to uh also try to look at what lies ahead and how we can get there in terms of uh solving these longstanding issues. So the features I've picked, there are tons of them. Like over 10 years, the RMF changed dramatically. It has kind of adapted to use, to allow new patterns or fixes uh issue that um user had in terms of uh and adding a lot of boilerplates to fix um common pattern when building a web application. So the focus of the features I I've selected is going to be on the ones that
brought the most momentum in terms of contributions. And the rule of thumb I used was What are the features that I would have a hard time using the RM if they were uh not in it uh today? So the first one um if you've used uh If you there there were a couple of talks even um in this edition of DjangoCon talking about the famous N plus one query problem. So uh the feature I want to look at was introduced in Django 1. 4 by LuPlan. and uh it's perfect related. Uh so if you're not familiar with what it does, it basically allows you to, when retrieving a collection, if there's a multi-valued collection of objects associated to it, you're able to retrieve it efficiently by doing a single query. Prior to the introduction of this feature,
it was possible to fetch all of that, but again, you needed a lot of boilerplate to do that and not all of the um parts of the framework that are heavily coupled to the RM such as forms and um even Django Rest framework were were playing uh which was not around at the time but uh other um uh API framework didn't necessarily play well uh with this uh kind of boilerplate. So uh what I found interesting about this feature is that yes it solved a real problem but the way um features were kind of like built on top of uh is a a good success story in terms of contribution. So when it first landed it was a very minimal implementation. Uh you could pass strings, I would fetch multivariate relationship, but that was it. You didn't have a lot of in the
way to um configure the the queries that were uh may being made there. But over the years, if we look at the the release and the feature that were added on top of it Under the UD in one in 1. 5 , there was a lot of refactor relationship caching that took place, improvements to make it work better with select related and Uh nowadays I I have trouble imagining using Prefetch related without the Prefetch primitive that was added in 1. 7. Um and the I the API uh at at the point at the point of release of 1. 10 had matched so much that What was considered a private private API was made public through prefetch related objects. And these are all features that were incrementally added over the years. And um
there are still things, improvements that are being made to the feature in the latest release and things that are still need uh still need to be fixed. So yeah, to me that was a big success story in terms of ship the minimal API and let the community decide in terms of like where we want to go and how we want the feature to evolve. The second feature I wanted to look into is uh transaction atomic and the large uh refactor that came to allow it to work. So prior to uh its introduction in Django 1. 6, the transactional link was very hard to wrap your head around, at least for me. You had to use APIs to manually rollback, commit
, do proper error handling. That was a very large API and that was um hard to wrap your head around. So uh yes, uh transition atomic uh is nice and something we we use a lot nowadays, but um I think the the biggest change there was in terms of defining what are the expectations if you use the public API in terms of transactional link and here's how you can go off-road and if you go off-road here's what to expect. uh in terms of things that Django won't do for you anymore. And that is proven to be very useful if if you've had a chance to do any form of triage against issues that are reported against transactions, the boundary between what's supported and what's not is very clear. So we can guide user in terms of defining is this a bug
or is this something that um well you turn off man uh transaction management, so you're kind of on your own here. And there's also a um a pattern that you see in other features in Django where they start to mature outside of the framework and they eventually get merged. So uh you might not know about this, but the atomic uh context manager that you're you might be familiar with. is something that came from like AMRIC took inspiration from another solution that was handling transaction differently in the past. And the same can be said about transaction on Comet Hook, which was maintained as a third-party application by CalMeter prior to being merged in core.
The third feature is what I believe was the most influential in the the recent years of um uh around the ORM. Um it's the introduction of custom lookups, transforming expression, and their documentation. Uh to me it completely liberalized the access to the RM. Prior to that, um it was very hard to wrap your head your head around around about uh how to um do anything beyond what was already included in DRM in terms of lookups and uh and transforms. So if you wanted to do anything that was not already bundled in this uh big monolithic rm that didn't have a lot of entry points, you need it to drop into extra in with all of its spitfalls and the way it needs to um You need to be very careful about how you make reference to objects and how you uh include joins.
Uh it Formalize the APIs in terms of what is a resolvable and what is a compilable. So how can you have a reference to a to like just pass a string reference to a model field and have it result in a join, in a column. And here's how you can actually have the objects bit SQL and have it used when this the query generated So yeah, it made most of the usage of uh extra obsolete at that point as well. And um from that point we've we've taken taken a stance where um We've documented that if you have any usage of extra that is not covered by this particular API, you should report it as a bug because we want the expression API to be flexible enough for you to not have to drop into
RASQL through Extra. Something we saw as well when uh this feature was added is kind of like the long tail of contributions, both in terms of bug fixes. We in the end we we still move a lot of stuff around. So there were a lot of bob fixes and optimization features that came in terms of contribution. So what you see here is a breakdown of the track ticket by their kind. uh the ones that were attributed to the ORM framework. Um and uh these bug fixes since the API was kind of clearly defined in terms of what is a result ball and compilable, uh it made it easier for new contributors to come in and uh f uh fixed bugs that would have been very hard to fix otherwise in the uh in DRM
because the the code base was so intimidating. The next feature is what I would qualify as expression v2, where we took the concept further in terms of choosing subquery, exist. uh filtered expression, these really uh build on top of the the giant uh work that went into introducing expressions. So uh if you think of uh alderf um that is used when you create a subquery to refer to an alter column. That wouldn't have been possible at all. if the uh expression um concept hadn't had not been uh existing. Uh it also provides you a way to have more control of joins. Filter expression does that. Prior to that, there was no way to tell DRM about, oh well, I want to be using joins in a certain way.
And that's something that uh users that are more familiar with SQL have been asking for a while and just give me a way to Tell the RM how it should uh join the data. It also made even more usage of extra uh obsolete by having more uh ways to uh do joins and have a bit uh way to inject subqueries as well, which sometimes the the queries that the RML is going to be generating uh aren't are going to be very hard for query planners um the way database interpret SQL and actually retrieve the data for you to make it efficient. So if you have more ways to help the database into retrieving the data the data efficiently. uh you need to rely less on extra and that's something this uh feature did
and uh fortunately or unfortunately uh these um Expression are still required nowadays to work around or circumvent long standing or IM issues or limitation, and one of them we're going to go into very soon. The last one is kind of like more of a big bet. It was introduced relatively recently in 2019. It's introduction of constraints. the it kind of built on top of um all the work that went into expression but it took it a bit further in the sense that um previously in expression I had already uh always needed to be resolved in the context of a query, but in this case uh it was resolved in a very limited scope where join were not allowed and you were not allowed to refer to columns from other models
and and so on. It's all it's also paved the way for possibly deprecation deprecating Unique together in the future. I don't know if it's something we try doing with uh indexes. um by uh by trying to deprecate the index field on constraint. And at all I think it also paves the way uh to revisit model validation, which is a kind of like a contentious point of discussion. about is it a good thing, is it not? Because once you get an object that points at a constraint, you can kind of like switch the the way you deal with data validation that is enforced by constraint the database you If you try to insert a data and an integrity error is thrown at you, well you at least have an object in the Django realm to uh surface.
You don't need to do prior validation. You can rely on the database and turn that into uh proper uh message for user consumption. So this was a review of features that I thought. Change the way we use the RM. There 's a lot of them, so if you've contributed to one and that it was not mentioned there, it's not because it's not important, it's just because uh there was only a limited amount of time that could be spent on this. So now that we've looked about the good parts, I think we think it's worth looking at things that we could do better or things that we've known are not so good that we know for a while. So the the first one I want to look in is the way Django deals with
multivalid relationships. So to understand what a multivalid relationship Say you have a book model that refers to author, that's like the typical example that is usually used to explain foreign keys and how they are queryable through the RM. So uh assuming the book has a foreign key to author and a second one for co-author, that is uh optional. If you look at the relationship from the book perspective, the book has a single author, so we'll see that the relationship is single valued, but If you look at the relationship from the other side, the author can have multiple books, so we'll say that the relationship is multi-valued. And this is what we're going to be looking into.
So the way Django deals with multivariate relationship is surprising to a lot of newcomers, or at least the ones that I've had the chance to. to mentor or introduce to Django. When you try to either filter or order by or annotate against a multivalid relationship, you're going to get duplicate results from the base model you're querying from And usually the um when you when I'm I'm uh helping newcomers with Django, I the default option there is just use this thing, right? It's you're going to uh reduce the the set of objects that are returned. And in most cases it's just going to work. But that's still a very surprising default that only happens when dealing with
multi-valued uh relationship there. So if you look at the example, if you were trying to filter out the author and have a book That contains a particular keyword. If the author happens to have multiple book matching the criteria, then you're going to get the author multiple times in the query set that is returned to you. And there are ways the ORAM could could do a better job there in terms of like preventing dupes from uh being returned. Um These are known. But the thing is, this feature has been around for so long that uh taking it away uh as a lot of impact. We need to account for that. And it's somewhat desirable as well. So if you're using um values or you're not dealing with models, it might make sense. You might want to return like all the books
from that. It's also desirable in certain context uh in case of aggregation. So if you're trying to retrieve all author and you're trying to annotate them with the the number of books they have authored In this case, the usage of an aggregate function would result in a GRUBI. So the duplicate results will all get merged together and you won't run into this particular issue. But this feature is useful till a certain point because there's this long-standing issue where if you use more than uh one aggregation, uh if you use If you refer uh more than one multivariate relationship when either during filtering and you try to uh ordering and you try to um You try to to uh retrieve the the results, uh what you're going to get is not what you would expect.
Um so in the case of trying to retrieve all the author and trying to retrieve uh all the the uh the count of books and the counts of books they um they have uh authored and co-authored, the um both the book count and the co-author count are going to be the same value. They're going to be a product of each other. And the reason for that is the way the RM builds joins on tops of joins when building with a multivalue relationship. You're going to start from an author row, which is going to be turned into duplicate books, and each of the these duplicate books are going to be turned into So what you actually get. are dupes of dupes and the accounts that you're getting are going to be the product of each other, which is not something you would uh the user would expect, at least for Django to do by default.
And the case of count, it's it's workable. You can use distinct. You can use kind of like the same work round. Um, but it's not possible for aggregates that don't deal with um uh unique values. So in the case of average or sum, that's something you you couldn't use this thing there because it doesn't make sense. And uh it's also an issue that is kind of hard to diagnose. the data uh will uh the bug will only manifest itself if you're dealing with a a data set that uh so if you're dealing with multiv value relationship but most of the time there's a single item it might be very uh hard to diagnose in which case the uh query that is generated is not right. And um with a subcurve expression it is possible to work around, but it's still kind of a foot gun
uh in terms of uh default behavior So for this particular problem, multiple backward uh multiple solutions have been identified, but they're mostly backward incompatible. So we could warn or error out when multiv more than one multivalu relationships are involved in queries. We could group by even when no aggregation is involved, so no dupros are returned. There's a lot of a solution that could be taken there. But the the big push over the years has been trying to teach RM to be smart enough to don't do these kind of things. So you don't have to worry about it. But it's a very hard problem and the fact that this ticket has been open for so long speaks for it. There's a lot of ramification.
So if you wanted to use Exist, for example, there are some backends that don't work so well with that compared to using join. The um second long-standing issue I wanted to discuss uh as recent a bit uh received a bit more vo focus in DjangoCon Europe uh this year. Um so I want to go over it uh quickly. Uh so To provide you a bit of context, the way retrieval is performed when using query set of relationship are going to be different by default. That means that if you're retrieving book, the author that's associated to them are not going to be joined against or prison by default. And if you want to change that, you need to opt in using select related On the other side,
fields are always selected by default. And if you want to prevent that from happening, you need to uh opt in opt out using uh Difer only. So If we take the same model and we look at what happens, if you you try to uh select all of the book, uh you will notice that um there's no joint, no implicit joint that's going to be perform from the URM. And what that means is that if you try to access the author that is associated with a book, this will incur a next draw query in order to do so. And you need to opt in if you want to prevent that from happening. On the opposite side, if you um differ the title, um so you opt out of selecting title. And you happen to push the book through three layers of serialization, and it happens to be accessed later on
down the abstraction layer where you're passing the data, you're going to get an extra query by doing that. And this is usually referred to as the n plus one query problem. So if you were trying to iterate over a collection of book by opting out. For selecting the number of name and you're trying to access both the author of the name, you would get one query for retrieving the set of objects, and to query for um dealing with the attributes that you've um opt out or forgot to opt in in order to to select. And uh while there are a lot of third party um solutions to kind of make this problem um less of an issue what the core provides are somewhat limited. So if you one pattern that is used to prevent that from happening is to have very good code coverage and
use assert num queries and you hope that there's no cache property somewhere or global state that is in your state and then you need to make sure the f the cache is cleared to uh uh that that to make sure that your tests actually provide some value and are not just very flaky as your product evolve. Um you can uh have a middleware that monitors number of queries. There are there are a lot of tools that are outside of core that um make the experience I would say less miserable, but that uh also that at least provides you some way to uh to fix this problem. So for what I'd like to look into is possible ways that we could make things better in core. One of the solutions that has been proposed and has been received a bit of focus is to uh
automatically use prefetch related uh when a n plus one query pattern is detected. So that would mean that um so you can look at at this project on Django auto prefetch and there's also an open uh merge request for it But uh what it bas basically means is that if you were to iterate over a collection of book and you try to access the author for it instead of solely fetching the book, um the whole collection of author associated with the um uh the the the peers or the collection of book uh where the the uh the book was retrieved will all be fetched in a single query. Uh so that would prevent this uh this problem from uh happening. It would kind of be an implicit or after the fact prefetch related uh call that that would be made.
There are cons to at like for the for this particular approach, at least at using it as a default. Um not sure if you've used Perfetch Little in the context of uh of a system that deals with a lot of multicont uh concurrency, but um since the data is retrieved in two-step The state of the database is not frozen between, which means that when it gets combined through prefetch related , some attributes that would you would expect to be there could result in exception when trying to access them. In the case of book, um if you happen to fetch all the book and you uh you try to access author and you view per schetulated to fetch the author Nothing guarantees that you won't get the does not exist when accessing the author if the author has been deleted in between. And usually it's not a problem, partially if you're dealing with data that is not being altered a lot.
Uh but if you're dealing with a data set that is evolving a lot over time, uh you you might have run into this this issue. Um and something else as well is that It kind of encouraged lack practice in terms of data access pattern. Um there are very legitimate cases where you want to fetch all the data up and you want to be explicit in terms of how it's accessed and trying to automate most of that I think is kind of doing a disservice to at least some users that could use DRM as a way to learn more about SQL. The other solution that I wanted to look at is the a way to uh log warn or error out and placement query. So Instead of having DRM aptly fetch data for you, you would at least get a warning or something you can take action on.
So there are two packages that have uh experimented with that. So in the case of book here, you would get either a warning or an error when trying to access an author if you've not select related it preemptively. Same thing with a field where you would have opt-alve. uh it's uh selecting and that's a pattern we've we've used at Zapier uh when building a GraphQL endpoint that was translating GraphQL into uh ORM queries where we wanted to make sure we We didn't miss anything. And the way we've used this pattern is to have warning emitted when an attribute is accessed. And just like when you try to update update from one version of Django to the other, uh there are ways that you can elevate the warning to an error and this is something we do when uh running the test suit. So uh
we can catch the issue um when running the test suit if we have coverage. And we pipe the warnings into a logging facility in production. So we don't get exception in production, but we also get a way to know that we've missed something with our tests. But there comes to the solution as well because it doesn't actually solve the problem. Yes, it's going to yell at you about, oh, you've done something wrong. But that's maybe not the user experience we want for all the newcomers. They might be way more happily in for when you're prototyping a project or um you're not actually interested in knowing about all about its QL. uh you might be more interested about having the framework take you by the end and make sure that um you can differ this problem
uh to later and it also requires some form of opt-in to maintain backward compatibility, partially in the case of raising exception. So what I think we need to do here is kind of like step back and think of a generic solution. If we look at what other frameworks do and the Python field, SQL Alchemy allows you to define a lazy field option. uh for defining a per field strategy and allows to retrieve that um or add a context manager or some form of like query where you could specify default. So I think that as a framework, we know about like the old landscape of solution, but what we should focus on first is adding a proper entry point. And then from there possibly ship a couple of solutions in core and um
let the uh ecosystem of third party application use uh the entry point and provide feedback in terms of uh what uh I think should be done better. Um and something that surprised me as well when I did the research for this particular issue is that this problem is not new. Like a lot of problems with Django, they've been discussed a lot in the past. And uh I'm gonna I'm gonna share to the slides afterwards, but there is a Django developers discussion that dates from uh 2013 that talks exactly about this problem. The last long standing issue I wanted to look at, um Was um how Django is kind of somewhat limited in terms of how it allows you to define
relationship between entities and models. Sure, if you want to have a foreign key or you know exactly what your database model looks like, you're going to be able to use foreign key, one-to-one field. Many to many field, these are certainly very useful, but uh when you defining an application, there's a lot of business logic that goes way beyond the the constraints that uh you'll be getting at the database level. And the introduction of filter relation kind of dip its toes in there where you could, when you're generating a query from an existing relationship, um Take it and limit it to a smaller scope so you can kind of like embed some application level in there and reuse this alias when building queries. But if you look at what um
Third party packages have done in terms of pushing the idea further. Django relativity and Django Reverse Unique, which uh was maintained by NC, which is one of the uh the Django core contributor multi-to-DRM. This I think this is something that the RM would benefit a lot from having. So what I mean by that is that if you you're defining a book It might be useful to have analyses that encapsulate this concept of give me all the books that were published the same year, or in case of like you have a lot of translations that are mapping back to a book. you might want to limit them to uh the current activated language, uh but you might want to do that at the model level because when you do it at the model level, the particular alias
uh remains available for all of the uh ORM utility. So if it's defined the model level it means oh you can do aggregation on top of it, you can do select related on top of it, you can do filtering, prefetch related Um so having it available at at such a low level uh allows for a lot of um reusability and um possibly very hard um uh relationship uh definition and constraint uh encapsulation into a reusable alias. So it does improve code reasonability. And uh I feel that it's also much easier to reason reason about complex data models. You can embed all that logic on the model level and From there, you don't have to make sure that, oh, um when I'm using this very complex um
Q object predicate when uh building this filter, oh did I miss uh did I use the correct lookup? uh in this particular call-out in my application. And one of the big benefits of it as well is that you inherit relationship caching. And once you have that, all of the the parts or ecosystem that revolves around the RM can benefit from it. And an interesting point about that to to finish uh on this feature is the the fact that Most of the machinery is already available in the code base. If you've used foreign key and you've accessed it through the reverse relationship, like we did when we accessed books from the uh author model previously. the um it's already something that the RM does internally.
So it would mostly be a a manner uh a matter of making sure that these API are ready for um public uh consumption but uh most of the work is already done it's more about pushing it through at the finish line making sure that again we provide the foundation and we allow the the community to build on top of it And so I went over three features, but I there's a myriad of them, like there's so many of them that have been around for so long. So Composite uh primary keys support for uh multivellus is something we could do now. Um form of generated fields, db default, db on delete. If you go on track and you look at these things. If users have asked for that for a while, uh the reason for it is that it's yeah, we're a limited number of people
and um it's very hard to uh commit to a feature, but not only adding the feature, but uh committing to the long tail of issues and regressions that are going to be to be caused by it. So what are the the current state of things looking at at all of these features around? It does take a long commitment to see this feature through. You when you come, you have to uh advocate for it, you have to show that there was some form of like adoption or that the the patterns that are that are used. I've been proven to work in a lot of contexts and it's not just kind of like anecdocal uh anecdotical evidence that it works for you. Um And uh there just a small number of individuals were able to review RM change.
Our fellows are doing a good job about doing that, um, but it needs to go beyond that if you want to ensure that there's proper diversity coming from different uh background that are using Django to make sure that we're just not short-sighted on uh making a particular change. And um some of the issues there are just stalled due to uh a lack of consensus It's very hard to um move things forward when you have just a small uh limited number of individuals. And the last thing as well is that Even if expression and uh introduction made things better, the RM is still perceived as this undesifiable um part of the uh of Django where it's this black box, you get the models, something happens, it goes to the base. Something happens, you get back the data.
But I think we can do a much better job into making this at least perceived less of a black box because it's n way better than and used to be And this is what I I want uh to look in in into here. Um if you look at the number of contributions, so this is coming from um Git commits. That were uh made partially to Django DB models. Uh we can see that there's a decline, but one thing that remains constant is the uh or somewhat constant or in the the same order is the number of contributors that uh have over two or over ten contributions. And usually if you want to get familiar with the internals, you you need to be uh keeping track there and it's uh it's an over the years it has been a constant even like when NC
or Malcom was around there it there's just not a lot of uh individuals who are able to um or are committed to uh reviewing these kind of changes. So how can we change things? Uh how can we make things better? Um I think that the problem is not um different than um than um uh Then when you enter it or you move to a company or enter it a very complex code base, right? You uh that is pretty documented in some sense. So um this is the approach I I've taken or this is why I perceive how we should approach a problem I think the first thing we should do is just document the RM internals. It doesn't have to come with this, oh Django will support it going forward, but at least just supporting uh documenting a bit of it
and uh putting this big disclaimer on it saying, oh, this is still somewhat of an unstable API, not a private one, but a an unstable one. And we're going through feedback on it, could do a long job into onboarding new contributors that are interested in making things better. I think it could also highlight uh the currently undefined behavior that we have in the RM because sometimes when uh we add a feature, um we try to think of ways of oh this feature is going to interact with this. other one and we need to uh add some test coverage to make sure that uh it works well. Um but When some issues are are brought up, it takes a bit of time to define, oh well, what actually should happen there? So I think it could go a long way documenting the internals
into um making sure that we were more focused or aware of interactions between uh components of the RM. It's it would also be a nice way to kind of rediscover or formalize reasoning being behind years old decisions. There are things in the RM now that if you go through you do some form of like um Git archaeology, so you go to track, you go through mailing list. you do git blame, um you will get all you need to understand why the ORM is is made in a certain way, but it's it's so long. It's not something we should expect contributors to uh to go through if the uh we expect them to make change the to the RM and finally documenting it would would be a a way to define the expected behavior beyond at the test suit.
Because when we make change to the RM, most of the time it's Oh well, does it pass the test suit like the change you made? But I think that it should go way beyond that in terms of uh providing a quality, uh enhancing the quality of the software uh we we ship. And um another another way to do that, uh having had the chance to deal with uh code base that were hard to approach is Start by adding typing to Django DB models internals. Uh I was someone that was against uh the the DEP four eighty four when it was first uh proposed uh partially because I I feared all of the complexities of uh um what comes with typing was something I was not that used to at the time as well.
But if you look at it now, it has mattured a lot. And partially the introduction of typing protocol is something that works that would work very well with Audi ORM, does a lot of like duct typing with regards to resolvable and compilable. So I I've kind of like changed my mind on the subject, at least for the internals, because I've seen the benefits. uh internally uh at zapier and and other um in other projects i I work on. And it will also go a long way into reducing the long tail of Type error and attributes error that we always get when we do uh release candidates or we um we release the the first version of uh a major release. These are things that are avoidable by using typing because the the linter will yell at you if you forget about a um
uh uh about a particular case. And the last one, again, is mentoring. If we want to increase the number of individuals we know about the URM, we need to have the ones that understand it now. uh pass it along and and make sure that new uh new contributors are able to get on board it and are able to um uh provide uh value to to uh to the the the community at least through through the rm um so uh i think we should focus on incremental improvements uh and refactors and ideally we'd get to a point where uh we if one individual is interested or one or more individual is interested in um contributing to the RM having a way of um following a bit the the summer of code.
uh model but per release uh would be uh something that I believe could work in terms of like commitment from the small number of individuals who Currently review change to the RM and this is something I'm I'm interested in doing at least one individual per release So thank you very much. This was my kind of like naive take on like how we we can make things better. I know that it goes way beyond uh these uh these words. Um I'm happy to discuss it further offstage and during sprints. I'll be there during the sprints if you want to speak ORM, you want to look at the RM, you have a completely different take on the on the subject. Yeah, so thank you for your time.
Django’s object-relational mapper maps Python entities to database tables and lets applications retrieve, persist, filter, and aggregate data without writing all the SQL directly. It also provides an abstraction that makes switching database backends easier.
Discussed at 1:07It replaced a difficult-to-manage set of manual commit, rollback, and error-handling APIs with a clearer transaction boundary. It also makes the line between supported transaction management and situations where developers are on their own much easier to understand.
Discussed at 5:48They opened up the ORM by formalizing how expressions resolve references and compile to SQL, making it possible to extend lookups and transforms without modifying a monolithic core. They also made many uses of the less-safe `extra()` API unnecessary and made ORM internals easier for contributors to work on.
Discussed at 8:13They provide more control over joins and allow subqueries, `Exists`, and filtered expressions to be represented directly in ORM queries. This can help databases execute complex queries more efficiently and further reduces the need to fall back to raw SQL or `extra()`.
Discussed at 10:35Constraints let Django express database-level rules using the expression machinery, while opening a path toward replacing patterns such as `unique_together`. They may also improve model validation by allowing database integrity errors to be turned into useful, user-facing validation messages.
Discussed at 12:09Joins across relationships such as an author’s books can produce multiple rows for the same base object, so filtering, ordering, or annotating may return duplicates. Removing that behavior is difficult because it is longstanding, sometimes desirable, and would be backward incompatible.
Discussed at 15:14Django can build joins on top of joins, producing duplicates of duplicates; counts from separate relationships then become a product of the two relationships instead of independent totals. `distinct` can help with some counts, but it does not solve aggregates such as sums or averages.
Discussed at 17:36Possible approaches include automatically applying `prefetch_related` when an access pattern reveals an N+1 query, or warning or raising an error when a relationship or deferred field is accessed without being loaded. The speaker favors first providing a general entry point so core and third-party solutions can evolve without breaking existing behavior.
Discussed at 23:00They could encapsulate business-level relationships—such as books published in the same year or translations in the active language—as reusable model-level definitions. Those relationships could then be used consistently for filtering, aggregation, prefetching, and `select_related`, while benefiting from existing relationship caching.
Discussed at 29:12The speaker recommends documenting ORM internals as an explicitly unstable API, recording the reasoning behind existing behavior, adding typing to the internals, and mentoring new contributors. Incremental improvements and a per-release contribution model could broaden the small group currently able to review ORM changes.
Discussed at 34:28Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.
Published July 15, 2026
Published July 15, 2026
Published July 15, 2026
Published July 15, 2026
Published July 15, 2026
Published July 14, 2026