Evolving Django: What We Learned by Integrating MongoDB with Jeffrey A. Clark

Speaker 1: Howdy folks. So as you heard, my name is Jib. I'm the current engineering manager for the Python team at MongoDB. and the engineering lead on the Django MongoDB backend. And for those of you who do not know what the Django MongoDB backend is, I'll be walking through it today. But specifically, I want to talk about what we've learned and how we believe Django is evolving, and we hope to help Django evolve. So I'll start by ripping the band-aid off. Django is not a SQL framework. Uh I really want to ask for the folks in this room who disagrees with this statement. Just by a show of hands.

Speaker 1: Wow, that's actually significantly less than I imagined. So I'm I'm happy to see that we've got a sort of more aggressive group. So Um funny enough, uh through my talks about this, I have witnessed uh, you know, a couple pitchforks, a couple academic theses, and whatnot to tell me otherwise. But For those who are sort of waiting with bated breath, please walk with me today as I kind of explain my point and show you how we've come to this realization that Django is not a SQL framework. Right, so first, here's the agenda. I imagine uh being at a Django conference with a lot of relational database management systems being the primary backend for this framework.

Speaker 1: Many may not be as familiar with what MongoDB is. And then as well, I'm going to talk about the deciding factors as to why we did this, then explaining the technical bits. Explaining how the sort of the combination of these two elements have really like led to the best of both worlds, the gains of the community, and leave time for questions. So first off, what is MongoDB? So uh first off, uh MongoDB is a uh NoSQL database. system, specifically a document-based database. And a key distinction about a document-based database is that all the information you're looking for generally is warehoused in what's known as.

Speaker 1: As the same collection. To many of you, that's viewed as a table, but for MongoDB, we call that a collection. Within that collection, rather than the traditional rows that you see in like RDBMS, we have our documents. And while I'm saying documents and rows, I think it's best just to give a good visual representation of what that is and what that means. So to the right to the right we've got uh the document model. Um as you can see uh we've got these you know, name, email, address, preferences, right? And to the left, we've got what? The relational model, where we've got the users table, we've got the addresses table. The addresses table has the foreign key to the users

Speaker 1: table, and then if you want to see all these things in aggregate, you do a good little join, you see your full information. And that's kind of generally how the dance goes, right? In the document model here, we generally try and avoid doing things like that. We're able to store scalar values such as arrays with their own sub-documents. So as you can see, we have name, email, addresses, that is an array of several sub-documents. home 123rd Main Street, Boston, right? And then similarly, we're able to do this multiple levels, store many sub-documents in many are What this changes is the fundamental way in which we look and query data and how we think about how data should be stored.

Speaker 1: And when we first tackled this problem, or when we first tried to tackle the problem of integrating MongoDB as a back end to Django, we thought, well, all we need to do is make it work as a database. And that was the fundamental mistake. Right. And I think that is the issue that several of these attempts prior have, you know, made. Like uh the you know Django Django Mongo Engine, Django Non-Rail, PyModem, Django. Some of these are you know iterations of our own. Others are warehoused by you know different companies or duh even by Django developers themselves. You know, for a lot of reasons we MongoDB was a little too young to to really understand or you know be able to tackle all the necessities uh of Django.

Speaker 1: But for other reasons uh we come to believe that It was because, you know, MongoDB viewed or the people trying to provide a solution for MongoDB and Django viewed it as just another database to put in. Right. And so as a result of that, maintenance issues developed, technical complexities developed, there was a clear lack of support. And then, you know, priorities change over time. It's been, what, 20 years of Django? you know, a little something over a decade and a half of MongoDB. So perspectives change, right? So the deciding factors for us now as to why we were doing it now was we've noticed that even in these sort of um frictional ecosystems, right? Opposite ecosystems. Folks were still substantially trying to integrate MongoDB in Jen.

Speaker 1: though. That means that they viewed, they believed that there was a world where this should work, right? And if people are willing to do the work, we should be willing to, you know, take them over the bridge. Right. And so because Django is still a popular uh web framework, because developers in the community wanted Monga to be with Django. And because we saw that so many third-party libraries were being built, even though they hadn't quite reached the vision that, you know, many of us would want. We figured it was it was it was time to really take a true crack at it. And so what that meant was really coming to understand the technicals of both MongoDB and Jang. them. And one thing we've learned is that there are a lot of similarities that come when you start understanding what Django is

Speaker 1: as an abstraction. And so I'd like to play a little game. Spot the difference. So we're gonna look at modeling, querying, and the extension of Django. And I want folks to see if they can spot the differences between our MongoDB solution versus the, you know. Elephant in room. Right? So let's start with modeling. Alright. For modeling, um, I wonder what people imagine when they say, oh, what uh my Django back MongoDB backend model. And my elephant in the room model, what will the differences be? What will they look like? And I'll tell you. Kind of look the exact same, right? And why do they look the exact same? Is because fundamentally what Django is doing is it's representing

Speaker 1: or defining what a model is, what the configuration of data is, how that is stored at the database level is up to the database in and of itself. Right? So we 're to keep going. This is how it renders at the database level. You know, remember back in uh back in the early example of the document model versus the relational model. For us, this is the this is our author document. We've got the ID, which is our primary key. We've got the name, which is author zero. We've got the address ID, which is a foreign key. It's just another ID, right? And then we've got the author city. And then similarly in the elf and in the room version, we've got the primary key, which is auto-incremented big end ,

Speaker 1: the name, Jane Smith, again, a foreign key. and author city, right? So because we know because we know that we can actually store data in much the same way, well what what what makes this different? Right? Well, for us and MongoDB, this right here is how we would probably prefer to store the data because it prioritizes uh quicker reads without having to inherently do a joint on the database. And then what that translates to is this, right? And so that's how it would manifest, that's how the database manifests it as a document. model. So to say the least, modeling hasn't changed. We've just added our little, you know, you know, what's that what's that uh extra bit, my little spice.

Speaker 1: Right? Okay. And the same thing goes with migrations. I think a big pitch in the early days was, oh, MongoDB doesn't need to do migration. It doesn't make much sense. We've come to realize that in building scalable, lasting enterprise systems, having a way to document and remember the changes in that database. are essential, right? Even if it's not necessarily doing anything, we've come to recognize from members of the Django community that having and maintaining and understanding a migrations file is essential. And we've even ourselves found functional use. So when you define an index, we will create the index on collection creation. So again, a modeling is looking and feeling much the same.

Speaker 1: Well what about query? This is kind of again one of those big topics. Like, well, you guys don't use the sort of SQL querying structure. Well, we don't. But to be honest, Django doesn't need really do that either. Right? So again, I'm gonna bring up this model, book model. Name, author. Great. Click. Look, the filter is the same. There's nothing different fundamentally about how you look at this filter. There's nothing different about how you are interacting or understand this filter. You are saying, hey, in this book collection, I want one of these authors. I just picked a random author and a list of authors that I created. And so that that doesn't change from whichever back end you're using, right?

Speaker 1: What changes, however, is the query that happens underneath. Um and that's the thing as the the the one of the purposes of the Django backend or the Django framework. is that we shouldn't have to worry about what the underlying query is. It should just work and it should just be constructive. It should use all the things powered by Django that are valuable. And so on the left, that is a MongoDB query. It's actually pretty easy to understand. Can you match on something with the author ID that has this specific object ID? And over there in SQL land, it's yeah, can we get an author ID in an app book that matches the specific string? Right. And again, forgive me if some of my SQL is poor. You know,

Speaker 1: it's not my specialty anymore. I'm sorry. But let's go even deeper, right? Let's get a slightly more complicated one. Let's say, oh, I want to do icontain. Surely the predicates aren't the same and now I'm doing a join on the foreign key of authors and I want to see something. different. As you can see, the construction of this is still the same. On the at the top level, it still looks the same. You're not learning anything new in the Django ecosystem. Underneath though, it does get a little nitty-gritty. I am not going to explain to you the one on the far left. It's quite involved, but again, that's the beauty of Django. You don't need to know that. That's for us. That's where we do the work. That's where we make it more optimized. That's where we make it faster.

Speaker 1: Right. And even to the right, I sort of barely know it. Right. It's like, um, and so that's where I go back to like what makes things powerful in our Mario B ecosystem is looking back to what we do instead of using a foreign key, we use an embedded doc, an embedded model. And so when we do that same query with an embedded model, what we get to generate is a much simpler, much faster query. And so again, we haven't upset anything about the Django ecosystem. We haven't changed anything fundamentally about how Django works. We just added some bits that that make it all flow, right? For us, for MongoDB. So great. Now fat last thing on the on this sort of spot the difference, and I hope you've spotted so many differences.

Speaker 1: right um is extension right again uh one of the features of uh like the sort of standard sequel is using dot raw right I know on the documentation it says, here be dragons, do not use this. Funny enough, in MongoDB we kind of encourage it because our API is continually growing and you're only dealing with sort of one system, Atlas, MongoDB's querying syntax. We say like please go ahead and use our raw aggregation. We have safety insurances and are when in the case that you accidentally do construct a query that doesn't make much sense, we have a robust error handling architecture that will tell you where something went in.

Speaker 1: miss that you can handle, right? So yes, you can directly use some more boutique bespoke MongoDB native things, similar to how you would use dot raw to use some more, let's say, Postgres bespoke or Oracle bespoke. you know, um type paradigms. But then there's more. We also extended it to support uh you know native polymorphism, right? I know some of you who may have used polymorphism in Django may, you know, use Django polymorphic, but through the MongoDB ecosystem, we We are able to, by just the ver by virtue of how our schemas work and how our data validation works, provide a type-safe way to say, I want multiple different sub-documents. to be queryable.

Speaker 1: And this all is queryable and works and it's um and will you know be error handled properly. in in uh in our Django MongaDB backend, right? And so we've done this in several different ways. Like we've got again, like I said, our embedded models, our subdocument uses. in making sure that our arrays and our arrays of embedded models are useful, in making sure our Atlas search, which is our full in-house full text search system, which you don't need to import another library to do. or vector search, which you don't need to use another library to do, or geospatial indexing, or geospatial querying. Or yes, even our MongoDB aggregation pipeline. We made it such that we can get all these things in without messing with the thesis of Django, right?

Speaker 1: Which is extensibility, which is, you know, um adding things in and bits and pieces. Um and so how do we manage to do this? Indirection. And what do I mean by indirection, right? I mean by exactly that. We know that there's a function that you call to render what you need, right? And so rather than going ahead and recreating the entire library from scratch, we poured through the code and we understood deeply what those functions are that we specifically need to override. Because Django has done a great job of making sure that these things are programmatic, that they're systematic. So when we see it somewhere, we know that it's happening in a proper, clear, and consistent way.

Speaker 1: And so when I say we do it everywhere, we do it everywhere. And the great part is because of this indirection, we are able to plug in into the Django native test suite. and know that even if a Django upgrade comes, we can run the test suite and ensure that we are still passing. And that, I truly believe, is one of the core faults of the previous systems, because they would think they just needed to build something for the the moment we are building something for the future. And so yeah, our Django models leverage you know registering these nodes and using the indirection. Our query set lookups register register these nodes Our transaction support, our MongoDB native transaction support occupies the same thing. Our admin dashboard support occupies the same thing.

Speaker 1: So truly name it. We'll tell you we've got some indirection system built on top of it. And the beautiful part about that is if it if it's a little clunky, if it's a little slow, we're able to continually iterate and improve on it. And so upon reflection, like we realized that one, NoSQL needs migrations too. Django is less interested in how the data is stored than we thought initially. right and there is no sea there is no single sequel to no single single sequel to no sequel path and vice versa honestly there's barely a let me not say barely Even from SQL to SQL, there are gaps, right? Many of you in this room, I imagine, have used Postgres as like the primary, but Postgres has several different extensions that aren't available.

Speaker 1: amongst others and again that's something Django allows and lets be right um then we've realized Django truly thrives in extensive Again, looking at these indirections and the like the this this concept of indirection, this the ability to just make sure that we are using the correct callbacks and they're providing. um what Django expects allows us to actually integrate well with other third party libraries, things like Django filters, Django Rest framework, our continued improvements in you know integrating with Wagtail and much, much more, right? Like much much easier to integrate ourselves into the system. And then above all else, we realize that, you know, Django is a full stack system. It is not just a SQL backend with some like batteries included pieces.

Speaker 1: It truly is a front end, a forums manager, a validation system, admin support, user management, so many different pieces that have several different third-party packages that have improved, iterated, or changed the the sort of ebb and flow of the system, right? And we've got to acknowledge that and realize that that's the commitment we're also signing up for, which we are very ready to continue doing. And also Django, as we've even heard this morning, right? Like the space of engineering is forever involving, right? And so the product will never be finished, right? And I'll touch a little bit more into the uh that that in a second. And finally, we don't plan to code alone. Every single time I've come up here, anytime I've talked to anyone, anytime we've put up any messaging, it's always please leave us feedback.

Speaker 1: We will respond and we have responded. We've made PR changes because of user responses. We've prioritized things because of user responses. We are doing our best to be part of the community. And so I like to say the Django Monga to be backend is now production. ready, right? Like I hope you guys go ahead and try it. Um because uh in this new you know Django non SQL slash NoSQL. Just in this new Django, neither SQL nor NoSQL framework. I hope you can find comfort doing whatever you feel fits your use case best. All right, and so now to kind of brag about MongoDB a little bit. So again, uh the benefits of using us, you get our document model, you get our integration with our data platform atlas, which has things like our advanced search AI capabilities.

Speaker 1: our aggregation framework system and a large suite of tools that come for free, right? And to install it's actually pretty easy. You just pip install Django MongoDB backend and then run your Django code. Okay. Um and then looking ahead to the future, yeah. We've already been thinking about AI. Right. And in fact, we've done many, many things to take um some of our uh acquisitions, which are recent acquisitions. acquisition Voyage AI and see how we can make this work already in a Django ecosystem, specifically leveraging the benefits of Django Monga to be back in. And so I hope you stay and look out for us in the future as we make these sort of first party toolings that Think about the future of not just web development, but database management, of just framework management, things

Speaker 1: of that nature. And then finally, yeah, like the community is what we grow off of. The community is what drives us to do what we need to do. We'd like to believe we're part of that community too. And so yeah, if you want to check out the back end, uh please check out this QR code or just do Django MongoDB backend on Google. Um and any questions?

Speaker 2: Thank you for your your talk. I want to ask uh in um creating the new uh uh uh packages to use Mongo in with Django. Uh have you found something to fix in Django or some feature to add in In January?

Speaker 1: Yeah, yeah. Um and I actually should have touched on this in our future work section, so I apologize for not. Uh one big thing for us is uh we don't operate by the you know, auto like big in auto ID field, right? And we think that that's a really big blocker for just NoSQL databases in general who don't necessarily use Big In or in auto fields. So one thing we're really hoping we can you know figure out is how do we kind of like restructure that default auto field setting to make it such that when you are using the model you are using it will just automatically know which like uh auto field to use. Another issue we noticed was the same thing manifested in tests because now we were trying to test our test suite.

Speaker 1: We have to fork the entire repo each time. to consistently then override that object ID auto field thing, like the auto field thing to make it all work. Right. Another thing that I've even talked about on like Django chats is Ease of startup is is is not is is actually not that great, right? Like if we could take what we've learned and like help the Jenga community and like help like allowing other developers feel more comfortable to enact change into these the system, especially upstream, that would be amazing. Like we've we've learned a lot because we've had to dig through a lot. And we have to dig through a lot and talk through a lot of professionals in order to get these. things working and we hope that doesn't have to be the future for Django. Yeah?

Speaker 3: Probably got time for one quick one. Yeah, thanks.

Speaker 4: Huh? Okay. Uh uh Elasticsearch also has a um uh document model. and is um MongoD the MongoDB document model, is it somehow, can you also use it for for full text search?

Speaker 1: Yeah, so uh Atlas Search is so our service Atlas is basically that built-in full text search system. Underneath it's using like the Lucene Analyzer. And what's even better is that you can do this locally. Like so we have a local like local versions of our Atlas deployment that you can leverage full text and even vector search on um to to test out these elements that you would even like things you would expect in like an elastic search yeah MongoDB handles um natively in our local in our free tier you don't have to do too much. And if you want to know more, feel free to drop by the booth. I'll give all the explanation on how.

Speaker 3: That's time. Thank you so much, Chip.