Evolving Django: What We Learned by Integrating MongoDB

Speaker 1: Hello, hello everyone. My name is Anaya Raisangani and I am a developer advocate with MongoDB I am super, super excited and also very honored to be here in front of all of you to talk about what we learned with integrating MongoDB and Django inside of our new Python package. So I recognize a lot of faces in this room and I'm sure I've seen many of you at our booth. Before coming here, I had never been to Dublin nor had I ever had a Guinness. So I built this little Dublin City Center Pub Finder. And I went ahead and I built it using the Django MongoDB backend library. And here's a screenshot if you haven't seen it at our booth of it in use.

Speaker 1: If you want to, you can check it out later at our booth in the snack area or on my Dev2 account where I can walk you through the code and you can actually go ahead and build it out yourself. But first, for context, I would love to just take you through how this PubFinder even came to be. And in order to do that, let's quickly go over what MongoDB even is for those of you who may not know. So in a sentence, MongoDB is a NoSQL document-based database So instead of rigid tables and rows, it stores your data as flexible JSON-like documents, complete with nested arrays and fields, making it very, very easy to model

Speaker 1: complex. Evolving data structures and scale out horizontally without to using very costly join operations Okay, I know that was a very quick overview of MongoDB, but with that, let's jump into our agenda for the next 30 minutes. And if you want to learn more about MongoDB foundations, come find me after. I have a lot of really great resources. I am going to be taking all of you through a little bit of a timeline to help give a very holistic view of how we got to where we are today and what our view for the future is. And I'm going to aim to leave a couple minutes at the end for some questions. So, first, of course, let's go over our inspiration. and our motivation for how we ended up today with our new Django

Speaker 1: MongoDB backend library , which I will give all of y'all the full details about in a bit But in order to fully understand the motivation and our goals with this package, we will have to dive into a little bit of history. Um first before we can chat about what exactly it is and what views we have for the future So the idea for backing a long-term solution to combining MongoDB and Django really is nothing new. We have several enthusiasts who have actually been pushing for this integration within the company itself. We have historically had support for SQL-based open source frameworks, kind of like the entity framework in.

Speaker 1: NET and C sharp, Doctrine and PHP, and many more. So it was always actually top of mind to support Django, and we even had some previous attempts at doing so, which I will cover in the next slide. But you probably have this question, um, why did it take us so long in order to be successful, right? Well, we needed a very strong narrative from the Django community to truly justify the organizational cost. To develop and also maintain this project. And very luckily for us, this actually came last year. We saw a growing presence of MongoDB usage within the Django community So a couple of statistics to back that up because I know a lot of us are very data-driven. In the 2021 Django Developer

Speaker 1: Survey, MongoDB actually wasn't even listed as a back-end database used by Django developers. This all changed though in 2022, and it's crazy how much can change in a year, right? So in the 2022 Django Developer Survey MongoDB was actually cited as the most used in the 6% of the other databases category. Yes and then and then again in twenty twenty three that number rose to a fantastic eight percent Which in the grand scheme of things, I know that that seems like a very small number, but that equated to about a hundred thousand developers.

Speaker 1: And that is what gave us the push that we needed. So we learned from developers and existing third-party libraries that a common pattern of integrating MongoDB and Django Was just by connecting to MongoDB using the APIs of PyMongo, which is our official MongoDB Python driver library, or through open source source ORMs like Mongo Engine. So, this sidecar was of course a very different way from the standard of connecting Django to a database by specifying the database in the database's settings, which all y'all know. And what was the major consequence from this, right? Well it meant that a lot of the great aspects of Django and MongoDB were not able to be fully utilized in a simple way, like the admin

Speaker 1: panel. It was awkward because developers weren't even able to use the Django ORM, even if there were some very valid use cases. And so what did this make us realize? This made us realize that projects that went ahead and tried to integrate Django with MongoDB needed our scalability and flexibility that we have on our platform. So this made us build a more comprehensive compatibility layer, fully developed and supported by MongoDB, so we could free these projects from any extra development burden and let them build fully fledged out. Django applications backed solely by our document database. But that probably reaches the question as well: why did we go ahead and build an entirely new library?

Speaker 1: I'm sure most of you in the audience today have either heard of or actually used one of the libraries listed on this slide. There have been some really, really impressive attempts at integrating MongoDB with Django over the years, and one attempt even came from us. And unfortunately, as I'm sure you are all aware, they fell short. Some of these libraries had developers dealing with maintenance issues, technical complexities, poor performance, and overall just very unideal ways of working with Django and MongoDB. But they were very, very helpful too to show us what had been built and how exactly we needed to pivot. So we created a new library. That was actually started with code from the Django non-rel

Speaker 1: since it took the approach that we were looking for. A database backend that allows developers to use Django models and query sets. And however though it was a little bit outdated. So we just used it as a starting point. And then our updated back-end work became the initial code commit to our new repo, which y'all can check out. Um so why did the other libraries fall short, right? While some of the libraries simply works by translating SQL queries generated by Django's ORM into MongoDB queries, which had very many drawbacks, as you can imagine Or we had, for lack of a better word, very hacky solutions to force different parts of the code to work without adhering to the latest versions of MongoDB.

Speaker 1: And then other libraries were started and then just simply deprioritized due to lack of Django specialist knowledge to make super proper calibrated decisions around the platform. And many developers ask, and some of y'all have asked, and it's a great question, why would a Django developer even want to use a NoSQL database? And you are gonna get your answer soon, trust me, and you're going to want to stick around to find out. But for now, let's jump in and chat a little bit about the learnings that took place while successfully creating the Django MongoDB backend library. These are some of our key learnings. First of all, our work with Django and third-party libraries like Django Filter

Speaker 1: actually involved fixing numerous tests. And boosting overall code quality, stability, and provided a library with a very low barrier to entry where developers were actually capable of using it to create super complex projects. Out of the five libraries we actually were that we were targeting, Django Filter was the library that we ended up working most closely with. And employees at MongoDB were actually boots on the ground fixing tests inside of the Django filter package and directly collaborating with the author of the library One of the biggest one of the biggest positives was being able to directly engage with the incredible Django community, all of y'all.

Speaker 1: The Django Software Foundation taught us that building the library required a lot more than just code. It required and demanded an understanding. of this community's needs and required us to really fill in any NoSQL knowledge gaps. We also wanted to create another option for Django developers to use and this was our biggest attraction with developing the package the way that we did. Another really big learning that we faced was discovering that transitioning from SQL to NoSQL is not a one-size-fits-all solution. This really reinforced the importance of flexible migration guides. And our support of the Django ORM goes very, very deep. We really took on the challenge of making it work, despite the fact that MongoDB is a NoSQL

Speaker 1: database. So these are just a few of the very crucial learnings that we discovered holistically while working on this public preview for our package. But what about getting deeper into the learnings while actually programming it? These are the three aspects that we learn the most about while actually creating the library. Testing, generating queries, and then of course our documents. So testing was one of the more challenging parts of building the MongoDB backend for Django because traditional Django tests expect a relational database environment, right? And despite these difficulties, the effort to develop comprehensive tests really, really paid off by building much-needed confidence in our system's reliability.

Speaker 1: Not only did these tests help catch nuanced bugs early on, they also served as a safety net for future development changes. And this made sure that the integration of MongoDB schema with Django's ORM completely maintains its integrity. Tim Graham, who I'm sure some of you know, was super beneficial in this process Because he started testing right away against the Django test suite and he made some really crucial modifications in order to make sure that it worked. Because of his changes, it is much easier to actually use our package in your developments than any other previous versions. And then of course, transforming Django ORM calls into MongoDB's query language was another very complex task that we faced.

Speaker 1: The process of query generation really required rethinking how high-level queries are interpreted and executed in a document-based database. Insights from our database experience engineers were really instrumental as they helped clarify and streamline the logic needed to support a very seamless and efficient translation between Django's expectations and between MongoDB's capabilities. So this very collaborative approach ultimately led to a more robust query handling that really preserved performance. And in MongoDB, the natural way to store data is by using documents, which pose a very unique challenge. With Django's model system that is built around relational field types.

Speaker 1: But I don't have to tell you all that. And to address this, the development team actually created custom fields specifically designed to support Django's ORM. So this custom solution not only maintained the flexibility and the power of our schema, but also allowed developers to work with Django 's familiar framework. Which meant making sure that data structures could be managed as efficiently as traditional relational fields. So all of that was to say that this brings us to the present. In February earlier this year, we announced that our official Django MongoDB Backend Library was available in public. preview. So what exactly does this mean and why should all

Speaker 1: y'all be as excited as we are? Let's begin by answering exactly what this library is, right? It's a third-party database backend that integrates seamlessly with Django. When a developer defines the database backend engine when creating their application, they can go ahead and specify our backend. And as long as the library is installed in your Python environment, Django is going to be able to connect without any issues. It also aligns with the familiar steps that Django developers know and love. The process is very smooth, especially if folks use the templates made to make sure that everything is set up properly. Particularly because MongoDB is determined to support as many of Django's core features as possible.

Speaker 1: There is a little bit of overhead with our package at the moment. But MongoDB is committed to seeing everything through and ensuring that the process is not only great, but requires a very low lift from the community. And we are very, very well aware that a successful library requires so much more than pure technical expertise, right? It needs to be the perfect appreciation of Django in its entirety. its ecosystem, convention, and most importantly, the needs of its developer community. So all of you. And we are fully committed to making sure that this library is up to par and that it not only meets the technical requirements of developers, But that it is a very painless and intuitive process acting as a very natural complement to the base Django framework.

Speaker 1: So with all of that in mind, let's jump into what exactly is offered inside of this library. In this public preview release, we are offering developers these various capabilities. We want to ensure the community can use Django models with the utmost confidence So developers can go ahead and use Django models to represent MongoDB documents with support for Django forms, validations, and authentication. We also wanted to keep Django admin support intact, so this library allows developers to use the Django admin page as they normally would, with full support for migrations and database schema history. And configuration is as simple as pip install Django MongoDB backend, followed by downloading the templates, which I am going to show all of y'all in a second

Speaker 1: some code snippets. Just so you can see how intuitive this connection process is. And we also wanted to enforce MongoDB-specific querying optimizations. So field lookups have actually been replaced with aggregation calls. Joint operations are represented through our $2. And it's actually possible to build indexes right from Python. We are also allowing for advanced functionality. While the package is still in development and there will be a ton of very, very cool features to come, there is already support for time series and projections. And we also encourage utilizing this application and this package for various AI applications because we are compatible with frameworks such as Langchain, LAMIT

Speaker 1: Index. And you can use it to do advanced search using MongoDB Atlas vector search, which if most of you saw my Dublin Finder, I explained to you all of those little functionalities. And this package has really great aggregation support as well. So you can use the power of our aggregation pipeline in conjunction with your Django framework. Raw querying actually allows for aggregation pipeline operators And since aggregation is a superset of what traditional MongoDB query API methods provide, it will give developers a more fle more flexibility and functionality. Okay, so I went over a ton of information, but the best part about all of this is that it really is the start.

Speaker 1: We have more functionality and features coming out, such as Bisson data type support and embedded document support and arrays on its way. So definitely stay tuned for that. And we have our general availability release later on this year. And at this point in my little presentation, we have gone over why we decided to create this library and some of the functionalities available for developers. But let's turn our focus onto the benefits of utilizing this package in your actual projects So this integration really is a glue between the best of Django and MongoDB. And you're able to highlight both both all of the great aspects by using this integration.

Speaker 1: As a Django developer, you are going to feel right at home with MongoDB's document model. It maps erratically to your Django models, lets you store related data together, which means that you can ditch complex joins And it allows you to handle hierarchical or semi-structured data naturally. That means faster development and cleaner code, which as we know is exactly in line with Django's philosophy. Beyond modeling, MongoDB Atlas brings features that boost productivity. We have Atlas Search, which offers built-in full text and vector search, while the aggregation framework which is essentially an assembly line where you can isolate documents through individual transformations, lets you run very complex analytics

Speaker 1: and transformations in database. So you don't have to use any extra services. And our security, of course, is fantastic. Data is encrypted in transit, at rest, and even in use with client-side, field level, and queryable encryption. And when it comes to AI and semantic search, MongoDB integrates seamlessly with Langchain, Llama Index, Haystack, which means that you can build your complex AI applications. And then of course we have our incredible ecosystem as well with our VS Code and PyCharm extensions, the MongoDB shell, Atlas CLI, Mongo Import, Export, and the GUI -based compass. Which means that you can work however you like.

Speaker 1: So, with all of that, how simple is it to get up and running with this library? It really just is one command. Pip install Django MongoDB backend to install our Django integration. We have even created a very easy-to-use starter template that works with the Django admin command start project Making it simpler than ever to see what typical MongoDB migrations look like in Django. It's not necessary to use these starter templates, but it is highly, highly recommended. They are the same as the default templates with a couple important changes. It includes MongoDB specific migrations and our settings. py file has been modified to ensure that Django uses our object ID value. For each model's primary key.

Speaker 1: It also includes MongoDB specific app configurations for Django apps that have our default auto field set. So we recommend that you start a new project or app using the template. Otherwise you can head on over to our GitHub for all the other steps to set it up if you'd like to do it yourself. And with all of this in mind, let's quickly jump into a demo and see some of this in action. Alright, so this is our Omni search, and I'm gonna be facing away from y'all a little bit so I can see what's going on as well. And we have just created a little basic search here. As you can see, this is just me showing all of y'all what our basic search looks like.

Speaker 1: And this is how you can do it with our Django MongoD backend. So I'm just going to go to our search route. To see this, what we can do, and listen, I know the front end isn't great, I'm not a front-end engineer. Um, we can go to our views And it's quite simple. I get the query as a query parameter defined from this title bar. So let's say that I want to look up the Transformers movies, right? What I can do is I can chip type in Transformers and it gives me all of the results for the Transformers movies And how this part works is I'm able to get a simple query where I get the query set. It will then take in the word that was passed in

Speaker 1: as our keyword argument. And then it checks to see if it matches or if that word is contained in the title of the movie. We also went ahead and just took the liberty of adding in some extra metadata. just to make sure you know that it fits easier and that this demo flows a little bit smoother. Alright, so now we have our Transformers movies and this is great, however there still are some limitations, right? within the basic search. Let's say that you don't know the title of the movie, but you do know the plot. In this case, how would you be able to search for that? Right? You could go, one workaround is you could go and you could go to your object filter and then contain all of the plots that you want, but we have a lot of movies in our database

Speaker 1: Not only do we have a lot of movies as the user, how are we going to be able to rank them? How would we be able to know that we're using the entire plot? Right? So that is not the best decision. And this is really where the power of MongoDB comes into play. Because in MongoDB we have the very powerful Atlas search operator. And that works seamlessly by taking the search term that we provide and actually searching it against all specified documents. inside of your database. And then what it does is it gives a database ranked value on how relevant the document returned is to the term being searched So right over here, let's walk through exactly what it is. And what we did was we inherited our basic search

Speaker 1: just to stress how simple it is without using anything else besides our Django MongoDB backend library. Okay, so what we have here is our dollar sign search operator. And if you look through this query, this is what allows us to use our Atlas search feature. So we can define another operator known as text right over there. And it looks over our tokenized words or our tokenized phrases and then provided on each field in the document. So let's say that we have the word transformer that we used before. That in itself is a token. And we can use these tokens and then we can search against them. And then of course we have a path, and within, and that's within the text operator.

Speaker 1: Because I already know that I want to search alongside every single path inside my model, I can go ahead and do that. I just have to specify the field name and say that I want every field name to be searchable under this query. And then I can define the actual query, which of course is just term from before. So the next part is fuzzy search. And what do we mean by fuzzy? Fuzzy allows for you to search even if you make a little spelling error. I'm not the best at spelling things out So if you have a character too off, fuzzy allows for you to still have the results that you're looking for. And then of course our next part is just the limit. Because we are using raw aggregate, I couldn't specify a limit, but our limit just lets us know how many documents we want to see.

Speaker 1: And so now that we have our Atlas search all accounted for, what we can do is we can move up to our URLs. py file. Scroll over and do that. And then here we're just going to be switching out our basic search for our Atlas search. And we can go back and refresh our page. And then we can type in transformers again just to see if you know we didn't break anything. And as you can see, Transformers still comes up, all of our Transformers movies. But now let's say that we don't know Transformers, but we do know somehow the natural enemy of the Transformers, which are the Decepticons.

Speaker 1: So we can search for our Decepticons, and as you can tell I misspelled that on purpose. On purpose. And Transformers still comes up because Decepticons was in the plot. So I'll I can turn back and face all of y'all again. But that is our search demo. And if you would like to learn more about that, just come find me after and I can explain it even clearer. Ah let me see All right. Okay, so at this point you might be wondering what's next and if this is it, right? I am very happy to say that this is just the beginning of our journey together. We have really big plans to pull out all of the stops. And ensure that we have a really great GA later this year.

Speaker 1: We actually have someone working full-time on third-party integrations to make all of this happen. Some of you might know him. His name is Alex Clark. along with continual beta releases until RGA is out later on. But that begs the question, right, what exactly can you expect from RGA? You can prepare for programmatic management of vector search, atlas search, and geospatial indexes by using the Django API. Vector search, atlas search, and geospatial queries through the Django API, queryable encryption and client-side field level encryption, database transactions storage of cache data in the database, and even more great features and functionalities to look forward to. We already have plans for PostGA releases.

Speaker 1: such as gridfs for large file storage, chain streams for data monitoring and schema validation. So we have a ton to look forward to. And hopefully, as our trusted community, y'all can let us know what you think about our continual beta releases and try it out for yourself and let us know what features you want to see integrated inside of your project projects A big part of looking into the future is our acquisition of Voyage AI. So if you found me at the booth, I've explained this a little bit to you. But Voyage AI is groundbreaking for developing complex AI applications because it has a state-of-the-art embedding model, which is solving the hallucination. and fragmentation problem that happens with a lot of AI projects already.

Speaker 1: So having all your AI needs directly inside of your database layer removes a ton of friction. And Voyage AI services are actually going to become fully embedded in Atlas with auto embedding for vector search, native re-ranking, and enhanced multimodal retrieval. So what does this mean for the Django community, right? It means that Django developers are going to be able to model and store embeddings as native vector fields inside of your documents. Perform semantic searches directly from your Django views, build multimodal experiences using the MongoDB language that you're familiar with. And of course, one example that I've talked about a million times is the Dublin City Center Pub Finder. And that I built using the Django

Speaker 1: library, Voyage AI for the embeddings, and then Langchain for semantic search So, if you are interested in AI or currently building an application that uses AI, stay tuned for updates as they come out. And speaking of the Django community as I've done for the fast 30 minutes, I just wanted to take a second and highlight y'all It's very impressive, as you all know, with your active participation, overall friendliness, helpfulness, and inclusivity And so we are very, very dedicated to working on our package and we hope that we can prove that to y'all along the way. So we just want to emphasize that we are dedicated to you. And if any of you have any questions, you can come and find me and I will Be bit as helpful as I can

Speaker 1: be to answer them. Um please check out our repo and then let us know if you like it and If you don't like it, if there are any changes that we can make, we would love for all of y'all to file issue tickets, give feedback, try it out in your own applications, or even submit pull requests. And let us know your thoughts inside of our Django Developer Forum. Thank you all so much. If you have any questions, I'll be happy to take them.

Speaker 2: Hello, thank you for your talk. I've never used MongoDB nor have ever considered doing so. What is the top reason why I should consider it over a relational DB?

Speaker 1: Yeah, and that is a very, very great question. I think a lot of times it depends on your use case, and I am incredibly biased, as you can see from this 30-minute talk. But I will say that the best thing about MongaDB is it's very easy to get started, has a very low barrier to entry, and we also have such a fantastic platform where if you're trying to build complex applications, you can do so right out of the box. And if anyone needs any resources on getting up and running with MongoDB, just come and find me and I will direct you in the right place.