Search Options in Django - Stefan Baerisch

Speaker 1: Okay. Hello everyone. Welcome from Munich. I hope you're well wherever you are. And yeah, thank you for joining the talk on search options in Django. In the next half hour or so, we'll just have a short overview about different ways you can implement full text search with Django in Django. And we will also, well, look at some well some background why you would want to do it. So yeah, just shortly about me. So I'm basically a software engineer by trade. Did a last well some project management on the side for the last couple of years And I started out in search, then did some data processing in Python and then slowly worked myself toward Django.

Speaker 1: Yes, well, first of all, some background on full-text search. Many of the things that I'm saying might be obvious, but I think it's Just interesting to repeat it. So why do we want to do search and not Kerry? Well, we still have a lot of video in our lives. This talk is Well an example. But there's also a lot of text. And if we search within text, we well we have to take many things into account. We might have typos, we might use different languages Different documents are written in different ways, some shorter, some longer, some use a more formal language. Some are quite controlled in terms of what they are. A tweet is very different from a book, and when we want to search for either a tweet or a book, we expect that we get good results document

Speaker 1: with hopefully a reasonable query, but we don't really want to Spend too much thoughts and too much time on the best way to do the query. That's the difference from databases. So if you do database query for any row, for any entry, You usually need to know exactly what you want. So in the simplest case, you could do a query by an object ID, or you know that a certain text part is within the document. Or you basically just narrow it down with searching by date, by name of an author if it is a document, etc. But whatever you do, you need to know exactly what you want, and the database system will give you exactly what you ask

Speaker 1: for. So if you are slightly wrong or if there is a document that uses a different writing form of a word or if you have a typo, a usual SQL query will not help you very much. Search is less precise, more fuzzy, and you could also say more in a way humane. So it tries to almost guess. what you meant by a certain query. And then also tries to give you the most relevant documents first. So you can imagine If you do search in a large set of documents and search for common words, that you will have many, many results. And so you you don't usually want to

Speaker 1: check, let's say, 200, 300 documents. before you finally find the one that you're most yeah so to summarize a database query is give me what i want what i said And search, hopefully if it works, is give me what I mean, even though I haven't really said it What can this mean? So let's take an example and hopefully somebody in the future will search for maybe this talk. And this person may look for my name, Behrish for Python and 2020. And well they may actually write my name as it's written in German with this A umlaut thing. And

Speaker 1: they may misspell Python. And well, if they enter 2020, they don't necessarily say that they want to search for the date 2020. So There is no 2020 in the title of this of this talk, for example. And no search system, give me what I mean. would do some rewriting. So it would know that German umlauts are not necessarily the way that the German that German words are always written. So it might rewrite it to this RA form. And hopefully it would also catch up that Python is not the right writing for Python and might rewrite it into Python because, well, that is a common word, relatively common term. Whereas

Speaker 1: the misspelled form is not. You might also know that certain numbers, nineteen something, twenty something. Very yes. So it may not search, for example, in the title or in the description, but in the metadata and see, okay, I'll know that this is a document So it will take the query, what I entered, and we divide it into something that represents what I mean. And in order for this to work, Of course, it would need to do the same thing with all the documents that it gets. So it would need both query processing and document processing. This is quite different from a database where we need to index our data, but we don't really change or touch the content of neither the query nor the document.

Speaker 1: Yeah, and finally we I also want if we want our documents back, we also want to have the most relevant documents at the top. I already said that. But it's it's worth getting into because that's the thing where search can get quite complicated. Because relevant is something that I as a user, every user, would need to decide for themselves. So for example, if I search for something in um description of some product if I do online shopping. A relevant product might be something totally different. Well might it may need different implementation, different query and document processing than if I was searching for a book. Or if I was searching for a tweet.

Speaker 1: A tweet might be relevant if it was made in the last say 24 hours and include my words. A book may be written in 1900 and still be relevant if it has a concept that my words allude to. And well in a product database, if you implement something like this with Django, or it could be an article database. just a content management system, for example, you would need to decide what's relevant for my users or what's relevant for me. Maybe I want to show them products or articles that I think are especially important for me for whatever reason. So I would need to do quite a lot of work to implement some kind of relevant system that really fits me, fits the use case that I want to serve.

Speaker 1: And these are all things that I can keep in mind together, with everything else that goes into an IT system, like a jumping over application. As a user, I want the system to work well basically all the time. So I don't want any downtimes. I want a fresh index. So if new documents are added, I want to be able to search for them. I want quick results because even with a good search system I will likely have to try different ways to search a system, different keywords, different terms. And as an operator of the system, of course I want something that is maintainable. So some piece of infrastructure that will still be around in, let's say, three years time that has a stable API, got documentation. It's not too resource

Speaker 1: resource intensive to run, it's not too hard to learn. And all these things go into the search system. So I need to know about what will my user search for? What do my documents look like? what's relevant for them and also how can I actually implement this in IT. And what we're going to do is just is look at two possible ways to have search with Django. And one particularly set of documents, what we are searching for. And what I came up with is an old set of documents originally from 2013. You can see a citation for the for some scientific article down there on the slide.

Speaker 1: And what these um people did is just gather some film reviews from Amazon, th eight a million in total. and brought them into um well half more less structured format. This is quite an interesting data set for search Because as you can imagine, some people write short reviews, some people write long reviews, some people take a lot of care to have good spelling and grammar, others less so. So you might encounter something where There's almost a little essay while Citizen Kane is the best movie ever made, while the next review is just didn't like it. And yeah, you can search for that on summary and on text. So you have this little title and you have a longer text underneath.

Speaker 1: And you have some extra information that can be helpful. So for example, you have an helpfulness score in there that is a judgment of other users how helpful this particular review is. We will not use this in this talk, but this, for example, would be something that could be quite useful if you did ranking on this document set. So you wanted to surface the high usefulness reviews first. So that's what we are going to use for this example. On the right side you see a very simple Django database model that actually represents it. almost exclusively interested in the two text fields, so the summary and the text.

Speaker 1: But if we wanted we could also well play around with the user IDs and other things. We'll see how this works later. Okay , now let's imagine we are in this situation we want to look at rather old movie reviews from Amazon and we build a website. And at the beginning we want to keep things quite simple. So we already know we have all the data that we might need in our database. And we already know SQL. And what we just do is that we just use the iContains operator. So case incentive contains. And we look for just one word. I'm interested about movie reviews, about water.

Speaker 1: Maybe I'm a fan of the film Waterworld, I don't know. And I see what I get. And well, it's basically what we would expect. Get some titles. We get a snippet of the actual text. And you can see here it's just of interest, we see that we've got about 2200 documents. And it took slightly less than a second to give us all these documents. I think I indexed with I think one million documents for this use case. So I didn't use the um whole document in the MZ whole number of documents. Okay, that's basically what we expect. That's SQL. Now we have heard about full text search and we wonder whether we can already use full text search just if we have Django and Postgres.

Speaker 1: Good news is we can. And we can in essence just switch it over. We instead of our icontains, we use search. And this already, together with some Postgres extensions for Django, transforms this into the well full text search functionality that Postgres provides. And again, we get some documents. And the first thing that we see if we go for this extremely simple approach is that first of all we get less documents That's interesting. We are searching for the same term, the same number of documents. What could happen? Well, our icon tains has usually found out all the documents where one was

Speaker 1: for some reason or other part of a word. So watery would also have been found. Reserve C we just search for water, so just for the term. And we also we only find water because with these functions functions we already do some pre-progressing and processing on the documents. So In a sense, it is a better way to search for normal users because there will be less surprises. Every document that we have found has water in it. On the NES positive side, if we look at these, it is actually not clear why we got these documents. So it's quite possible that there is a water somewhere in it, but it's not obvious why did we get these documents.

Speaker 1: Also, it got quite a bit slower. So it now almost takes 10 seconds to a pause to give this results. And this happens because we have to do this processing of the documents. So we don't really touch the index when we do that. But we well have to look at every document and do some processing. Let's address both of these issues. So, first, why do we get these documents? Can we do that better? And we can. So we can start with a very simple approach to relevance And we can say, okay, the documents that I'm interested in should contain water in either the review or the summary.

Speaker 1: And I think if they if the word water appears in the summary, so in the title, the document is likely more about water than it is somewhere in the summary So my relevance model says look at these fields, build a vector representation, which is basically a full-text search representation of these two fields And weight the summary title field with priority A, the highest, and the full text slightly lower And also, that happens explicitly if we don't say it, but we can say it here, consider all these texts English. So both the query processing and the document processing will work with English words

Speaker 1: Also, please rank these documents according to these weights and also only give me documents that are highly likely to be relevant. So we move everything from the query set that may have water somewhere, but where it doesn't seem to be too important. Okay, what happens now? We got a few less documents. That's to be expected. We told the system that we were only interested in the most relevant ones. And also now it seems yes we have water. And if you look at it closely, you see that each of our top three hits here has watered twice in the title

Speaker 1: and that happens if you say okay I prioritize the title highly and if our relevance module is counting words So the more often the word that we search for is in the text, and the shorter the text is, the more likely the document is about this particular word. That's not the most sophisticated approach to ranking, but at least we understand why we got the documents. And well, it is doubtlessly about water. Negative thing though, we are still there with our 10 second query time. Okay, let's look at something else. We can add in additional index, full text search index, to our Postgres database, and this will speed up our queries.

Speaker 1: So we'll have our regular dataset with our model and we now add a new index to keep our full text search implementation. So we can hopefully get faster If we do that, we'll have to do a slight change to our data model. So we add this new index. And in order to keep the model updated, we also make a change to our database migration. And we add an extra trigger. So in essence what we do is every time that the document gets that the new documents get edited or changed. We will run this trigger. This trigger uses the internal Postgres options.

Speaker 1: and builds one new index field that this time combines the summary and the review text Well, and to reverse this migration, we would drop the trigger again. Okay, that's nice. So we add our new documents. We do almost the same as we did the last time. with one relevant change. So what we do is this time we church only in the index. We defined only one. We could divide multiple indices and um gave them different priorities, but to keep it simple, we just have this document. And since we have a different relevance model, we also change the cutoff point. So this time we are

Speaker 1: take more different documents. And what happens? So things get significantly faster, even faster than the initial scan when we digest the SQL contains. But since we have this time only one index, our ranking model is we don't no longer have our search term title in the water in the title but if we look we can see it in the text so it's quite quickly in the text but no longer in the title Okay, so looking back, this is all more or less built in. We don't need much code to have a simple search model with More or less manual

Speaker 1: migration, okay, to make it performant. We have to define some extra indices, but we can have a reasonable search, or at least a search where we know what happens. Now if things get slightly more complicated, you may need to look into something else. And This could be Elasticsearch. Elasticsearch probably things that most of you know it in one form or another. is a commercial and open source search application. In a way it is the commercial version of Solar. It does the more than Solar. Solar is You may know is a search server based on Lucene, which in part is an open source

Speaker 1: Apache Java library. And it has many, many, many features. So for example, if you wanted to learn from previous queries which documents are the most relevant for your users. So if you wanted to use deep learning technologies for ranking, you could do this with Elasticsearch and some additional things. If you wanted to say, give me a ranking function that will use review usefulness, you could do that But it also means that once you step to Elasticsearch, you get a lot of extra complexity in your Django application. Let's see how that looks. Yeah, we basically looked at that.

Speaker 1: So with Postgres, we just have one Postgres database. And we have a trigger in this database that keeps the database in sync. in terms of full text search and our usual data and everything works fine. Granted, we will add extra data and extra load to our Postgres, although that's a scaling issue. We might have to keep an eye on that. But we don't have to bring in any extra systems. This changes once we have Elasticsearch or Solar or another system in there. In this case, we need to think. So if we search for a new document, we would first need to hit our search system. then

Speaker 1: most likely get some document IDs, get them from the regular database, and then surface them in a Django view. Okay This is not so much code we can do that in a view and we know how that should happen. Gets slightly more complicated if you add documents or change documents. Because if you do that, you not only have to change the data in Postgres, but also in Elasticsearch. And you have to decide whether you do this synchronously or asynchronously, how you do this in the background. So you could in S In C we write another trigger that calls an external program that posts data to Elasticsearch. You could have a job queue to do that. You could in C we do it in view, so I wouldn't necessarily recommend it.

Speaker 1: Yeah, you have some extra complexity. Okay, it doesn't really buy you so much, except that you have an extra server and the search functionality is not in Postgres, so at least you don't have all the extra load on Postgres. Yeah, here we have that as an overview of research. And this is what we're going to implement for the in the next remaining minutes. So what we are going to do is just have a simple search form, the one that we used previously, send a query to Elasticsearch, our usual water, but it could be any other search query. Then Elasticsearch will give us some document IDs and also just to demonstrate some extra features, some facets.

Speaker 1: So basically counts how much a certain how often a certain feature uh we presented in all our of our result set. And then we get the actual data from Postgres and render view. Okay, let's look how this actually looks what this actually would look like. There would be different ways to implement this. We could use the Haystack library, which is an abstraction of different full-text search backends for Django. So it gives you essentially the same API that you would have with the Stunter Django ORM. However, it Also really fits in some search functionality, which is nice.

Speaker 1: You could also use the official Python API for Elasticsearch, or you could just use the REST API. I used the REST API in a certain sense for simplicity's sake, so I didn't want to distract too much from what happens behind the curtains. But also just to show you how the API for Elasticsearch looks like, because you have to specify quite a lot of things that Postgres does automatically or where Postgres has some defaults. First of all, we want to put our data into the database And what we need to do is well create an index. And an index

Speaker 1: is well essentially a definition how do we need to process our data. So what we say is we define a filter and this filter says remove English stopwords. Stop words are words that are extremely frequently in documents and that we don't really want to process when we search. They, and , I, you, they. and a couple of hundred other words that are expected to be in every document and not that useful for search. Okay, so we move all these words. We also use a stunt tokenizer. Tokenizer. Remember our first example, water. We don't want to

Speaker 1: look for the word waterly necessarily, but we want to break it down. We want to break the text down into different words. So we define a tokenizer for our analyzer. And then we tell them how to process these words. So we lowercase everything and then we remove our stop words. So this is the way that Elasticsearch will look at every document when we index it and when we search for it. Okay, so we have defined how to process our documents. And well, as you can see here, the rest of the mapping is pretty close to the definition we have

Speaker 1: In Django, we just have a name and then we define the different data types and for our text fields how they should be processed. Then we just post our documents. In this case, I just post every document by itself. There is also a batch API that makes things faster, but for this demonstration it wasn't really required. And every document that we post to Elasticsearch via the REST interface is just, well, a regular JSON document that looks essentially like that. So like everything before. And now we want to search. Now if you remember with Postgres

Speaker 1: We only had to say I want to search in these documents, so I define a search vector. I want to search for this particular term, so I define a search query. and potentially a ranking model and then search away. With the REST-based API, we really have to build the complete REST request And that's what you see on this particular slide. So this is our search function that is called in our view. We call for our query string. We do call our search function and we get our results back. Our results have both the documents that we're interested in. And also so-called aggregation, which keeps count of certain

Speaker 1: elements of the results. So essentially it says we have so many documents in our results that that weights the film a five versus a three versus a four, or this user is um has written so and so many doc reviews in our results So this is called our view. Then we have our multi-search function. There we define this Multi-match, so we tell Elasticsearch if we want to search in two fields with our usual query. And then just essentially fill out a template that tells Elasticsearch, the details of what we want. And we also define that we want certain aggregations to be returned.

Speaker 1: So our query is here, and this part tells us about how we would expect our results. Okay. And if we do that, we get our results set back. So we get this extra facet functionality, which is nice and not something that you would do with Postgres out of the box. We also have quite good performance. Also I'd say that the cache was warmed for this particular example, so it wouldn't it would be slower from the first hit. And we once more have a different ranking, but this time again, since we um Looked at the title and the review went differently. We have water at least in the title, so we know what our documents are from.

Speaker 1: And well, if we wanted we could say give me all documents that this particular user has written. So this person has written nine documents. Or give me all the reviews for this particular thing. So there might be a movie that is related to water. Or give me all the high scores. So this is one use of Elasticsearch and we can already see it has slightly more advanced features. It is more difficult to learn, even if you look at the API, because you have so many different features and it is well can do everything from log processing to complex product review with external rings. But is let's say manageable

Speaker 1: So which search system should we use? Well, it always depends on a use case, but if you only need to look up words Within a document, if you already know what you're looking for, maybe IDs, maybe specific error messages, maybe even a regx, you can use Postgres and you don't even need to touch the full text functionality If you just want one system and you want to have a quick implementation with some let's say reasonable defaults and a rather simple ranking model, then you can stick with Postgres. And if you have more complex requirements, especially in terms of ranking, but potentially also in terms of scaling and

Speaker 1: If you're willing to do the extra work to keep different systems in sync and also learn different systems, then you can go to Elastic or if not to Elastic, you could go to Solar or Wuche or any other kind of external Vortex system. Elasticsearch just well happens to be the most popular at the moment. So it's not a simple solution what to use. There is a lot of going back and forth. And ultimately it will all depend on your particular use case. So do I need all the features? Do I expect that I will need many features in the future? And as you have seen, it's ultimately quite feasible to start out with Postgres, use the functionality

Speaker 1: there. And then at some point in time, change your system to use an external search system, either for ranking or for scaling, or because you want to keep your database focused on only your core data or for whatever other reason that your use case requires. Okay, that's already everything for this talk. We had an extremely short, let's say, overflight of search functionality. And my takeaway when I'm when this search is useful for many different categories, it is not necessarily easy. So we had sync five or six examples and we saw that each time we got different documents.

Speaker 1: And if this was an in-depth problem, so if we had really needed to have the best ranking possible. You would likely spend a lot of time working on your different relevance models and a lot of more time than you would spend on making it work in either Elasticsearch or Postgres. Technologies, the technology here is simple and it's part of the problem. And the good news is that we have multiple options available. One comes out of the box, that is Postgres And many other systems are available to help us. And we also have quite a lot of choice on how we actually integrate them. So we could use Haystack, we could use the API Or well, we could

Speaker 1: go for different solution altogether and even implement our own full text search system. Okay. And with that, I'll thank everybody who listened to the talk. I think we'll switch over to Jitsi now for the live questions. And yeah, thank you very much and see you there.

Speaker 2: multiple days just trying to get everything off the ground running and then on the uh second side how much work was it to implement it into your stack

Speaker 1: I think getting it to run is reasonable straightforward. And I say I'm at the beginning, so I haven't I don't use Elastic productively at the moment. I'm still with Postgres. That said, learning the API was about a day for the use case that we've seen here. Operating it again is a different issue once especially you go for when redundant implementation and clustering. You get a lot of out of the box. So I would say to set up at the development environment and start learning, it's maybe two days, maybe a week. It depends. However, the API if you go into the more sophisticated use cases. So not only what kind of data you want back and in which order and how filtered, but the different relevance models, the different pre-processing options, etc.

Speaker 1: That will likely take some time because not only do you have to change the configuration, but you have a well little go-around when you need to re-index your documents with different settings. You need to look on how will this all work if it have in a background process. Elasticsearch, I think, is the most complicated one. Solar is slightly simpler. It cannot do as much and but it also doesn't try to do everything. Another thing is you could use an abstraction layer like Haystack. Haystack will help you a lot to make it closer to what Postgres provides. Um I didn't do it in

Speaker 1: part also because I don't like abstraction layers because I don't really see what happens under the covers. This is something you may have noticed with Postgres. So if you just use the Postgres default search options. makes a slight some assumptions. So how do you want to tokenize your documents? How do you want to have them processed? Is everything in English, etc. And this can lead to some surprises. For example, with these the water search has slightly less documents than the than the water i contains. And you don't necessarily understand how the documents actually look like.

Speaker 1: So yeah To bring it short, you need let's say two days to get started if you have your own virtual machines and don't have to work much with infrastructure. But budget at least a couple of weeks if you want to use it for a productive use case and you want to be able to debug problems that your colleagues or customers come up with.

Speaker 2: Thank you.

Speaker 1: Welcome. Okay. So yeah, maybe just one hint if you want to get started. Um there is a GitHub repository. It is not much more code than you saw here. It's basically just a view that produces its Extremely ugly HTML and you can then switch around between the different search options. You will have to set up Postgres. You will need to install Elasticsearch. If you just wanted to follow this tutorial, it's will probably take you one maybe two hours because it's just a default configuration. And then you can play around with these different systems. If you want to try something else, let's say solar, for example, it shouldn't take you necessarily longer.

Speaker 1: And the documents Like I said, are from a research paper. They are all available, so you can download an archive of a couple of hundred megabytes and you can then search for this particular document. What I like about them is basically a smoothie review, so you don't need much domain knowledge to see what fits or what you would expect and what wouldn't. And there is also some metadata in there, so these review scores, the usefulness ranking, the user IDs. So if you want to go beyond pure full text search to do some filtering on re ranking or something like that. You can also do that. So if you're interested, that would be my personal recommendation just to get started.

Speaker 1: You don't necessarily even need to do a complete Django implementation since Elastic Solar etc. have REST APIs. You can just download them, index them, and then play around with them and see whether every whether the documents that you get back make sense before you even build a view around them. Yeah, and potentially yeah you could do the same with Postgres. So if you look for Postgres um full text search, you will find a quite complex and complete implement um documentation on Postgres itself and you will see some of the terms that also are visible in the Django API. So this gives you some background on

Speaker 1: what seems to happen behind the covers.

Speaker 2: Thank you again for giving the talk.

Speaker 1: Welcome. Thanks for joining the talk. I hope it was interesting or at least to got you started. And yeah, if there are no further questions, I think we'll end this one and I wish you much continued fun on the conference and hopefully see you in person well at a later conference. Bye.