Beyond Filters: Modern Search (and more) with Vectors in Django with Kumar Shivendu

Speaker 1: My talk today is going to be about modern search with vectors in Django. And uh so first of all, who am I? I'm Kumar Shivendu. I'm a software engineer uh from India. And this is going to be my first international talk. I work as an engineer at Quadrant, which is a vector search engine. And I have used Django in most of my roles in the past and I have built search at billion scale. Um and yeah, so let's get started. So first of all, the topics to cover are like we'll briefly talk about traditional ways of building search in Django. And then we'll talk about vectors and vector search and bit about vector databases and the HNSW index, which powers vector search, and then comes a bit of snippets for integration into Django.

Speaker 1: And then a bit about trade-offs and limits of vector search and finally some more use cases which are beyond text search, which are just impossible with uh traditional search mechanisms. And yeah, so let's get started So first of all, uh how many of you have tried building search with Postgres or anything in Django? Please raise your hands. Okay, quite a few people. Great. Okay, so uh the most simplest way to build search in Postgres with Django, right? is that you use these iContents uh filters, right? So for example I want to search for laptops which are priced less than two thousand dollars I could do a query like this. I I I'm gonna like create an index on top of the price and it does like

Speaker 1: it does work at least at a normal scale with fewer points, but But there's a problem uh that it is slow and like because it does a full scan this icon teams does not have a dedicated index unlike the B tree on the price So it's slower, there's no dedicated index, so that's the problem and there's another problem that Like it does not understand the language. So that's why what happens is that when you're searching for something like laptop, there might be entries around with talk about notebook, which is also a laptop, but it will just completely miss that. So that's about it. Um then we also have the Gin Index. Uh how many of you are familiar with Gen

Speaker 1: Index? Okay, okay, cool. So somewhat fewer people, so I'm gonna describe it a bit. So basically the whole point is that uh you have any term And you map that term to a list of documents which are matching that. So whenever you have to search for the term laptop, you're just gonna have a hash map which uh just directly points you to the documents that are gonna have that term and you can just return that is the more naive explanation for gen index and it's implemented in Postgres and it does work. But it also again has some cons. The first con is that it decreases the right, like it increases the light right latency because you have more indices to update, right?

Speaker 1: And then it does not have any typo tolerance uh facets or boosting of the fields. For example, facets is what you see whenever you go on Amazon, you see on the side, you know, uh a categories for brands and all that stuff, like filters. And then boosting the fields means, for example, you want to prioritize whenever there's a title match or a description match. So all that customization is just not here. So you will have to either re-implement on your own uh or use something which is more dedicated. So then we come to the part with Elasticsearch. Uh how many of you have used Elasticsearch? Okay, great, great. Um so yeah, so we have Elasticsearch. So Elasticsearch also has this concept of indices. And indices are very similar to tables in uh Postgres, let's say.

Speaker 1: So you create a product indices like index and on top of that you provide it. the fields like name, description, price, pretty much the same as the existing model that you have in Django. And it kind of Ham manages this library kind of manages the sync for you. Whenever you upsert anything in your database, it also does the same thing in the Elasticsearch cluster. So that's the thing. What's the problem with uh text search? Anybody has any idea? Like where would this fail? Anybody? Okay. Uh yeah, yeah, please. Sorry?

Speaker 1: Yeah. So it's not always relevant. Yep. Yep. So that's the thing. So for example, if your user is searching for the term user login. And that's a common term for any non-technical person. And then your entire documentation maybe just doesn't have that term. It won't just match. So it will not show up in the results because it does not understand language language. And that's the bottleneck that we need to solve. And um if you uh a lot of existing systems kind of try to solve these problems, uh like Elasticsearch also can do it uh using synonyms, but you will have to maintain them yourselves. which is quite painful and your data keeps on increasing so it's just getting too difficult to manage. And also you cannot search on top of the text images

Speaker 1: or audio. because you know the way image search with something like Elasticsearch, like like text search used to work is like you need to extract out some features from your image and create those labels like entries and then only you can search. So effectively it's just text search again. So that's why you need something uh like a new fundamental technology shift. So that is where vectors come into the picture. So vectors are like uh, you know These uh points in an n-dimensional plane, and these points are not uh random, these are organized in such a way by a machine learning model such that if two points are close to each other, they would represent something which is actually similar So if two uh images of a dog are close to each other, that means they represent the same breed of the dog probably, or maybe even the same dog, depending on how well your model is trained.

Speaker 1: So And the whole point is that you are trying to compress the meaning of whatever is being fed to the model. And this could be like an image or a video or anything literally. And uh you can convert anything into a vector like that depending on your uh uh vector model, like embedding model. And and the thing is like this technology is not new. It has existed for quite a while, uh for a decade, more than that, I be I believe. Uh but it was only accessible to big tech big tech because uh it needed a lot of uh data and um like custom model training to achieve that. But now it has all become accessible thanks to the like invention of Language models. Like we have generalized the access to vectors because of language. So language and vision vision models like Chat GPT, they are able to understand

Speaker 1: these images and text. and give you vectors which kind of represent the thing that you passed in and there are also models which have similar architecture to these uh GPT models. Which understand it. And there's also this trick called metric learning where you specify positive and negative pairs, and this algorithm will try to push the two pairs, the positive pairs closer to each other and the negative pairs away from each other. So yeah, one of them is clip model, which is able to understand text and image at the same time. So yeah, that's a bit about vectors. I hope it was clear. Um moving to the next part. So vector search, I think the best example for vector search is uh this Google lens. I think all of most of us have used it. So you just point it to anything like uh

Speaker 1: and it is able to find out the exact breed or um the any kind of cloth that you provide it and it it it is able to find that product. So Google and also Spotify recommendations, they are also powered by vectors. So you like a song and Spotify knows that you like it and it's gonna find similar points in the space and these points will represent different songs and that's how Spotify also works. So and and the thing is like the whole point is that humans think in terms of things, not strings, right? So And while keywords such at the same time, it has low recall, that's what you call in like information retrieval space. Like it misses the documents and it cannot obviously do audio and images as I have described previously

Speaker 1: So and and the whole point is that once you have these n-dimensional plane, right, and all these numbers, uh like you can just find the nearest uh vector and it's gonna give you the relevant documents that are gonna match So it's all about like finding the nearest points. But there's more to it. I'm gonna talk about a bit a bit later. So uh and the problem is like uh you know when you're having just let's say hundred points, it's very easy to say like I have this point, I'm I'm gonna find like I'm gonna compare uh the numbers uh with like other hundred points and just find the Euclidean distance or the cosine distance to figure out uh you know the closest points but when you have like hundred million and and one billion it's n none of this is gonna work you need something which is like which will be scalable. And that's where like there are certain algorithms uh which which mostly utilize approximation to make the scale.

Speaker 1: And we are gonna talk about that. And that is where like you have dedicated solutions uh which Which help with like optimizing and scaling vector search. So yeah. So that's where like databases and vector search engines like Cordent come into the picture. It's an open source vector search engine. uh with more than 25,000 GitHub starts. Uh it's written in Rust, has SDKs for all the popular languages like Python, JavaScript, Golang. It is being used by Twitter, Discord, HubSpot, Flipkart, many other big companies for different use cases. And the fundamental uh algorithm behind this is the HNSW index. Um the whole crux of this HNSW index is that uh Imagine if you want to go from here to Paris, uh the

Speaker 1: like there will be like different levels of you going from here to there, like different levels of flexibility. So for example, you could take a walk out of this building. And that would be like the lowest layer and you have the maximum freedom there. It's like highest connectivity. I'm gonna s I'm trying to simplify the algorithm because it's a bit complex. But I hope this will make sense. So the lowest level has uh the maximum freedom, the most number of connections. And then the upper layer is let's say you taking a cab. And the the most upper layer, right? That one is like taking a flight So when you're gonna go from here to Paris in front of the Eiffel Tower, what you will do is like you will, let's say, take uh get out of this building, they then take a cab, then take the flight. And then from the flight you will again take a uh cab in Paris and then you will walk towards the Affil Tower, right?

Speaker 1: And that's the whole thing. So that's the crux of this algorithm. So the topmost layer uh is all about taking huge jumps. The the first one that you see around the entry point uh that is taking a huge jump. It's you going from Chicago to Paris, you're taking the flight. So it's about that and with different hierarchies. And uh this is how it works basically and the whole concept is like a combination of skip list and graph. And this is approximate and tunable and you can filter during search and there's this technique called quantitation which allows you to speed it up a bit more. So yeah, let's come to how you create like collections, which is like creating a table. So let's say we create a collection for products in our e-commerce dataset or whatever. So you specify the distance that you want to use between vectors. So you are

Speaker 1: going to use, let's say, cosine distance. It's basically like comparing the angles and you specify some size for each of the vectors. And with this vector could be generated depending on your uh embedding model. And then you also spec can specify different indices on top of your different fields. um like on top of for example price so there are different uh categories of filters that you can have so that's there and you can just do go ahead and do an upset To insert the points and you can see that we have vectors, these are generated like numbers. This would be like that 384 that I previously showed you. These are like 384 numbers that will describe the image or whatever I have passed. So, and to do the search, you just specify again another vector query, and that will just uh

Speaker 1: with some filters if you want, and it will give you some results. Each with each of them will have certain score and that will tell how close the like results are So now let's come to the part of integrating uh vector databases into Django. So I did show you the internal HTTP APIs so that you can generally understand what's happening under the hood. But when it comes to Django integration, so we have a library called uh Django Semantic Search. Uh you can use it and in the settings. py file you're gonna specify something like this. where you specify the vector store that you are going to use. There are different backends which can be used. I'm using cordon here, but you can use others as well. So And we are open to contributions, so it can be extended. So you can specify because quadrant runs as an independent serve.

Speaker 1: So it runs as six ports, six 333 and you're gonna specify that and you also need to specify your jang like the embedding model that will power your search. Um So I'm gonna use the sentence transformer library. It's a very popular library for doing embeddings and this popular model called All Millie LM. So uh and the way it looks in Django is something like this that uh similar to how we did it in Elasticsearch, you need to register it as a document and with the meta fields. uh you specify the model and whatever you want to uh get a uh vector index out of. So for example there'll be a lot of plot data like for every movie there'll be some story behind it and that's where we are storing all of the uh text and this uh what this model this document will do is like it'll

Speaker 1: convert that into a vector so that you can search on top of it. And uh note that like previously uh you know it was all about you know you looking for individual terms and now you can just talk to it very much naturally. So you can just ask movies where AI became sentient. Like and it will work. So I'm gonna show a demo as well. Um so yep. So yep. So you can see that I have searched for uh movies where AI becomes sentient and it was able to show me you know plug and pray Um computer experts around the world, like you know, development of intelligent robots. And you can see that this term

Speaker 1: sentient is, I think, not even present anywhere, right, in none of these documents. But the AI, because of the AI, we were able to understand the document. We were truly able to understand the plot of the movie and find out the relevant results. So that's the power of this. And I can just change the query, like feel good movies And I'll still work. So yep. And it the thing is like your language model might be able to understand uh different languages so you might be able to type in German and it will still work. So you can see that this not uh English but it was still able to understand. And also with vectors, right, you can build a lot of things. As I said, Spotify also uses recommendations, uh powers recommendations with vectors. So you can just point in with the like this particular uh

Speaker 1: movie. And you can press like get recommendations and it's gonna show you like different movies which are very, very similar to this movie. So it might not satisfy the previous criteria of movies being feel-good, but it will still be closer to the movie that we chose. So that's the whole point. And so yeah, I hope that made sense. And so but uh nothing comes for free, so we are gonna discuss a bit about the trade-offs of doing vector search. So uh first of all, the main thing is that uh you're gonna do a lot more computation, you are gonna uh consume more RAM, so the cost is going to be higher. So uh the computation, the indexing part is uh like heavier, so it will consume more and it will also consume more RAM and but uh there's benefit uh

Speaker 1: which is like The latencies are a lot more predictable than doing a text search. With text search, like latency degrees decrease very fast. If your query length is higher Like you will suddenly see that your P95 and P99 are going to be very different. But with vector search, it's very smooth. So there will be like a very small bump. So and vector search scales very well. So that's a good thing And results will become like massively relevant. So which could be like great for your users depending on your use case, you will have happier users and uh hopefully that also increases your revenue. And uh the thing is like the cost is higher by default, yes, but you can also change like some configurations uh in the HNSW index. Uh so for example the in the graph search algorithm in the S HNSW, I did show you like that. It does a graph traversal

Speaker 1: And you can change the number of connections in each layer. So that will decrease the overall cost. And there's something called as quantization. Uh so basically the whole point is that uh By default the numbers are stored as uh like 32 bits, right? F32 uh or F64, but you can just use 8 bits instead of 32, and suddenly you are using 4x less bits So it will decrease your uh RAM usage and hence RAM's RAM is the one of the biggest reasons for uh like mem like cost to go up and if you are saving on RAM you can like uh save a lot. And then obviously you can offload these vectors to disk. So it will become slower, yes, but at the same time you're gonna save a lot on the cost front. So it's all about tuning uh the database and search engines to whatever you need

Speaker 1: And uh there are also limits to doing pure vector search at this point. Like there's a lot of ongoing research on this topic, but the current limits are mainly that uh it struggles a lot when it sees too many unknown terms So for example, uh it cannot search when you like pass in something like product, you know, some ID. So it does not understand but because it's not able to uh break that down and not understand like what does it does it mean like for example when it looks at the term dog it knows that what dog means but product x y z something like point i like some id it's not able to understand So it does not do great with this. It also needs a lot of like in some cases like it needs dedicated models with a lot of domain understanding. So for example, if you are building something in the medical domain Right. There are mentions of the medicine

Speaker 1: or uh like the treatment, so which are very specific terms which are not used in normal English. If the model just does not know it, it won't be able to do great. So the solution is to like in such cases, either you you are gonna uh fine-tune your model. Um, that's something that your data science team will do. Um, but the other solution, the simpler solution is to like combine it with keyword search and filtering to handle such cases. Yeah, but the long-term solutions you can just fine-tune and and the the whole point is that overall when you train your model on your data, it becomes more statistically relevant and uh the result end results are gonna be better So yep, um and a bit more about things that go beyond simple search. So for example, as I said, like uh I have described you like

Speaker 1: stuff that is mostly about text search so far but uh as like what google ends can do uh you can do multimodal search like for example you can search for images video audio all that stuff is like doable now And then you can do recommendations. I did show you a glimpse of that. Then you can do content discovery, which is a new frontier. So what you can do is like you can explore the vector space. And this is impossible to do with text search, right? Uh with vector space you can do like you know I have a point here and I I like this and I like this and with this you can just traverse through the space and like build very new uh engaging experiences for your customers where for example they start with a shoe And then the specify I want to add red to it and then I wanna decrease some texture from it. Then it will move to a different direction. Then it will move to a different direction. So it can be a lot more iterative.

Speaker 1: It's a beautiful like new concept that's a lot of people are looking into. So And there's also like the concept of clustering where you can uh cluster your data and understand your data better. And there 's a lot of use cases around anomaly detection. You can find out uh you know like outliers in a data set and And also there's this very common thing called RAG, where you can combine uh these vector search engines with a chat GPT model And uh it will just basically query your search engine first to get the relevant results and then you feed those results to the LLM and it like makes sense of your document. So for example, a good example is uh perplexity if you guys have used it um or uh chat with your pdf kind of use cases where you pass in your pdfs

Speaker 1: and it creates vectors out of that and it is the results are passed to uh ln lm and you are able to ask questions directly from uh your pdf files so yeah that's one more use case and there are many more like people are still scratching the surface so Yep. So overall uh the whole summary is that vector search is faster uh at scale and more relevant and uh vectors unlock new paradigms for search, which is like multimodal search, recommendations, exploration of the vector space, anomaly detection. And these are like just not possible with traditional text search. And also vector search are like great, like the dedicated search engines are great if you're running at scale. They allow you to basically tune between the quality, cost, and the latency of the search depending on your requirements

Speaker 1: And you can find me on Twitter at uh k shavindu. dev slash twitter and there's my LinkedIn QR code. Um and thank you. Uh if you have any questions, feel free to ask. Uh

Speaker 2: we have a couple minutes for questions. Yep. Cool. Thanks for that.

Speaker 3: Uh f for development, I'm curious, do you have experience with uh Um having your vector database like locally basically like do you work with a remote database for for development or would um a developer have to to create that locally? So I'm thinking of like a common Django setup would be You you clone your Postgres database locally and you've got your uh vector fields in there, but I don't know about the the vector databases.

Speaker 1: Okay, so it's similar to again Elasticsearch. So you can run it as a cluster, right? It will be like you can run it locally in the same machine, right? It could be like a Docker composed thing where you spin it up together Or you could run like as a separate cluster in Kubernetes or however you want, um, and it would work. I hope I was able to answer the question.

Speaker 4: I'm curious about that content discovery use case you were talking about. Can you go in a little more detail on that or some examples maybe?

Speaker 1: Okay. So the uh whole point is that um Like with vectors, you can like not it's not like you can add that whole point of direction now, right? So uh but you cannot just specify arbitrary directions. So the way we do it, I unfortunately do not have the slides for it. But the way this API works is that you specify it positive and negative pairs at any point of the iteration. So you are let's say at this point, you specify okay, this is the positive uh point that I want to go towards. So for example, your original query vector is this, which is a Nike shoe. Right. And you say like you know you convert the term red into a vector, right? And it gives you the direction of vector. And uh let's say the vector is somewhere here. So the engine will try to point a bit towards this, right?

Speaker 1: Keeping this one in mind. And then if you specify something negative, uh it will try to push away from that. So it's like you query it over time, the same API with the new context that you are trying to build. Right, and you are gonna like be starting from the shoe, then you go towards a new point, like this is the red factor, and then maybe it doesn't exactly go towards this because there's a negative pull as well. So it goes somewhere here, then you again call the API with the new to context. And like this, the user can just discover new stuff in your space. So it's it's very powerful. Like so, yeah.

Speaker 2: Uh we are at time. Uh if you have any more questions you can reach Kumar in the hallway. Um please uh give again a big applause.