Elasticsearch: Accelerating the Django Admin by Kate Kligman

Yeah, I think that's a good thing.

Speaker 1: All right. So we're just about ready to start. Thank you, everyone. For coming to my talk. Today I'll be talking about Elasticsearch and accelerating the Django admin. My name is Kate Kligman, and I am a staff engineer. on Grove Collaborative's platform team. I'll be posting the slides for this talk on my Twitter account and website soon after, so you'll be able to look at some of the code snippets. Grove Collaborative is a San Francisco-based startup. We are a Django shop. Right now we're running a monolithic setup. With the 1. 11 line. We host with Heroku. I'd say we're a kind of like mid to end of life on Heroku

Speaker 1: because we are expanding. We also have a fairly beefy setup on Amazon's RDS Postgres. And our solution for a long time has been if the website is Add another read replica, and everything will be alright. But then there was that one time that we grew over 7,000%. In three years. And adding read replicas didn't quite cut it anymore. On the one hand, we made Inc. magazines top 5,000 fast-growing startups. We were number three. 37 on the list this year. But on the other hand, we had a Django admin that we desperately wanted to survive this transit.

Speaker 1: The Django admin has been very important for us in the past because we hadn't yet reached the point where a lot of our customer service tools would be spun off into their own systems. The company started with a monolith and we still have that monolith. And so protecting the monolith so that we can move into To the next phases of growth was critically important to us. Likewise, preserving the Django admin's full text search was important. Important as well. We have customer service representatives who have to perform a substantial amount of customer lookups, and it's much less cost Cognitively challenging for someone to rapid-fire lookup if they're just pointing and clicking into a single text field.

Speaker 1: So we didn't care so much about the other portions of the admin. Quite simply put, we wanted to be able to search from it and display records and then navigate the data structures without the admin taking too long to do so. This was our data model for search. We simply had a first name, a last name, an email address, and phone number, and the first line of a customer's street address, just one address. But behind the scenes, Django would produce a query like this. Now it would full text search all of the fields, transform them, and then on top of that it would walk any relationships

Speaker 1: That we had. When there were a few thousand records, this flew. It was pretty good. When there were a few hundred thousand records, worked fine. Few million records, it still worked, but it would add a few seconds to the search. But as we expanded beyond that, the time that it took to perform a search increased considerably. At the time we started the project to migrate our search To Elastic, we were at the 12-second mark, meaning that one query took 12 seconds. And the doing a single field full text search almost doesn't matter if it takes so long to actually search. You have an interface that pretty much is unusable. And for every couple of weeks that we didn't have a solution, one second would be added to the search time.

Speaker 1: So We were it was it was visibly the the ship was kind of visibly um sinking. So we decided elastic search. We knew we were gonna use elastic search almost right away. We had heard good things about it. We had some domain experience at the company with large data sets. And more importantly, we were designing for For exponential growth. We didn't know when our growth would end, but we also didn't want to be in a position a couple of years from now or even six months from now where we went with a lesser solution and then had to re-engineer everything. Everything, finding ourselves right back in the place that we were. So full speed ahead on Elasticsearch. Elasticsearch is an open source project Project, it's written in Java. Um

Speaker 1: it is uh built by and maintained the folks at Elastico, so it's a company-backed, well-funded open source project. The terminology is a little bit different because Elasticsearch itself is not a database and it's also not really a document store. actual search engine but it has a lot of similarities to SQL. So for example, an Elasticsearch, an index, is kind of like a database. And And you define mappings which sort of act like a schema to define your data. Fields are kind of like columns, and because Elasticsearch is document-based, that sort of kind of resembles. But

Speaker 1: unlike SQL, Elasticsearch mappings are brittle. If I wanted to change or manipulate my fields after I've stored data in them, in Elasticsearch, it's not an uncommon proposition. To drop the index or the database altogether and then re-import your data. In SQL, that wouldn't really fly so well with large data sets, but in Alaska, Search because it is so fast, it's not a problem at all. Likewise, it is fast because it uses inverted end indexes. If I had a document that says Django is awesome, Elasticsearch would split it up as directed by us It would then produce tokens for each of the things to be searched on, and it would map them to a document

Speaker 1: ID. So if I search for Django in this, Elasticsearch would say, oh, document one One, two, three, four, five has Django and send that back to me. This is perfect for hooking it into the Django ORM. If I have a customer record and I've split it out and this particular one, it's my name, Kate Klingman, and a phone number, I can then correlate that to an Elasticsearch document ID, and I can set that ID to be the primary one of the primary keys on my Django model. So customer 12345 could be my customer ID. And this makes it very easy to hook. Into the ORM. This is what the workflow looks like. So I perform a search in the admin.

Speaker 1: The search gets shipped Off to Elastic. Elasticsearch will return a list of IDs corresponding to the ORM and to my SQL database's list of IDs for these records. And then all I have to do is go back to Back to Postgres and look them up and rehydrate them. And this is what it looks like. So in the top block, I'm pulling out the IDs that I've set for the result. of an Elasticsearch query. And one cool thing about Elasticsearch is that because it is a search engine, it does ranked ordering as well. So, if I didn't care so much about the order of what was being returned, I could just simply look it up using the ORM.

Speaker 1: But in this case, I've added a little bit of extra code so that we can preserve Specificity. And now that my lookup is a query set, I can feed that right back into the admin, and we have a complete customer search. Hooking it into the admin is also pretty simple. So when I perform a full text query search, Django uses, in this case for us, the Q parameter. I can then hook it into the class definition. There is a function called getSearchResults that will take a parameter. We can then take Take that query, those query terms, ship them off to Elastic, and then

Speaker 1: receive the result set back, rehydrate it into a query set, and ship it into the admin. It's actually this simple to do it. Updating Elasticsearch isn't so bad either. We can use uh post-save, post-create, and post-delete signals. Model hooks can also be used. When we're adding data to Elasticsearch, we have to to keep it up to date. I would highly recommend using streaming services or schedule jobs. So for Grove Collaborative, what we did is we have a celery beat process that runs every couple of minutes. And we pull the customer's table to see if any records have been updated. If they've been updated, then we pull the records, ship the relevant portions off to Elasticsearch.

Speaker 1: And we are good to go. And in our particular use case, the interval is important because we have customer service representatives querying this system all of the time. So if a customer Within the first few minutes of a customer's interaction with our website, they need customer support help, the customer service representative is going to have to be able to find their records in very short order. And so this is a these this particular pattern is something you'll see again and again. I don't recommend um hooking in individual records simply because Because if you're designing to for a system that could scale massively, um, it's much, much better to use batch inserts. That left us with having to figure out how to get the data in and out

Speaker 1: of Elasticsearch. One of the first things we did was check to see if If this was a solved problem. There are many folks with Django admins, and there are many folks using Elasticsearch. So we went through and looked at Django modules that could help advance us fairly quickly. We decided to do our own implementation for two reasons. The first is that Elasticsearch moves very quickly as a search engine engine. And many of the modules that were publicly supported were pre-beta 1. 0. I think I saw one for 2. 0. And there was one for five, but it didn't look very well maintained. Or supported. Elasticsearch right now is up to version 6. 4, and every major version change is almost like a new engine.

Speaker 1: Even the minor versions are subs. Substantially different from each other. And I will get back to that. Likewise, we didn't know how long we would keep our Django admin. We wanted to have, we'd already started to break out search into separate API endpoints. We knew we wanted the admin to work this year. We probably wanted it to work next year, but we didn't want to isolate and cement ourselves and To a heavily Jangified system if we didn't know that was the direction we were going to go. Nevertheless, we had to get it working. There are two official Python implementations. For Elasticsearch. There is a high-level DSL library that makes it ridiculously easy to model and serialize and query data.

Speaker 1: from Elasticsearch. It sits on top of a lower level Python library called Elasticsearch Pi that more clues closely resembles the Elasticsearch documentation and any like raw queries you might see. If any of you have looked at YouTube videos on how to use Elasticsearch, the output you've seen is probably closer to Elasticsearch So here's our customer data model. Modeling this into the high-level Elasticsearch DSL was trivial. This is the model. Well, we have to do is create a class, define a few fields, and the library did the rest for us. The main difference between the keyword field and the text field in this case

Speaker 1: are that the keyword field is an immutable field where Elasticsearch, when it sees a text field, it will sometimes shred it depending on on how you define it within the search. But other than that, this is functionally equivalent to a bunch of fields storing strings. And we were off to the races. It was very easy. The queries were simple too. We could define a search here. There was a little bit of work in looking up the Elasticsearch documentation and then trying to To translate it into the DSL, but our search needs were very simple and everything appeared to work fine. But there was one issue with that. And the issue was that the more we used Elasticsearch, the more we wanted to use it and do more things with it.

Speaker 1: Suddenly our needs changed. We were originally Originally modeling just one line of the customer's address, but a few months later we realized that we actually wanted to model more than one address per customer. which necessitates a nested query within Elastic. We went to look up the DSL implementation and we found out that it wasn't quite there. Now things have changed since then. Some of the bugs and issues that we encountered. Have most more than likely been resolved. But we found that as our implementation got more complex, every time that we wanted to utilize the DSL for its simplicity, It became almost a major undertaking trying to figure out what the translation would look like.

Speaker 1: Eventually we got to the point where it became a hazard. We didn't have a full-time Elasticsearch developer on this. People would float in and out of the project. And so having an abstraction that was so abstract that we couldn't match it up to the public document. documentation um yeah uh was was a hazard because with developers floating in and out of it they would have to learn they would have to understand the documentation then understand the DSL then try to figure out if like what they were doing in the DSL was really actually close to the documentation and then also figure out any bugs and underlying structure. So we walked back our DSL employee implementation and decided to just go straight Elasticsearch, which is what I presume everyone else does.

Speaker 1: This is what the new query looked like. It's to me it's actually a little bit more readable. Than the old one. It's the same form and structure. It's JSON, it's dictionary-ified, JSON-ified Python. And as our queries got more complex, The structure got a little bit more complex, but this matches what's on the website. I can go to the Elasticsearch website right now, and if I don't quite remember something, I can tweak it and look it up, and it'll work 100%. So the takeaway from this one is own your search. If you abstract your search, it might not lead you where you want to go, particularly if your complexity increases. And keeping parity with the official docs for us was a huge way to de-risk

Speaker 1: our project. All right, the second lesson we learned is that Elasticsearch is pretty cool. If you don't define A mapping, which is a schema, for your data, Elasticsearch will define one for you. And it might not do the best job or any job. for that matter. It'll just run away with it. And that will result in your queries malfunctioning. In our case, we would the queries would not quite they would work but not quite work. were slower than we expected. Our performance in Elasticsearch is near instant for millions and millions of records. But that one time that we forgot to create our own mapping really nailed us. And the slow

Speaker 1: The solution is just delete and try again because it takes almost no time to import entire data sets into Elasticsearch. I will also say that when you're working on your Elasticsearch implementation, like 99. 9% of the time if you ever have a problem with Elasticsearch, like this is the problem. Just you're done. This is it. Just see this slide. All right, this is a more embarrassing version of the same problem. Elasticsearch changes very, very frequently. Even the minor point changes can reflect substantial Substantial renaming of terms. Elasticsearch also doesn't have, it's not very verbose with its error messages. It'll just tell you like this field doesn't exist.

Speaker 1: So when you're designing your queries and you're modeling your data, be very sure that you are looking at the exact version of Elasticsearch that you are using because it does matter. And It matters quite a lot. They make it very easy to do on the website, and they also make it very easy to accidentally ignore. So when we started, we were at version 6. 0, which was current. And as the implementation progressed and we got busy with other things, Elasticsearch moved to 6. 4. Um last week I was working on modeling sort orders with nested data and I was using the 6. 4 version. And that did not work out so well for me. So this is a very simple risk mitigation strategy, and it will save you a lot of time. Okay, testing. This is a big one.

Speaker 1: Spot checks on Elasticsearch Quarts. Can appear to be quite valid. My name is Kate Kligman, so it's a very unique name. Not very many people have the name Kate Kligman. When I was testing our customer search implementation, I used my name to test it. And voila, everything seemed to work great. The queries were working, they were finding me, it was perfect. But my spot check wasn't enough because we had a co-founder named Chris Clark, which is a uh more common name. So, and that did not exactly produce the results that we were expecting. Testing these queries is important because a lot of Elasticsearch can be very, very specific depending on how you write the queries and model the data, or can be very, very expansive.

Speaker 1: And there's a there's kind of like a fine point between in your queries where you're returning just enough data to be relevant to what you want to do, particularly if you're searching very large data sets and the searches themselves are A bit fuzzy. How we mitigated this risk was we started running batch sets of data through a testing system to check the ordering. And this became part of Of our integration tests for Elasticsearch. So now, whenever we decide to modify Elasticsearch in any way, we have sets of predefined data that go through it. First, we compare it to data that was run through previously, and that lets us know if the scope of our search changed or if there's anything else That's going on. And it helps to keep our searches honest. And I can't recommend this enough

Speaker 1: because with Elasticsearch, sometimes you're potentially potentially searching such a wide amount of data, you can't spot check it. It's not possible. So use um use tools to your advantage to do this. All right, this is a big one. Explore all of your hosting options. When we started our implementation, we hosted on Postgres through Amazon. So it made sense that we should also host Elasticsearch on Amazon. So that we could have all of our managed hosting together in one place. Perfectly logical at the time. But what I found was, or what we found, was that not all managed hosting is created equal. With um

Speaker 1: now this list, I'm presenting this uh without endorsement. Uh this is not an inclusive list by any means. There are many providers here. You can also host Elasticsearch or yourself but when we when we made that decision to go into Amazon our first roadblock was that it's hard to connect to Amazon Elasticsearch instances are very easy If you are going to have an open data set that's visible to the world, you can just set it up in one click. But if you want any kind of security or authentication, particularly at the time we did it, you have to go through a more advanced header signing mechanism. And there was not any clear documentation on how to go about doing that. So it became like an extra bit of work.

Speaker 1: Work we had to do. But that wasn't it. Amazon also didn't support a lot of the plugins that we wanted to use. In fact, they were pretty inflexible about it. My understanding is this is just Changed a little bit, but at the time we were diving in, we could have really used that support. There are workarounds now for things. But honestly, when I look at the amount of operational work that goes into To running something that is available by default on another managed hosting provider, you know, I don't know if we would have made the same decision. But not to knock Amazon. They did do one incredible thing for us and that is their high availability for their hosting. So I gave a version of this talk uh A few months ago at a DjangoCon meetup and I was praising Amazon.

Speaker 1: Oh, they're never down. Our hosting is great. And the day after my talk, the day after we had a hardware failure on one of the nodes. Um it was Karma maybe. And uh but with Amazon itself healed and we were back up before I could even sit down and action, you know, and take action to deal with the failure. So there is something to say for managed hosting. Particularly if you don't have a dedicated team or a dedicated developer who just does Elasticsearch. All right. So this was our final solution. We went with the lower level Elasticsearch Pi to have parity with the website documentation. We finished on Elasticsearch 6. We haven't had a reason to move to 6

Speaker 1: 6. 4 simply because every time our data models change , 6 seems to do fine for us. Our Django admin uses the same search as an A API endpoint that we have. So we achieved that goal and we preserved our admin. And our searches are really, really fast. In fact, the latency is the the f is the largest factor in our search speed. The elastic search portion is hardly measurable. It's also very high precision. We replaced this amorphous full text search in Postgres with a high precision search engine. engine that does a much much better job of collating the records that we need. So this this like I said, I can't recommend Elasticsearch

Speaker 1: enough for this. These are some resources for Elasticsearch. I would suggest checking out some of the Django projects in particular. For this to decide if you're going to do a s like a small data elastic search solution or if you're planning to grow and scale with your company or your business for a long time. And And I do believe let me see yeah, and we have some time for questions.

Speaker 2: Hi. Uh uh you said before that you were using DSL in the first place. I'm wondering how hard it's to convert from DSL to use the elector. Shh, pie.

Speaker 1: Sure. So it's as hard as the learning curve of learning Elasticsearch in the first place. We We are with our particular issue, we were using the DSL as a crutch because we wanted to be up and running very quickly. We were under time pressure, the admin was not doing So well, and the DSL seemed like the way to go. The largest portion that we spent time-wise was learning how to map a lot of the data ourselves, and then To some extent, performing the queries that we could not do with the DSL. So I'd say the difficulty level is medium. If you already have an implementation, you are halfway there. And the query syntax doesn't change so much.

Speaker 1: The real issue with the DSL is if you're completely embedded in it later and then you have to go back and change everything.

Speaker 2: So um can you s can you say that it's like pretty it will be pretty early when you hit that point that you have to switch from DSL to Pi? Because I'm I'm wondering like if I wanna use elect search for one of the um one of the project uh uh in the company, I wanna do like a little demo first. to and present it to everyone and to say, oh this is good. And then we dive into it and well like pretty early we'll we will know that oh wait we have to switch to pi. I should have it down here like a first place.

Speaker 1: So there's a little bit of reputation on the line for not having to walk back. Yeah. I would say go directly to Pi. Um skip the DSL. It's not worth it's not worth it.

Speaker 3: Hey, thanks. Um maybe I missed this, but how did you did you create a new custom UI or did you just somehow like hook into the Django admin and then s maintain the UI but then it hit the uh Elasticsearch backend?

Speaker 1: Cool. So what we did is um so if I understand your question correctly, so we kept the we kept the Django admin, we created a separate class to encapsulate the hydrate the rehydration. logic and then the admin would just use that class, feed it the search parameters, and then spend it back. And we were also able to reuse that class for multiple API endpoints and other search parameters. use cases down the line.

Speaker 4: Cool talk. You mentioned that you ha when you have problems with implicit uh schema mappings, you just delete and like recreate or Create a new index. Is it easy to move data out of one index into another very quickly, especially if you're changing the mapping?

Speaker 1: Yes, it is. Elasticsearch has some features of it. its own that will that that can allow you to like migrate to perform data migrations without having to do a full external import. In our case though, it was so fast to simply just drop everything and redo our setup that that's I that's still how we're managing that particular use case.

Speaker 4: Do you care about losing data when you drop it?

Speaker 1: No, because we use a form of versioning. So each of our indexes or databases is a version. And if we are making a schema change, what we will do is increment the version number by one. And the index is named, or the database effective. is named by version. And then we will go ahead and port all of the data and set it up as a mirror. And then we will flip our system over to use the new index. and then drop the old one. But it happens very, very fast. When we initially did this implementation, we would just turn Elasticsearch off, wait 10 minutes for everything to import. And then turn it back on. It takes a little bit longer than 10 minutes now. So we switched to having rolling indexes. But um yes. But yeah, to answer your question, yes, a lot of times if you're changing the map.

Speaker 1: You do have to drop the database or the index, or it's a little bit more involved with Elastic to have it migrate some of the data from one to the other. Oh here, can you repeat the question to the microphone?

Speaker 5: Um how how big is the customer table and uh how often do you quarry it to like push it to Elasticsearch and then

Speaker 1: What I can say about the size of our customer table is this for every one million customer rows, it distills down into less than a Megabytes of Elasticsearch storage. So far, we have not been able to make a dent in the query speed with how many records we've been adding. And our update frequency I believe that was the second part of your question, is two minutes. So we have a task that just polls every two minutes to look for updates.

Speaker 6: Hello, thank you. Good talk. This is semi-comment, semi-question over here. Okay, I have a question too. Okay. When we implemented our search a couple years ago, one of the problems we encountered. uh was with um using signals was race conditions and we kept getting records that were being deleted and added at the same time. Is that something that you encountered when you tried signals out or do you tr you do you try that at all?

Speaker 1: Yeah, I looked at signals early on but it became clear from the sheer volume of data that we didn't want to do it piecemeal um and that like bulk updates were a much better choice for us. So we dodged that particularly. bullet.

Speaker 7: I was just wondering if you could say a little about uh doing geographical searches and how that affects query time.

Speaker 1: Ooh, I have not done geographical searches Graphical searches. I feel like I have done, I worked on a different implementation than a customer search. I did the product search for Grove as well. And we used almost every type of search that allowed

Speaker 7: Okay. I'll have to try it.

Speaker 1: Yeah, just give it a try. Do a talk about it.

Speaker 8: Uh did did you uh if you guys are using Postgres, did you look into the uh full text search uh properties of of uh Um Postgres as opposed to going straight to Elasticsearch?

Speaker 1: Oh yeah, yeah. There were several more than several attempts were made to save the Postgres implementation. Yeah, um I would say for each attempt that we made, um like our like the amount of data that we had to process increase because we were at at the time we were working on an exponential curve. And so um for us, we would make a dent in it with Postgres. We would, we would, um, we like we added indexes, we worked on some of the full text features. And what we found is we were spending a lot of time massaging something that was still slowing down. And since we didn't have an endpoint for when our growth would stop, hopefully there is no endpoint for that. Um we decided to go with a longer lasting solution.

Speaker 7: You mentioned that sometimes when you handle like a a a large number of records, even in um Elasticsearch it can become really slow. And you said something about um finding a sweet spot, you know, oh yeah to get just enough data for the search.

Speaker 5: What criteria are you using to kind of limit the scope of the search.

Speaker 1: Cool. That's a gr that's a great question. So the first thing is we tried to figure out exactly how much data any given search should return. If I'm a customer happiness um representative And I'm searching for a very specific customer address, I might be disappointed if more than 10 records are returned. Um, or I have to cognitively go through a lot of extra results. So the first parameter was always a visit. Business case, like what would a good search look like from the perspective of the user performing the search? With Elasticsearch itself, they have a uh there's there's something that's very similar to SQL's length limit parameter, so you can just cap the number of queries, or you can cap the number of search results you want back from it. And Elastic will use its algorithms internally to adjust how it queries.

Speaker 1: And It's an interesting point because I found that if you say I don't want any more than a hundred records to be returned, Elasticsearch doesn't consider that a hard limit. It'll say, oh, okay, I'll give you 50 to 80 instead of 100. Another uh thing that we looked at was Elasticsearch has a lot of implicit syntax in its queries. Very, very early on, we were hit with slow Elastic Queries because because we had one of our queries had an implicit or instead of an and. And so Elastic was doing too much work to query too large of a data set, and we didn't actually need most of the records that it was querying. So keeping the searches very, very specific and also matching them up with your use cases is really important.

Speaker 1: I've also found that they're There really aren't a lot of use cases that could overload Elastic, at least for our purposes, simply because there was never really a scenario where we want Elasticsearch to return more records than our systems could handle.

Speaker 7: Thanks for the talk. You mentioned uh you are using also Elaska Search for ordering and you are using the case when to get the index from a lot of search so those that paginate well or it should Just like generates for all records. Do do you have an answer for that?

Speaker 1: Oh. Because the number of records that we ultimately want to return is small. So if you're if you want this to return like 100,000 records, okay, you might need a slightly different way of doing it.

Speaker 7: Using that or like it generates for our records of the of the query.

Speaker 1: Can you repeat that?

Speaker 7: Because like you are doing a case win, an expression, right? And like you are you could be paginating by that expression like on the f on the front and forward Django admin. I don't know if pro Postgres will paginate using part of the result or it will generate all the case when and then

Speaker 1: Oh, that's a good question. So the answer is that it will generate them all. It won't, yeah, yeah. And uh how we are able to get away with the is because our query needs because our the data sets are large but our query needs are small um we d we eventually dropped pagination from our Django admin. But if we wanted to add pagination, it's act it's it's fairly easy. We can Can return the larger subset and truncate it, or we could have Elasticsearch. Elasticsearch itself is capable of pagination too. And so we could add a little bit more complexity to our system. to um to enable that.

Speaker 7: Okay. Uh but if you paginate on Elasticsearch and like you add an additional filter on the Django side. Then you wouldn't be able, right? You are doing only filters on the

Speaker 1: Yeah, yeah, that's right.

Speaker 7: Okay.

Speaker 1: Yeah, so it's it's yeah, it's kinda in between it's in between both systems.

Speaker 7: That's great, thank you. Okay, well then let's um Thank you all for your questions, and let's thank Kate again for her wonderful talk.