Where next for Wagtail Search?

Speaker 1: Let's get started. Hello everyone. I hope you've been enjoying Wagtail Space today. I'm Ian , the Chief Growth Officer for Wagtail at Torchbox. And I have the great pleasure of introducing Matt Westcott, our Wagtail Tech Lead. He's going to tell you where next for Wagtail Search. Matt is a member of the Wagtail Core team. He's been working on Wagtail since day one, over a decade ago. And if you have questions for Matt during the talk, please put them in the Q<unk>A. Over to you, Matt. Thanks

Speaker 2: Ian. I'm gonna eat. So find my uh window to share. Hopefully you should be seeing that now. Yep, so yes, as uh Ian said, I'm uh I'm Matt. My pronouns are he, him, I'm the tech lead on Wagtail and one of the original developers on the project, and you might also have seen me around under the name Gasman. Um, I won't explain that one. Um so I've uh spoken a fair bit about the guiding principles behind the creation of Wagtail at previous talks and writings like The Zen of Wagtail But if there's one strand that runs through all of them, I think it might be this. Choose your battles So the main battle that we took on was to create a great interface for authoring web pages.

Speaker 2: But one battle that we decided not to take on was creating a great user interface for defining the data model. of a web page. We decided that writing that in code was good enough. Could there be a better system? Maybe, but that wasn't a problem that we were trying to solve. Likewise, we didn't take on the battle of making a tool for doing page layout in the browser. We felt that HTML and CSS would serve that purpose and we didn't see a need to replace them But if other folks, like uh Code Red, for example, see an an opportunity to fill that gap and make Wagtail into a user interface for page layouts, that's great. Uh everyone can choose their own battles.

Speaker 2: There is a pool of wisdom from the days when everyone was running their own blogs saying to uh never write your own CMS. Now, um obviously if we uh if we'd followed that advice ourselves and if the Royal College of Art hadn't made the leap of saying We've evaluated all the CMSs and they suck. We want you to build us a new one. That might not have been their exact words. Then there would never have been a white tail. We could maybe revise that statement to never write your own CMS unless you're on board with becoming the guy who makes a CMS. possibly to the exclusion of everything else, beginning with that blog that you were intending to write in the first place before you got distracted by making a CMS.

Speaker 2: Now one battle that I really wish we could have taken on and I still think someone should pick up is a really good video management system for Wagtail dealing with things like transcoding, large file handling, thumbnails, subtitles, and so on. As Tom showed in his talk yesterday, the building blocks are basically all out there. But doing all of that would have been a pretty sizable project that it that essentially runs parallel to Wagtail itself And that was a distraction that we didn't really need. And for most purposes, embedding videos from YouTube or Vimeo is a workable solution. All of which brings us to search. It's an essential ingredient of any content-based site, and yet it's

Speaker 2: another separate parallel problem to the task of getting can content into a site. And in this case, there wasn't really an existing solution that handled this adequately We wanted to be able to perform search queries as easily as Django allows us to make database queries, uh, being able to just take the existing ORM syntax and just add in this search clause to be able to search any any model that we have this set up on. So in the end, this was one more battle that we chose to take on. Now this was uh the the work of Carl Hobley, another of the original Wagtail developers

Speaker 2: And as I said, it it's a problem that makes sense to tackle independently of the rest of Wagtail And so while this has always been shipped as an app within Wagtail, there's very little about it that's specific to Wagtail. And unlike many of the other subaps of Wagtail that you're used to, like images and documents that depend on the Wagtail core and admin modules to function. For the search module, it's actually the other way around. Wagtail Core is built on top of Wagtail Search as a building block. And this meant that we, in principle, we always had the possibility open to split it out into its own standalone projects, which

Speaker 2: sounds kind of attractive, but it's the sort of thing that you would only want to do once we felt that it was stable enough and it had proved its worth within Wagtail. And like many sort of things in Wagtail's development, that has always remained this sort of nice to have pipe dream. Um other more practical concerns have always tended to take priority over th those sorts of things So what makes this such a tricky problem that apparently no one had solved already? Well, we didn't want to be limited to the functionality of the database itself. A best in class search implementation isn't just about matching strings of text.

Speaker 2: You also need to take care of details like stemming and synonyms, so that's tweeting similar words such as bake, baker, baking, bakery, all as equivalent so that you're not missing out on relevant results if you don't happen to use the exact same wording as the page does. Also ranking results based on like proximity and frequency of search terms. So if you're searching for pages about banana bread, yes, I'm going for the uh bakery demo uh analogies again. Uh you probably want to favor text where the words banana and bread are in close proximity, not pages that mention bananas in the first paragraph and bread somewhere in the tenth.

Speaker 2: These are often invisible details that uh users wouldn't think about and even developers would rather not have to think about. Uh but search results will just seem off if you don't get them right This behavior is often complex and unintuitive. We sometimes have clients questioning the micro details of search results, asking things like Well this page has the word banana in it twice, so that should be ranked above this other page that only has it in once, right? And yeah, it's not quite as simple as that. There's uh all sorts of factors that uh You can't really uh product on predict on anything more than more than vibes, really.

Speaker 2: So for a long time these features were only really available in standalone search engines such as Elasticsearch. And that meant that for your site's data to be searchable, you had to keep this separate parallel copy of your site content within Elasticsearch 's data store. So firstly that meant that you had to keep that copy in sync with your database as the primary source of truth. Usually that's not a big deal. We have a management command to do that initial population. And we can use Django signals to respond to individual pages being created or updated and push that to Elasticsearch as needed. But secondly, you have to get a little bit creative when

Speaker 2: you're making queries that combine text search with database style filtering in the more conventional sets. For example, if you want to search for pages created in the last six months, that would be a query like so take uh the page dataset, filter that by ones with the first published at greater than uh the April the 9th, and then search that dataset for The word baguette. And Django knows how to turn that filter expression into a SQL query to run on your database. But we want this query to happen on Elasticsearch

Speaker 2: and Elasticsearch doesn't talk SQL. So inside this search method, we have to pull apart that query object that we've already constructed up to that point and translate that into the uh JSON query that we can send to Elasticsearch. And that requires digging into a certain amount of Django internals to do that sort of translation. And uh so that means that Elasticsearch has to keep a copy of even more of the database contents to allow it to do these queries based on these other fields. And also a large chunk of the logic that exists in Django for making sense of these filters had to be replicated in Wagtail's search module.

Speaker 2: Now over time the databases that we use within Wagtail, so that's Postgres, MySQL, SQLite started y adding their own full text search functions and those were made available within Django and so in Wagtail 2. 15 Following a Google Summer of Code projects from Al Dan Creo, we added a database backend that's now the default one for uh new WhiteL projects out of the box, which adapts to whichever database you're using and calls the relevant full text search functions from that database. It's always been a key design goal of Wagtail Search that all backends work in the same way.

Speaker 2: It should be possible to switch from one to another without changing any code. That's a goal that we've had mixed results, mixed success with as each backend comes with different challenges and points of friction. If you're running a full text search directly in the database, then on the one hand you don't have that problem from Elasticsearch of keeping two data sources in sync and translating queries from one format into another. On the other hand, um because databases have this sort of much more rigid structure where the schema has to be defined up front. That means that um the way we have search set up, uh we essentially have a single field that holds all of the indexed content of a page

Speaker 2: in a form that can be queried. Um, it's all like one big pot of tokenized text soup that doesn't really keep track of which fields that text originally came from. And so it's not currently possible to run searches that just search the title or just search the summary field or whatever it may be because If you're searching, you're searching on everything that we indexed for that particular page. So Carl, the primary developer of Wagtail Search, left Torchbox to move on to other things and left the Wagtail project along with it. And this is where we encountered the bus factor in full effect and the hazards of taking on one battle too many.

Speaker 2: So um we lost a lot of the institutional knowledge about Wagtail Cert that Carl had and Really that's not the only fact that has held back developing Gwytel Search in the way that we wanted to But one thing, Elasticsearch has been a bit of a moving target. In Elasticsearch 5, if you wanted to boost a given field in the results so that for example a search term that matches the page title appears in the results with higher priority. It used to be that you'd set that up at indexing time and then you would have that take effect on any queries you perform afterwards But as of Elasticsearch 6, they suddenly deprecated that whole thing and

Speaker 2: now they want you to specify the boosting rules at the point of making doing the search, which didn't at all match how we have the search index set up. And it took us a long time to catch up with that and restore the boosting functionality in later versions of Elasticsearch. And they also moved the goalpost on the license, switching to a commercial license that made it unusable by organizations like Amazon Web Services. So in response they created a fork. which is OpenSearch, which was backwards compatible with Elasticsearch seven, meaning that we could use the same backend and client libraries. At least that was the case up to version 7.

Speaker 2: 13. 5 of the client library, um, which uh doesn't sound like the sort of uh version number way would be making massive breaking changes, but that's when they just decided to shut out anything that wasn't an official Elasticsearch server. So you could continue using the older version of uh of the client library with open search, but obviously that's not a great solution as these versions reach end of life and uh You want to carry on using a supported version of OpenSearch. And WagTail's own ambitions have expanded in this time in a way that the search module hasn't really kept up with When we redesigned the uh page listing view in Wagtail 5.

Speaker 2: 2 and later extended that to other listings in 6. 0, we put searching front and center. And at this point search isn't just a way of discovering information, it becomes a primary tool for navigation, for getting around the admin. And there may well be occasions when those two goals are at odds with each other. So take this scenario in the screenshot. If you know that you want to edit the the baguette page then being able to just type in the start of the word baguette and head straight there is a major speed boost to uh using the um the the admin. But if you're performing a search in the conventional sense of

Speaker 2: knowledge discovery and you want to know like what pages do we have that talk about bags. it probably isn't helpful to have results about baguettes mixed in there. So maybe that's something that we can handle with intelligent rankings promoting the exact word matches over the uh partial ones. Or maybe this calls for a rethink of the UI uh such that so that running a search is a deliberate step that's distinct from getting these results continuously as you type. Because this uh particular user interface is actually a fairly uncommon one. Um usually you would see a sort of drop down by the search uh the by the search box. with the auto-completed results

Speaker 2: and actually getting search results is a separate step. So maybe that's something that uh we need to think about uh adopting. This is all something that we're going to have to decide. Another aspect that may not be at all obvious uh on this screenshot is that here we're searching the uh d specifically the breads section of the of the website. Um so if uh so so that would be transl again that this is uh Uh filtering to just this section is something that we need to uh do as a sort of classical database query, which then if you're running Elasticsearch, is then

Speaker 2: passed to Elasticsearch to run. So that code would look something like this sort of descendant of Bread's index and run an autocomplete search on that. But um this anything that's of involving querying the tree structure that's uh reliant on the uh the Django tree beard library and that one is doing all sorts of complex um querying and and different and esoteric filters behind the scenes that um we don't currently support in uh i i in uh in the elastic search back end So there's a sort of nuclear option we can go go go for at this point, which is to do the initial query to find the

Speaker 2: candidate pages. in on the database and then pass Elasticsearch this list of IDs. Okay , do this search but restrict it to IDs 100, 101, 102, 103, 104. That's the sort of query that it's you can get away with if you have a hundred or maybe a thousand pages in this section. But um once you reach a certain size, um uh if you get into sort of it tens, hundreds, thousands, millions, then um that's all going to fall over and this is really a kind of worst case scenario for uh of for for uh for putting a site into production if it only breaks once you reach a certain size of real-world usage.

Speaker 2: And uh unfortunately this is something that uh that some clients have raised as their sites have hit a certain size. And this is still an open project problem that we're we're hoping to sort of tackle in the uh the near future. So um so so yeah, the the um for the White Tail seven point two release Search uh has been my primary focus. There's kind of a lot of technical debt that we've uh needed to address. And top of that list is the uh the pipe dream that we never got around to all those years ago, which is splitting out Wagtail Search into its own package. And this is not just good for people

Speaker 2: using search on Django projects outside of Wagtail. this is a profound improvement to the way that we develop uh Wagtail 2. So one example being testing. So as as I mentioned, we want all backends to work interchangeably as far as possible. And in an ideal world, we'd verify that by running the complete white tail test suite uh under each back end. But not only is that A lot of tests to run on the on GitHub Actions once you consider all of the backends and versions of uh search engines we want to support. because we do care about our green credentials after all. It's also not quite as simple as that in practice, because

Speaker 2: as I mentioned earlier, search ranking algorithms have this black box aspect to them And that's a feature, not a bug. If Wagtails unit tests expect a particular ordering, but on one search eng one search back end results number nine and ten happen to be switched around. Well, that shouldn't really be considered a a test failure. But there's lots uh the the way that Wagtails tests are written, they're it are writ in this very sensitive way where it will flag up if results are slightly different or is there w there are one one more or one fewer query happening. So it's kind of in practice it's very difficult to take our existing Wagtail test suite

Speaker 2: and expect that to work across all backends So by having Wagtail Search in its own test harness, its own test front independent of Wagtail, we can be a lot more thorough and careful. and fine-grained about how we define what it means for functionality to be correct and how we define successful tests. So I've been able to do to make a lot of progress on that uh with this as an independent uh independent library and I've been improving the test coverage and documenting the uh the internals. as we go, again hopefully avoiding the reliance on this uh previous undocumented in institutional knowledge.

Speaker 2: But uh in fact, this is already old news because uh it seems that as I was setting up this uh new GitHub repo and moving old issues from the uh Wagtail repository to this new one some of those old issues uh naturally had Carl C C'd on them and that caught his attention and so he popped up on the uh the Wagtail Slack asking, um, you do know I've been working on the same thing, right? And yes, indeed, so s since uh early this year he's been working on this project Django Model Search. which had the same goal of taking Wagtail search functionality, moving that into uh into its own package. And so we decided to join forces on this project.

Speaker 2: And clearly there was a bit of face palming that we'd been duplicating work, but in fact this was the best possible outcome. Because so firstly, we don't lose Carl's institutional knowledge after all. And as Carl's focus has been on making it work independently of Wagtail, he's made some substantial improvements to the documentation. And meanwhile, my focus has been more on making the transition from Wagtail Search to Django model search as smooth as possible. And my aim is that when you upgrade to the next version of FigTale that uses this, you won't have to make any changes to your installed apps settings or change any imports around. Everything should just work as before.

Speaker 2: And uh yeah we're having some good success with that. Just in the last week we've got the full uh Wagtail test suite running successfully against Django model search. So this is really only the start of the process of uh improving our search uh functionality. There is there's lots of uh the bugs that we want to address and uh catching up on uh uh on new search backends uh new newer versions of search engines. And we've already already seen the uh initial fruits of that work. So um we we now support uh Elasticsearch 9 and crucially Elastic uh OpenSearch 2 and 3 uh using the native client library

Speaker 2: um rather than the uh old outdated version of Elasticsearch 7 But really this is um just the the beginning of what we want to do. Um I think as no no talk would be complete without a mention of AI and um while the yeah doing being able to to query to to uh l LLMs and uh and vector indexes to actually get different kinds of results is that's not exactly the s the kind of search that would fit into the API that we already have, but given that we already have this existing mechanism uh in elastic uh for Elasticsearch for being able to push

Speaker 2: data into this external data store and keep that in sync. That is something that we can definitely uh benefit from if we decouple that that bit of functionality from the actual searching logic. But yeah, really this is uh just uh m for now it's it's mostly uh about uh improving Wagtail's development process And yeah, it's really exciting to have have Carl sort of back on board indirectly on the White Hill project and uh And having the this the extra outside perspective as well to for to develop Django model search in a way that benefits both Whitetail and

Speaker 2: and for the wider Django ecosystem. So yeah, it's I'm really looking forward to seeing how that all develops. And that's me. Thank you very much.

Speaker 1: Thank you, Matt. That was a really fascinating look into an important aspect of Wagtail. We've got time for a few questions. The first one, uh, if you could just can you just confirm when will we see these improvements released? Uh

Speaker 2: yeah, so uh we so the uh the the plan is for at least the uh integration of Django model search to uh to to ship as part of uh White tail seven point two um uh next month and uh is it next month or December? I was like yeah, it's next month. And hopefully at least some of the the tangible improvements beyond the Elasticsearch and OpenSearch as well. But yeah. Can't say for sure on that, but yeah, this is definitely a longer term thing for uh getting all of the uh technical debt sorted.

Speaker 1: Great, thanks Matt. Is Elasticsearch or OpenSearch a requirement to get full text search using Wagtail Search?

Speaker 2: Um no, that's uh so the uh database backends that are they all support uh f full text search. Um there are Currently some limitations, I think the one I mentioned being uh it doesn't support searching on individual fields, but um yeah, the the the the s the searching the the underlying functions in the database backends are that they're sort of on a par with uh what we have for in Elasticsearch so that you do get that. with just the database without Elasticsearch or OpenSearch.

Speaker 1: Next question. Any tips or resources for creating filters or facets for a Wagtail search UI? Will Django model search help with that, maybe with a combination with Django filters?

Speaker 2: Right. Oh no, that's uh an area I've not really looked into. We there are there is uh faceting um support in i in in White Tail Search that has now made it into Django model search. It's not really something I've looked deeply into, but uh yeah, I th I think that is uh that that that should work nicely in conjunction with uh Django filters. Uh ultimately that what what you select in Django filters will be translated into filter expressions. So as long as th those are not too complex, then that should be the supported um with with all the backends.

Speaker 1: Thanks, Matthew. We've just got time for a couple more questions. Will Django model search support database backends? Uh

Speaker 2: yeah, so uh yeah, they the plan is very much to ensure that everything is uh backwards compatible uh with how things worked. Um in WhiteL search. And yeah, so the the the database backends will uh they're they're here to stay. Um so we uh We've reintroduced the MySQL backend which was temporarily taken out, but that's all back in now. So yeah, everything that uh has previously worked will continue to.

Speaker 1: Great. And just one more question before we have to wrap up this session. Are there any plans to support multiple index versions of the same page to deal with different setups for the admin and front end or multiple languages?

Speaker 2: Uh yeah, I think we we do want to uh to handle that sort of thing. Um I know um uh Baptiste who who uh has uh recently joined Torchbox uh and is uh a l a long time uh Django uh Django dev uh on the on the on the uh on Django Core is is keen to look into that side of things. So yeah, I think we we definitely want to we w I don't n know exactly how that's going to look yet, but uh I think Baptiste has some ideas and yeah, I'm sure we can make that work.

Speaker 1: Great. Thank you, Matthew. Uh um that was a really fascinating look at the history and the future uh of Wagtail Search.