A Pythonic Full-Text Search - Paolo Melchiorre

Hello everyone, this is my fifth pre-recorded talk, but I'm very happy to be here with you at DjangoCon Europe 2020. I want to thank all the organizers for making this online edition possible. And now let's start with the talk If you're asking yourself what is a Pythonic filter search, I'll show you an example. This is the search function in the Jago website. How many of you have searched for information on it? I did it a lot of time. The search function is based only on Postgres and Django itself. And I was the one who built it.

So the next question is, who am I? I'm Paolo Mecchiore and I'm the CTO of 20Telb, a Pythonic software company based in Rome, for which I work remotely I'm a software engineer and a longtime Python backend developer. After using Django for a few years, I became a contributor to the project. And now I want to try to explain a bit more about the title of this talk, a Pythonic Full Tech Search. I think you can read the definition of Pythonic

by entering import D 's the Python interpreter These are only the first principle of the Xenopython. The most important for me is the third one and I think it's also the most difficult to follow Full text search refers to the technique for searching a computer store document in a full text database. There are a lot of search engines that already provide a full text search The most popular search engine library is Apache

Lucine, an open source software written in Java. Based on Lucene, there are two very popular search engines that I used in the past. Solar, which is part of the Apache Software Foundation, and Elasticsearch a product of the Elastic Company. The last big project where he used one of them is Docs Italia Docs Italia is an Italian government website where to find public documents. I worked in this project to improve the search function.

Hand the Hood, Docs Italia, is a fork of the open source project Read the Docs. So, as the original project is a Django-based platform and it requires a lot of Python packages. to access the elastic search instance asking for results. The search function is working very well, but we can consider this a simple solution. We can say various things about external engines. On the good side, they are very popular, they have a lot of features, and you can find a lot of online

resources about them. On the bed side, you always need a driver to use them from Django. You have to use their specific query language, and it's common to have synchronization problems. Oh no, this is embarrassing. Jerk aside, this is something similar to what happened in an e-commerce website when you find a product in the search result. but it's not available anymore when you click on it. This happened because search results are fit from the search engine which is not already synchronized with database.

So why don't we search directly on the database? Maybe a big one and with a lasting memory. Postgres is a very popular and long-lived database. It added full text search years ago with specific data types and special indexes and since then Many useful new features have been added every year until the last version. The main concept of Full Text search in Postgres is documents.

A document is the unit of searching in a full text system. For example, a magazine article or the union of its parts. The title, the abstract, the body text, and so on. Implementing a web search function directly on the database can be a very low level task. To do this, we can use a web framework, maybe one of the best, and you already know which one Django had a full text search a few years ago

in the Django Country Postgres module with specific fields. Expression and function. Since then, many new useful features have been added every year until the latest version The Django documentation defines a document-based search as a full-text search with advantages feature. Weighting, categorization, highlighting, multiple language, and we can implement all of them with Django itself

But to better understand how the full text search in Django works, we're going to see how to perform some queries from the basic one to the more complex. We can perform well also with big amount of data We can use the blog models as defined in the Django documentation. Here we have three classes with few fields, a blog. an outer and an entry connected with both of them with a foreign key

We can perform basic queries using field lookup. For example, we can search an author using part of his name. Performing a case insensitive query, we can have more results. To find words with accented letters, we have to activate the unaccent extension. After that, we can search an author also if we don't know exactly all the accented letters on

its name. Toever is us Also, if you don't remember well the author's name, we have to activate the Trigram extension. Searching for an author, we can have results with similar but not identical name. To use the above feature, we have to add the Postgres module in the Installed Apps in the settings. After that

, we will be also able to perform a full text search on a single field. For example, we can search for a word in the plural form and have results in the singular form. To search a text in more than one field, we have to use the search vector function We can define our documents as the union of the entry body and the related blog name. After that we can search for a word and have more accurate results.

To search using more complex text, we have to use the search query expression. We can also use common search syntax directly in the query text using the web search type After that, for example, we can search for two words at the same time, having potentially more results. To perform a full text search in a specific language, we have to use the search config expression We can specify the language in both the document and the query. After that, we can have more precise results than before

To list relevant results first, we have to use search rank function. Based on the query text and the document, Django will calculate a rank. We can order and filter our result using this rank. To perform a fine-grained full text search, we have to use the search vector weights attribute. For example, we can decide that words in the headline are more relevant than in the body text. After that we will see a new rank in our result

also performing the same search than before. To highlight results, we have to use the search atline function. We have to specify the highlighted field and after that we will see results with tags on it To speed up the search and simplify the queries, we have to use the search vector field. We have to manually update our search before running a query

After that we will have the same results as before, but way more quickly I started using the full text search in Django 1. 10 and I searched frequently the Django documentation for information on it. After a while, I started asking myself how was implemented the search function in the Django web side itself I noticed that the search was performed only on English and in some cases there was tag-in results. I studied the Django with

SAT source code and I found out that the documentation was generated with Sphinx. All the data was stored on Postgres, but searches was performed in an external search engine. So I propose to fix that on the Django developer mailing list. A lot of developers share different opinions about the proposed updates. The dApps were the amount of work to be done, the equivalence of search feature, and the increase of workload on the database. The safetings were less maintenance, a lighter setup, and the exclusive use of Django

on its own website. So I organized a Django sprint during the EuroPython 2017 in Rimini and some developers joined me to work on the search updates. In a spring day we created a draft of a PostgreLe-based full text search But we also spent a lot of time trying to set up the Django website on our PC to replicate the Elasticsearch function. In the months following, I wrote an official pull request with a complete version of the full

text search for Django. I received a lot of suggestions from other developers and maintainers, and after a lot of comments and comments, they emerged my pull request. So today the Django website for Ultra Search is multilingual is based on Postgres and return clear results is a low maintenance solution and easier to set up than before As I already said, new full text search features are released every day, every year, sorry, in both Postgres and Django.

And I hope to add them on the Django website search, like spelling support, search suggestion, highlighted results. web search syntax and search statistics and I'm working on it I want to share some useful tips with you. Read the documentation in the Django website. It's full of information. about all the full text search features. Read the details about full text search in the Postgres website. It helps you to understand how things work at

lower level. Read the source code of bot project in GitHub because there is something you can learn only from the code. Search for questions on Stack Overflow, but try to answer them and you'll go to the next level in using full text search with Django and Postgres Last but not least, you can also study this presentation because it is released with a Creative Commons license I hope I have been able to show you how it's possible to develop a more complete full text search using less software in the stack. Doing more with

less is the motto of 20Tub. It's our version of Pythonic. You can find more about our open source project and our Platonic worlds using this context. To find out more about my world with Python and Django, use All My Contexts. Using this QR code, you can download this presentation on my website. Thanks again for having me. Enjoy the next talk in the conference.

Thank you for watching my recorded presentation. Despite it being the first I ever recorded, during the conference I'm still online and available to answer your questions about full-text search in Django with Postgres. So see you