Pair Programming after the Pandemic and Beyond
Published July 11, 2024
This video features Gerald Carlton and Tobias McNulty at DjangoCon US 2026 in Chicago, Illinois, USA.
Searching for human names in large datasets is harder than it looks. A user searches for "Smith, John," but the database contains "Smyth, Jon" or "Smithe, Johnny." Standard LIKE queries are too strict, full-text search doesn't handle name nuances well, and unindexed fuzzy matching won't scale.
This talk covers how we solved these problems for RecordsDB, a Django app that searches over 50 million North Carolina court records for legal aid organizations and public defenders.
Talk Structure
The session is structured as a technical walkthrough using a Jupyter notebook and a sample Django app. I’ll demonstrate the pitfalls of name searching and how to implement performant solutions. The notebook and source code will be available at the start of the talk so you can follow along.
We’ll cover how to use PostgreSQL phonetic algorithms (Soundex, Daitch-Mokotoff), fuzzy string matching (Levenshtein), and specific indexing strategies to keep searches fast across 50M+ records.
Problem Space
The Challenges: Why name search fails (spelling variations, typos, nicknames).
Scale: Why techniques that work for 10k records break at 1M+.
Case Study: Lessons from a 54-million-record production database.
PostgreSQL Techniques
Phonetic Matching: Using Daitch-Mokotoff to find similar-sounding names with different spellings.
Fuzzy Matching: Using Levenshtein distance for typo tolerance.
Indexing for Performance:
GIN indexes for phonetic matching.
B-tree with text_pattern_ops for prefix searches.
Performance gains from COLLATE "C".
Trade-offs between index size and query speed.
Django Implementation
ORM Integration: Writing custom QuerySet methods and wrapping Postgres functions with RawSQL or custom Func expressions.
Custom Indexes: Defining specific Postgres indexes in Django models.
Fast Type-ahead: Building search-as-you-type interfaces that stay responsive at scale.
Benchmarks: Comparing query times across different search strategies.
Lessons Learned
UX: When to use fuzzy matching versus when it creates too much noise.
Maintenance: Managing large indexes in production.
Accuracy: Balancing precision and recall in legal and public service contexts.
This talk was presented by Tobias McNulty and Gerald Carlton at DjangoCon US 2026 in Chicago, Illinois, USA
https://2026.djangocon.us/talks/search-as-you-type-for-54-million-names-postgresql-django-for-fuzzy-name-matching-at-scale/
Automatically transcribed, so expect mistakes in names and technical terms.
Speaker 1: Thanks everyone. We're really excited to be here today in Chicago again this year. This is a semi-interactive talk, if you would like to follow along. There's a Remo Notebook, uh Django project. and uh management command that you could run to generate some fake data at the URL or the QR code on the screen here. I'll leave that up there for just a minute. I'm Tobias. I'm joined today by my co-presenter, Gerald Carleton. We're both employee owners at uh Cactus Group. Which is a web development company based in North Carolina. We are proud sponsors of DjangoCon and the Django Software Foundation. And yes.
Speaker 1: And uh we had a lot of help um with this talk uh from our colleagues Simon and uh Colin who helped with the initial Implementation for the customer and these slides in the demo project as well. But before we dive into this project, we want to take a step back and talk about criminal record expungement And for this walkthrough, I'll hand it over to Gerald.
Speaker 2: Thanks, Tobias. First off, I want to say this is my first time being at DjangoCon. Uh so far it's been great. Um this is also my first time giving a talk, so Double. Um but I keep hearing that this is the place to do a talk if it's your first time doing a talk. So um I'm just excited to be here So before we jump in, I want to give a bit of context. So this talk was inspired by some work that we did for one of our clients who operates in the legal aid space. So they came to us with an issue that we were able to help them solve. And what I'll walk you through is what their specific specific workflow was like um
Speaker 2: and where it became uh a hurdle for them in their day-to-day work. Uh so Criminal record expungement. So this is an area of legal aid where nonprofit organizations , public defenders, private attorneys help members of the community. uh to expunge uh clear records from uh clear eligible charges from that record such as dismissed cases or cases that they were found not guilty of One thing that some people may not realize is that a not guilty or a dismissed charge will still show on your record. So for someone trying to apply for housing or apply for a job, an expansion can be life-changing.
Speaker 2: Uh under the hood, the process of preparing these petitions for expungements is painfully manual. Uh attorneys and paralegals must track down potentially hundreds sub-records spanning decades. review every single charge, then manually enter everything into a strictly formatted court form, which could take a while. All right, so here's an overview of what that process looks like. Everything starts with the intake. So you have a client who will walk into an office looking for expungement. Uh staff members will then collect uh That person's name, every name that they've been that they've gone by, their aliases, uh dates of birth. Um And now the all the counties that they've lived in as well.
Speaker 2: So nothing gets missed in the record search that follows. Record search. Uh so this once all the all the data is collected, it's time to do a search. So this will be what we will be focused on today. Staff members then have to search court records and law enforcement databases for every record that is tied to that particular client. And this is where things can get tricky because names are sometimes typed or written inconsistently. There could be misspellings, nicknames, or abbreviations. uh such as someone who goes by Matt instead of Matthew. So a simple exact search text will miss those records that exist in the database.
Speaker 2: Once every record is found for a client, each charge gets checked against a state statute to see if it if it qualifies for expungement. And then from there, every eligible charge is then transcribed into a strictly formatted court petition by hand, which is another manual error-prone process that could take time Finally, uh the petition is completed, it's filed with the court, uh, and if it's granted, those records are cleared. Uh, but getting here can take weeks of manual uh work for a particular single client. So when we first took on this project, we assumed that the main challenges would be uh things like
Speaker 2: uh Mapping the uh map eligibility mapping, applying those North Carolina uh statutes, cleanly uh inserting them into standardized court forms. But that changed when we did a dedicated discovery workshop with the client. So during the workshop we realized that finding their proper records quickly was the true bottleneck preventing eligible members from uh eligible community members from getting the help that they needed. Uh thorough record check thorough record checks are critical uh because Conviction conviction expunctions in North Carolina or uh a one-shot opportunity. So once a court grants a conviction expunction, the client can't use that statue again.
Speaker 2: So this means any omitted convictions will stay on their record if it's not caught in that first filing. So running a comprehensive statewide search is essential to make sure that uh none of those eligible charges are left off before the initial filing. All right, so uh here's the core problem. The state transitioned uh Away from its legacy system to a more modern web portal. So you might think more modern means faster, better. That's not always the case. And this new upgrade system severely restricted uh
Speaker 2: how uh staff members could search uh for records. Uh It kind of really slowed down the process from what they were using before. So we have a video here, and I'm not going to talk while it plays because I kind of want you guys to be in their shoes to see how painful it can be to sort of do this for multiple clients um just to see uh get a taste of like what it's like to sort of spend this much time trying to find all the records And this would be searching for uh you can imagine a person comes to an office and they give their names, or now the staff memory is trying to search for all the records that might be tied to this particular name.
Speaker 2: Hey. As you can see, that took quite a while. Simple search turk uh search took 27 seconds, um, and it returned 200 cases. There are other instances where searches might have taken maybe 10 to 15 seconds, but not all searches were this long. It kind of just depends on what the name was that's being searched. Um Yeah, okay, and yeah, also uh yeah in our so basically in our discovery we discovered that The time that it took for staff members to pull these records for one client grew from three hours up to 10 hours with the new system.
Speaker 2: So you can see how this was a big pain point for them. Uh to make matters worse, uh while the system uh it does claim to offer fuzzy search. The underlying logic was poorly implemented, where it can balloon a tight six item return to over 200 useless matches, which a lot of those matches included false positives. um rather than giving the attorneys the results that they really need. So why is finding everyone so hard? In our case, we're dealing with a database of over 550 million North Carolina court records, and searching by that name in a database is not easy.
Speaker 2: We're not just looking for a single record, we're looking for every record that belongs to uh that particular that for that single person. Secondly, think about the lifespan of a criminal record. You can imagine how frequent typos could be. A name could get typed in on the side of the road by an arresting officer. It gets transcribed by a handwritten form by a Rush court clerk, or it's provided by a client who uses nicknames, hyphenic names, or spells their names differently across different legal documents. And on top of that, uh collisions are everywhere. Thousands of people share the same name, first and last name. Uh and data birth is one of those fields that uh could could disambiguate two people, uh, but sometimes even that's missing in core
Speaker 2: data. Uh so we realized that if we wanted to build a tool that was truly uh would truly serve legal aid organizations, public defenders, private attorneys. To solve that multimillion FUYU search record problem, we wanted to use tools of our choice, which were Django and Postgres SQL. And to talk more about that, I'll pass it over to you, Tobias.
Speaker 1: Thanks, Joe. Um So yeah, uh as uh evidenced in the demo that uh Gerald gave and uh just in general, I think there's a lot of assumptions that we can make about human names. And there's an article linked here which was helpful for us as just to sort of set the stage and I recommend reading it through when you have a moment. And I don't won't go through all of them here, but I thought it would be fun to highlight uh just a few of them. Um one is that uh people have exactly one uh canonical name Um when in reality, you know, they may have uh different aliases
Speaker 1: or nicknames uh or spellings even that they use regularly on legal documents. Um you know, depending on the day or the week, and they'll alternate back and forth between the two of those. Um another assumption is that people's names uh you know don't change when in fact uh you know over a person's lifetime uh through marriage or uh divorce or simply deciding to change one's name, uh a single person might not have the same name in all of the records in the database. Another uh example is that uh people have different first name and last name or Uh perhaps and this one may hit close to home for the Django user model that they have uh you know exactly one first name and last name and that's it at all.
Speaker 1: Um And where, you know, because in different cultures and countries, uh, even here in the US, uh, names often don't fit into those neat first name and last name boxes. Another assumption we might make is that uh names are written in uh ASCII. Um we uh For better or worse, did not have to deal with this because in every single of the fifty-four million records in the North Carolina uh state database, uh there's not a single non-ASCII character. It's true. Um lastly, uh and this was another one that hit close to home for our project is that
Speaker 1: you know two different people entering the same name will enter it the same way uh you know into a terminal or web form. when in fact as Gerald talked about there are uh you know the frequency of of typos or just you know hearing you know one name or spelling in their head and typing that down the spelling could be completely different from the person's name And we found yeah, ample evidence of that in our uh database. Um And f of course, we're not going to demo the the actual um you know production data of 54 million people here, um, but we did take time to put together a uh sample data set that you know roughly uh has the same shape of the the database that we are working with.
Speaker 1: Um For this, we found a dataset on Kaggle that had 1 million last names and 600,000 first names. We randomly weighted those according to the frequency of those names in the United States, you know, combined them randomly and assigned a random birth date. And each name, like Gerald talked about, is not a like a nice den uh normalized database with a person table and then a court record table. It's just one big table with all the names, which might be different uh in different places. So each name uh might appear anywhere from one to eighty times in the in the sample database with some typos injected just to make this uh more interesting.
Speaker 1: And so this is the this is the data model in the sample project here that again loosely follows the model of the data that we received from the state. There's a first name, last name, a middle name, and a date of birth. Just for the demo project, we we added a person ID to that, which We didn't have, or at least we didn't have reliably in our data. There actually was one, but it wasn't always consistent, so we couldn't rely on it for tying different records together. And let's see. So yeah, let's dive into what that looks like. And like I said, this is a um
Speaker 1: These slides are a Maremo notebook. We got to learn how that worked for this project. So there we go. Um so yeah, we can dive into that here. Um so this is running on my laptop. Um We're in a notebook connected to this sample dataset that we generated. Let me see if I can make that a little bit bigger Is that semi-readable uh for folks? Yes? Okay, thanks. Um So we have our court record model and we're gonna filter for
Speaker 1: uh you know exact match case insensitive on Sonia Mercias. And we get back 14 rows for that search. You can see they all have the same person ID and the same date of birth. So this is These are all corresponding to the same person. If we add the date of birth, we can see again that we get exactly 14 rows back. However, if we then search by that person ID field that we added, we can see that we missed a few rows in the initial search. We now get back 18. rows from that and you can see some of the typos that are script injected here.
Speaker 1: Sony, uh Sony , Sona. So there's yeah, 18 in total that we're trying to find. And um you know a naive approach to this might be to inject some wildcards into the search or give the users the ability to to put wild cards in there and you can you can almost get to it uh by searching for you know sun and reci and then you get back 17 of the 18 rows that we were looking for in that way. But there's got to be a better way. So uh this uh this talk could also be called why is Postgres Awesome? Uh
Speaker 1: we relied heavily for this uh implementation on the fuzzy stir match extension in Postgres. And in particular, we use the Deitch Makotov algorithm, but there's a Soundex algorithm, which they 're roughly equivalent, just do the same thing in different ways. So let's walk through an example of what those are. And so yes, just to start out with sound X, um this assigns a alphanumeric code to a word. Uh so you can see here if we calculate the sound x code of Sonya and Sania and Sania and so and all of these different spellings. Those all have the same code here, S500. So if you uh create an index on that and then match on that,
Speaker 1: you can find all of the you know rows in the table that correspond to that uh sound. According to this algorithm. And so yeah, if we update our earlier query to search on the sound decks of the first name and last name and date of birth, we get back the 18 rows that we were looking for uh without the person ID um and I can see yeah all of the same typos Sona and Sony et cetera in there The issue with this phonetic algorithm though is that it does introduce a lot of false positives. We don't know the underlying implementation of the court system that we shared the video of. Uh but we can assume it it might implement you know something like this.
Speaker 1: Um and uh sort of stops there. So you can see if we search for Ricius. On the last name, we also get back the last name is Rakowski and Rosus and Rohus and uh lots and lots of different false positives. Um and just to briefly demo the Deich Makotov algorithm, this works in pretty much the same way, except uh It assigns an array of sounds, sound codes to each word, uh, and then you can use the overlap operator in Postgres to see if those codes overlap, and then according to this algorithm that you know they're determined to to maybe have the same sound. So here
Speaker 1: we're searching for Michael and Mikael. Michael has you know both codes 658000 and 648000. Whereas Mikhail has uh just the code 658000, and those do overlap, so they're determined to have the same sound. So, how um how do we deal with all the false positives that we saw that Gerald mentioned and that we saw in the results here? One function that you can use to do that is also in the fuzzy stir match. Module in Postgres is the Levenstein distance. And what this does is it calculates the number of character changes that are required to move from one
Speaker 1: word to another So in this example, going from Michael to Mikhail, we need to substitute the C for a K, and then we need to remove the H. Uh so that's determined to have a Lebenstein distance of two uh according to this algorithm. And then if we take one of the false positives that we saw from the earlier results, the Sandex code, you know, RISCS and ROSUS both have the same soundx code of R220. But the number of character changes or the edit distance between those two words is three. So they do look more different and they have a higher edit distance. So that's one
Speaker 1: mechanism you could use to tell how far apart these words are Um when you're you doing this, that's pretty inefficient to do on a large database. There's not really a way to to create an index on that because it does need to you know compare a certain word to you know every row in the table. But there is a slightly more efficient function that you can use for that which is Levinstein less equal. And that, you know, instead of calculating the actual distance between two words, you can give it a stop point, you know, at which point it won't calculate the actual distance, it will just turn a value, return a value that's um greater than the value that you passed in. So yeah, how do we use this in Django? Django does not have built-in support for
Speaker 1: these functions, but you can easily uh create them. uh just by creating uh subclasses of the func uh class uh and uh you know telling it what the uh output field is and what um the function is it should be called in Postgres. And to see what that you know looks like in practice, you would add a annotation with the soundx code of the last name in this example. And then you could filter on, add a filter to find all rows where the sounddex code is equal to the sounddex code of the word of the name Smith. And
Speaker 1: um Django does have uh built-in support for trigram matching, um which is another um yeah module folks may be familiar with. We uh tested this um briefly and it it didn't end up working for us, but thought it would be you know worth mentioning nonetheless. The way this uh module works is it uh it splits a word into slices of three characters uh in in Postgres 's implementation it will pad the f the first uh letter, you know, first with two spaces and then one and so forth. Uh you know, for so for uh you know Geraldine might be split into G E R, ERA, REL, et cetera. And if you you can use a similarity function in Postgres to calculate
Speaker 1: the the similarity of those two words. So Geraldine and Gerald have a similarity score of 0. 53. Uh Morgan and Morrison have a score of 0. 23. Two completely different words have a score of zero, and of course the same word will have a score of one. And these are the trigram similarity and trigram distance classes are how you would use those in Django. Yeah, we found that they were particularly sensitive to uh short names because just when you have fewer characters to work with, um there's you know just over only so much uh um that you know the three three characters can be determined between the two words. Uh and there's a few different in
Speaker 1: indis indexes that you can create uh for um uh trigger matching. Um the one that we opted for um was pretty slow to generate like 15 to 20 minutes for 54 million records on a pretty substantial machine. And um it can also only uh well uh it's it's nice because you can use order by on it and just uh you know take the top. And of those, you don't have to calculate the similarity of everything in the table. But the problem is you can do that only on one column. So in our case, we really wanted to do that on both the first name and last name. So that that Did not work very well for us on this project.
Speaker 1: So yeah, how what what was the solution that we ended up on and how did that work for us? This uh is a lot of code on the screen. Uh um we'll blow it up a little bit more. Um Just uh yeah, to reiterate what we saw earlier, first we need to uh annotate uh the first name and the last name, uh or excuse me, annotate the query set with the DeitchMakatuff codes for the first name and the last name. And then do the same thing for the edit distance. Just anecdotally, we found that an edit distance of two seemed to, you know, for the most part included the true typos and eliminated the majority of the false positives. So that's what we settled on.
Speaker 1: And then you can filter on that here. So and then this is a live cell, so I can run it and we get back. the uh eighteen rows for Sonia Rosia. So that's what that would look like in Django. Of course, uh none of this would uh be possible or fast without uh good indexes. Um We um we settled on a uh a case insensitive starts with matching. Um for um again just enabling the operators to find records as quickly as possible.
Speaker 1: And there's a uh a special uh ops class for the index and Postgres, this text pattern ops, that gives you uh really fast uh index matching on two columns at once. You know, in our case We may even add a third in here, but for the demo we just have first name and last name. And Then yeah, second , we can create a separate indexes for the Sandex code, the Deitch Makatoff code on the first name and last name as well for the phonetic matching that we talked about. So any uh talk wouldn't be anxiety inducing enough without a live demo.
Speaker 1: So uh Here we go. And since it's Gerald's first talk, I thought we should uh let him do the the demo. Cool.
Speaker 2: All right. Uh is that big enough for y'all or could could um how'd it make it bigger? Okay, cool. Very um I just thought of this idea, but I thought it would be cool to Nope. Just show the John Doe example that we showed earlier. So that was John Doe. So the video we saw earlier, you saw how long it took. But with this new prefix, you can see it took 60 milliseconds. So big improvement.
Speaker 2: But to showcase some of the other search algorithms that we also work with, I'm gonna use one of the cases that uh Tobias was showcasing. Yes. So that's just the last name. And as you notice, as you type, it's all it's searching. So as soon as soon as characters are typed, it's pulling in results. So if we just go for SON you can see it pulled in seventeen results in sixteen in fifteen milliseconds Let's go Son, you get Sonya, Sonny, and this Ed Sonya. So this is the full name
Speaker 2: on exact prefix. So it pulls in the names exactly as they're typed into the name fields. You can see we got 14 names returned, which matches what uh Tobias was showcasing in his Jupyter Notebook. But we might be missing some. So if we apply the Dutch Makatoff, which is again the sounds-like uh search algorithm. So now we're up to 200. And a lot of these are false positives, but a lot of these names will sound like the name that we've entered. So let's see if I can and find uh well I'll use the other example for that.
Speaker 2: Um so these are all sound like uh things that might sound like Sonya reas But the leveraging will reduce all that noise, so it'll apply a scoring to all the names that were returned to closely match more what was entered in the first and last name field. So if we apply that search, narrow down to 18. So you can see we gained some by applying this search algorithm as well. So now we're up to a total of 18 record return. As you can see here, it kind of shows you what was added. So with the Dots Makatoff added, you can see that It picked up this name here, Soney Ries, Sona Ries.
Speaker 2: Uh Song your Ries, which built slightly different. So you can see how it uh will open the net to make sure that you can include names that you might be looking for and then also uh close in on more specific names that you might be searching for. Uh can also demo another example. Let's uh clear this We'll go with Celine, I think. I mean already down with the E S E
Speaker 2: There you go. So for this person here, again, exact prefix. We got 15 results returned matching exactly what I entered. Also if you want to cast the wider net to get more sounds like results, I apply TouchMuktoff. And Wait. Of course. Uh I don't know what happened here. What happened?
Speaker 2: Oh you restarting it. Yeah.
Speaker 1: Yeah, and this is just running on my laptop, it's not on a big fancy server somewhere. Now we do it fly. Yeah.
Speaker 2: Is he hosting?
Speaker 1: We found a bug. I don't know. Too many names or something.
Speaker 2: Yeah.
Speaker 1: Try it in that.
Speaker 2: Yeah. Okay. Yeah. All right, so ignore that. But um so as you can see is if we apply it to Levenstein, um initially the Dutch Makatoff would uh before I got in front of people, it showed 200 results A lot of those results were false positives. So for example, you might have seen like the first name might have been spelled. C O L I N, which could be Celine or it could be Colin, however you want we want to say it. Um you have examples like that as like those are like false positives that are uh in the set that the attorneys might not want. So applying this levishing sort of weeze those out. So we would go from 200 to 34 results.
Speaker 2: And in a case like this, let's say for instance you you do know the birth date of the person who's looking for expungements. Uh you will select their birth date. Uh that will whittle it down to a set of uh results that you are looking for. So uh best to have a summary.
Speaker 1: Yeah. Yeah, yeah. Um and um and just to emphasize, this it's not like a user interface that we would recommend giving to a customer. This is more for uh you know folks who are curious about these things to play around with them and understand how they work and how you can layer them to a achieve a you know result that that makes sense for the data sets um that you have. In our case um we always started with exact prefix matching so there wasn't like a any other option to start with And then we added a button that would limit it to the similar names based on the combination of the Deutsche Moketoff and the the Levenstein.
Speaker 1: So we don't you know require the user to understand um you know everything about these algorithms in order to use the the application. Um we have maybe five minutes for questions. Okay. Um Yeah, really briefly, just to summarize, um we're not experts in these algorithms. Um we're just sort of sharing a story of what worked for us. I had to look up how to pronounce one of them today before the talk. But the yeah, the a few wins that we found in the working on this project were the text pattern ops. uh indexes and Postgres for really fast I starts with matching in Django.
Speaker 1: One thing we didn't mention earlier is that if you can avoid sorting or delay sorting until as late as possible. Postgres of course will return the you know top N records much faster if it doesn't have to sort everything below that. So that's another strategy that we used. And then like uh yeah, Gerald demoed, you can use Soundex or Deitch Deitch Makatoff to broaden the search and then the Levin Steinlust equal to uh remove some of those false positives. One hiccup that we hit is just making sure that the indexes that you create match exactly the uh you know function calls that you're using. And again, um Postgres is really amazing that we could take this 20-second query uh that the user was experiencing and then turn that into a
Speaker 1: a live search um that comes back in you know under 10 milliseconds um felt like a pretty big win. So we were very proud of that. And yeah, that's all we had. There's a few links here to the demo site. This is deployed to a little 16GB DigitalOcean instance if you want to try to break it. Or figure out why it broke uh just now. And uh the uh git repo and there's a companion blog post up in the cactus blog as well. So thanks very much. And I'm not sure we have a few minutes for questions if folks have any. Seven minutes, okay.
Speaker 1: Do you want to um Okay, cool.
Speaker 3: Uh one of my problems when I've done ETL on names is uh They get entered in the wrong field, like John Roberts comes in as Robert Johns and J. Edgar Hoover comes in as just Edgar Space Hoover. Uh and you for your client, did you take these into account?
Speaker 1: Um Yeah, I I don't know if we observed that. Um particular issue or not. We definitely did observe some of the similar inconsistencies. Nicknames were one thing that we didn't touch on specifically in this talk that we did uh have to deal with in the project. And We decided in the end not to try to solve that problem in a magical way, but instead to just enable the user with a tool that was So fast, and they're you know, they're fairly their attorneys, they're fairly expert users, but enable them with a tool that was so fast that they could find records like that.
Speaker 1: So if if a um you know a full name was entered in uh the field you know one field or they were swapped like you're talking about that they um could um you know test that out or quickly find it. So
Speaker 4: Thank you for sharing your findings. I'm next. I was curious, maybe I didn't follow. Oh I tried to To pay close attention, but maybe I didn't follow. What was the correlation between having that index and having to use this um method that just matches the first few letters at the beginning of the word, right? Yeah, there seems to be this correlation between applying the index and then you have to use this starting widge.
Speaker 1: Yeah. And yeah, we didn't starts with matching works pretty well for names, so we um we didn't explore too much like uh you know I contains match. Um But um I don't I'm not aware of the underlying details, but we um we definitely confirm that it is slow or does not uh use the correct index if you don't use this ops class, uh which was yeah, I had not to my m knowledge, I had not come across this before in a Postgres or Django project. So yeah, that was one win for us. Thanks. Um
Speaker 5: just curious, s uh I know with with names, so not every edit distance is the same. So let's say like John D. Smith versus John Smith is edit distance one. Versus and that probably isn't the same person because it's a very common name. But then like I'm Paul Zrotsky, Z-K-I. If you see us misspelling SKI, it's probably the same person kind of thing. So I know there's other uh I think Splink is one for probabilistic matching for for that kind of category of problems. Did you encounter that or uh Would that be like a extension to this? I'm selfishly curious too because it'd be cool to see it in Django.
Speaker 1: What what was the um
Speaker 5: uh probabilistic matching and uh Splink I believe is uh one library. I don't know if it has an integration with post I forget.
Speaker 1: Okay.
Speaker 5: But yeah.
Speaker 1: Yeah, we I don't think we came across that on this project. So good to know.
Speaker 6: I'm I'm hoping I'm getting this right. The uh for the l Le Levenstein less equal, you mentioned that you had some limitations with that approach only because the there was one field that you could use order by on. Is that Is that right?
Speaker 1: Um sorry, that was with the trigram uh matching. Okay.
Speaker 6: Wondering if if you considered maintaining like a shadow full name field, you know, with an overridden save method or you know something of that nature, you know, in a normal form, maybe separated by a hyphen or some other character that's not normally in a name, and then using that one field as a way to sort of speed up the index creation. I mean did you play around with
Speaker 1: the With trigrams, you're saying to combine the name together and then uh use that. Yeah, yeah, that's a good idea. We didn't we did not explore that in the project. So thanks.
Speaker 7: Thank you very much. Mazing Gaffel from New Zealand. Uh I have nothing to do with this conference. I'm public health physician. But usually I attend this odd conference to find ideas and to hunt for ideas. And today I got a lot of ideas really from this conference. When you do the fuzzy match, you will get a score with it. And the score will decide, is it 50% matched, is it 90% matched, is it 80% matched? What's the cutoff point which you used in your Software.
Speaker 1: Yeah, we um we did think about trying to score the um results in some way and um you know be I don't know, more complete or or um not sure what the right word is, but yeah, to really try to come up with the exact right result that the, you know, take some guesses and come up with the right result that we think the user is looking for In the end, just because of the expert nature of the the users on this particular project, we wanted to keep keep it as simple as possible and enable them to sort of try these different combinations on their own. But that's another good idea, yeah, good a great idea to try to create a score out of these and and um you know, depending on the confidence, you know, build a result
Speaker 1: set um based on that. Thanks.
Speaker 8: Okay, that's all the time we have for questions. Please give a hand since Bayas and Joe. A
August 24 - 28, 2026
Chicago, Illinois, USA
Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026