Lightning Talks Day 1

This video features Ernist W. Durbin III, Filipe Ximenes, Flávio Juvenal, Griffith Rees, Katie McLaughlin, Raphael Michel, Sarah Heim, Simon Willison and Tim Allen at DjangoCon US 2018 in San Diego, California, USA.

Lightning Talks Day 1
0:40:45
Published November 8, 2018
732 views

DjangoCon US 2018 - Lightning Talks Day 1

00:00 - Sarah Heim
05:26 - Tim Allen
09:21 - Raphel Michel
13:50 - Katie McLaughlin
15:51 - Simon Willison
21:02 - Griffith Rees
26:18 - Ernist W. Durbin III
30:08 - Flavio Juvenal
35:24 - Filipe Ximens

Follow DjangCon US 👇
https://twitter.com/djangocon

Follow DEFNA 👇
https://twitter.com/defnado
https://www.defna.org/

Summary

The presenters cover a wide range of practical and personal topics: finding recovery and support, automating application screenshots with Selenium, headless Chrome and pytest, building a GeoJSON-based map and API of cryptid sightings, and creating collaborative artwork with Django. They also explain how the Python Software Foundation supports the community, how fuzzy matching and geocoding can deduplicate messy records, and how design patterns emerge from observing how people naturally use systems. Across the talks, they argue for asking for help, using familiar tools creatively, designing around real human behavior, and contributing back to open-source communities.

Key takeaways

  • Changing one difficult thing can transform a person’s life, and asking for help is often essential to making that change.
  • Selenium, headless Chrome, pytest fixtures, and parameters can automate screenshots across languages, themes, resolutions, and application states.
  • GeoJSON, SQLite, Travis CI, and Datasette can turn crowdsourced cryptid range data into a queryable geographic API.
  • Fuzzy string comparison, geocoding, and record-linkage tools can identify duplicate records when real-world data lacks reliable unique identifiers.
  • Design patterns should be treated as communication tools that emerge from observing useful solutions, not as status symbols or rigid recipes.
  • The Python and Django communities depend on people proposing ideas, contributing to projects, and supporting events and infrastructure.

Summarised automatically from the transcript.

Transcript

6,777 words · auto-generated Show

Automatically transcribed, so expect mistakes in names and technical terms.

0:21

Speaker 1: All right. I get I'm I'm coming up here as is I feel like I'm Miss Schwag here. I've got my uh got my new little Cell phone holder and water and and shades. So I'm like stoked on all the free stuff. So I don't know if a lot of you guys here are for the free stuff, but I'm here to uh Uh so next week I'm going home. I'm v I I live here so I'm kind of on vacation all the time. But next week I'm going home to Minnesota And um yeah, school, right? Anybody who's here from Minnesota. Uh so uh next week going home, my cousin is uh in stand-up and I want to do that with him. So this is my first kind of public speaking, a little practice. But meanwhile I'm a programmer, so I feel like I have some funny things to say about programming, alright? So

1:06

Speaker 1: I'd just like to say thank Chloe and because as a programmer who's locked in a padded cell all day, it's nice to laugh Um and and I think we don't pro laugh enough as programmers, so um so here I am trying to make us laugh. And and and you know my boss Shoves me away in that padded cell says go do XYZ little inside joke because I do plot I did just did a plot on XYZ and uh and I work at Scripps Institute of Oceanography. My goal, you know, save the oceans So so that's what I do on my day job and meanwhile I crack myself up all the time so I thought, hey, might as well give it a try, you know, cracking some of y'all up So we're in the land of the hippies, aka San Diego, and um we like labels, so I actually don't have a button because I don't really care for labels, but

1:54

Speaker 1: I I respect other people's labels. And so, you know, except for food, I'm always like, is this local organic shade grown fair trade? You know, I like my food labels, right? Well Another thing of Land of the One, we are one with we like to be one with each other, one with nature, one with ourselves. Let me tell you about what I wish we were one with time. It's just like there's just never enough time, right? And anybody program with time? Time zones? Time zones? Oh, those are fun, aren't they? Cause oh man, there's more than just twenty-four when I'm talking to normal people. I'm like, oh, oh, I know a lot about time zones, man. There's more than twenty-four. Some of them are 30 minutes or 45 minutes apart. And then there's daylight savings time. I am like please Can we get rid of this?

2:39

Speaker 1: Like I I've been looking up on the history of this because I'm like, this just is archaic. This needs to go. Programming, you're just like, oh my God, can we not just all be one with time? It's called UTC. You know, I wish. That would be that'd just make my programming life so much easier. So uh new to social media I just started my Twitter. Thank you. I started my Twitter today to to because I was like it's in a spreadsheet and I love to fill up spreadsheets. I've been doing spreadsheets since I was ten, so Twitter. Alright, been meaning to do this at procrastinate sometimes just need a little push. Alright, made a Twitter. Twitter is Pico de Loco, there one. But there because there was already a Pico de Loco. I just made an Instagram account uh like a month ago, only a few posts, a lot of cats, dogs, you know, and I

3:25

Speaker 1: skateboard and so anyways, so it's me, right? Well I was Pico De Loco. I do Twitter, there's already one, so I'm Pico de Loco one. Well, okay, well, all right, so it'll get on Instagram, change my change my name now, and I am now Pico DeLoco1, because guess what? I am crazy. I was crazy once. They locked me in a room with bugs. Bugs make me crazy. I was crazy once. They locked me in a room with bugs. Oh Oh yes, cause programming box. Programming bugs make me crazy! Especially when you're all alone trying to figure out a bug. It makes me crazy. So I'm crazy. I embrace the crazy. I am proud of my crazy label. Some people think, oh, well, crazy is derogatory. No I'm crazy and I'm proud. So oh oh I got one minute left.

4:11

Speaker 1: Sick, I'm going fast because I talk fast. So try to listen fast because I ain't slowing down. Can't stop, won't stop. Social media. I'm on social media So so I'm new to things. I got this 360. I want to get a YouTube. Haven't done a YouTube video yet? Going to. I see ya. And uh Focus, focus, focus. It's what I need to do. I have a little bit of ADD. I'm having a hard time right now staying to this microphone because I like to move. And my boss, meanwhile I just sit in an office all day trying to sit on that chair. I'm trying to be a real good. I'm trying to be a good employee, trying to sit on that chair, sit all day. Sometimes you get tunnel vision, you get in code mode. You're just going, going, going, going, going, right? Meanwhile, it's Oh dang it, I gotta go to the bathroom. Boom. That's when you figure something out, is when you step

4:57

Speaker 1: away, right? You step away, you realize I need to step away a little bit more often, you know? Because we all need to step away. Anyways Last thought, trying to save the oceans. Sometimes there's labels, respect those labels, including recycle labels. Some of y'all use in recycle cups or whatever. Hey, try to put it in a cup. Try to put it in a bottle. If you're not, if it's a recycle, make sure it goes in the recycle Thank you. Wanna hug? Well done, well done.

5:29

Speaker 2: Thanks. Hi everybody, I'm Tim. I write code for the Wharton School, various open source projects, and most importantly for myself. I help organize DjangoCon US here and uh also the Philadelphia Python Users Group back home. I started writing code when I was six years old because my mom won a raffle and uh have been doing it ever since and absolutely love it. I consider myself very lucky to get paid to do what I love with colleagues and friends that are close enough to be considered like a second non-DNA family. So let me show you my uh GitHub chart from 2014. Did that look like the chart of somebody who likes to write code a lot? Uh the truth is in 2014 I wasn't doing what I loved. I had been slipping further and further away from doing what I loved for a long time.

6:16

Speaker 2: And uh, you know, while I still worked hard and had a lovely home, squaring who I thought I was and who I should be with how I acted was getting harder and harder every day. And uh looking in the mirror was getting harder and harder every day. So it was because of this that I had to change just one thing. So what happened here? For many, many years I had been trying to change just one thing on my own and failing. Finally in April, I asked for help with my alcohol addiction problem. In May, I got out of rehab. And you can see a sudden uptick in activity there. And uh I've been clean and sober one day at a time ever since.

7:02

Speaker 2: So thank you. So what happened here in July and August? I made a terrible, terrible, tragic mistake and reopened my World of Warcraft account. So twenty sixteen looks like it might be a bit of a lighter year, but I was contributing to several private repositories. And uh me and my friends from Wharton were also a little busy helping host this conference that you may have heard of the year it was in Philadelphia. But uh when we take a look at 2017, this is where I really started to hit my stride. having gotten more involved with a bunch of open source projects and from Wharton we also started to put some of the packages we had been developing internally on uh GitHub and PyPI. And uh if you take a look at 2018, it's up to

7:49

Speaker 2: so the first year was 21 contributions in the year, 1,053 in the last year. So as you can see now, I'm pretty active again. And doing what I love, writing code with a community of friends, old and new. It's been such a wonderful way to connect and just such a better life. So what I just really want to do here is ask everybody. If there is one thing that you could change about yourself, one thing that you don't like, what would it be? And how much better could your life look if you could change just that one thing? It doesn't have to be an addiction like I have. It doesn't have to be, you know, only you will really know what it is when you look at the mirror at night. But I want to encourage everybody here to you. You know, if you do have that one thing that you want to change about yourself, it is worth doing. It is worth asking for help for.

8:35

Speaker 2: I couldn't do it until I asked for help. And I've had support not just from the rehab center I went to and not just the recovery communities I've been in. My colleagues at work have been incredibly supportive. The Django community, which I have immersed myself in ever since, has been incredibly supportive. And it's been an amazing amount of support everywhere I've looked to try to keep this going one day at a time. time. And the same thing could happen for you. Because uh if you need help to make that change, you know, absolutely ask for it because it really did save my life. And you know, as Gandhi said, we must be the change we want to see in the So if you have a problem, sometimes changing one thing can change everything, and it really did for me. Thank you very much.

9:25

Speaker 3: Hello everyone. Um I'm a software developer and I write software. But occasionally I do not only need to write software, I also need to write documentation. Or put up a website for the software that I've written. Both things usually include screenshots, and doing screenshots is cumbersome. You want to do them on different screen resolutions, you might want to do them with different locales. And once you're all done having created them, they are outdated and you have to start over again. So let's automate this. And let's automate it with tools that we already know or that probably most of you have already seen. The first tool that we want to use is Selenium. You might know Selenium. as something to write front-end tests for your web application.

10:11

Speaker 3: Basically it's a way to remotely control your browser from a program. The next ingredient is Chrome Headless. Chrome Headless is a way to start Chrome without requiring a display or anything attached so you can run it on a server without needing to have a full-blown desktop operating system. And then we use PyTest. PyTest is a test runner, but if you look at it closely, it's not only a test runner that makes it easy to define tests and run them. You to run them in in specific ways, but basically it's a way to run functions in a specific way. So usually you have a test that is just a Python function. Prefixed with test underscore and that checks for something and then you run PyTests and it finds out of those functions in your project

11:03

Speaker 3: and runs them and informs you about the result. But PyTest can do so much more, it can have parameters parameters for input and um fixtures and so on. Uh fixtures for example are s things that you want it uh uh that should be there before the test. um is run. So for example in this um in this code snippet we're uh testing in the SMTP library and We have a fixture that creates an SMTP connection object and then we have a test that declares that it needs this fixed fixture parsed as an input. And for Django there's PyTest Django and in Django itself there's live

11:50

Speaker 3: server test case which allows for uh creating a test case that accesses an actual run server like version of your app. So with all these ingredients um we already know them they're usually used for used for testing In this case, we will use them for screenshotting. And to make it a bit nicer, we use PyTest configuration to redefine some strings. For example, we want to define our screenshots in scene files with shot functions and not in a uh we don't want to call it tests. And then we can use fixtures to create objects because nobody is interested in screenshots of an empty application. We want to populate the database in some way beforehand.

12:35

Speaker 3: And we can define or um and as another feature we define the Selenium client that is already logged into our system. We can also define our screen resolution in a fixture as well as any other options that we want to pass to Chrome. And then we can define every one of our screenshots by simply writing something like a Python with specifying what fixtures we want to be run before and then just calling a utility function that does a screenshot with Selenium. And we can run it by just calling PyTest on our folder and we're done. And with PyTest parameters, for example, you could

13:21

Speaker 3: Do this for every language that your project supports or for every theme that your project supports or whatever, and you end up with a bunch of screenshots. For the application I'm working on, you can find the repository with the screenshot definitions and also the utility code that is a bit more complicated than I've shown here in this GitHub repository. And thank you very much. If you have any questions on that, feel free to talk to me in the hallway.

13:54

Speaker 4: Hello. It's time for our story. Today's story is that's not my emoji. That's not my emoji. Its head is too shiny. This is two emoji put together. This is not an emoji. This is not my emoji. Its face is far too animated. Can you make a unicorn sound? Emojis do not have sound. This is not my emoji. Its existence as a character in a movie is far too distressing. You see, when a studio loves a marketing opportunity too much, they can make a terrible movie out of it where a bunch of the main characters aren't actually emoji.

14:53

Speaker 4: That's not my emoji. It is partying far too hard. You see the pretty bird? He's dancing. Emoji, do not dance. That's not my emoji. Its permutations are underdocumented. Can you waive the Kai emoji? Hello. That's not my emoji. Its ligature is only vendor implemented. You see the cat drinking coffee? Cats don't drink coffee. And this only appears on Windows 10 operating systems.

15:39

Speaker 4: That is my emoji. Its code point is so standardized. And that's the end of our story.

15:56

Speaker 5: Okay, sorry about that. So yeah, this is a talk about crypto. But it's not a talk about that stupid new thing. This is a talk about the original crypto, which of course is cryptozoology, the study of cryptids or um legendary stroke um mysterious creature You know, we're talking Bigfoots and Chupacabras and um the Michigan dog-faced man and all of these wonderful things. And the reason I'm interested in cryptids at the moment is that last week I was in in Ohio and I went back to texting with my wife Natalie in the woods in the dark and it was dark and it was the woods in Ohio and we realized that we hadn't really done our research and this is America. There are weird creatures out here. We didn't know if we were within the range of any of these myths. creatures and I'd like to know so that I can greet them and say hi.

16:41

Speaker 5: So um I obviously hit Twitter and on Twitter I said So just out of interest, um uh does anyone know where I can retain range maps of crypto cryptozoological creatures like Yetis and Chupacabas and so forth? And then I asked a question about it on Ask Matafilter, and I realized that actually no, the internet does not have a conclusive source of range maps for different cryptozoological creatures. So I started a GitHub repository, and this is um This is a GitHub repository which I would actively encourage people to contribute to, where I am trying to get a directory of information on cryptids and their ranges. And so the way I'm doing this is using a file format called GeoJSON. It's a brilliant format, it's a way of representing geographical shapes in JSON.

17:28

Speaker 5: So let's take a look at the Loveland Frogman. This is my GeoJSON file for the Loveland Frogman. A great thing about um GitHub is that GitHub knows how to render GeoJSON So it's rendering this shape for me, but if I click on that shape, I can see that it's got a Wikipedia URL, the name it was first cited in 193 Last seen in 1972, and it's a human-eyed frog described as standing roughly four feet tall. So that was my first cryptid. I have been, you know what? Um I've got some pull requests. Uh Russell has sent me a pull request adding the drop bear. I'm going to merge this right now, and if I'm lucky, it will deploy by the end of my lightning talk. I should have done this a couple of minutes ago. So we now have a drop bear, which is very exciting. So you get the idea.

18:13

Speaker 5: So then what am I doing with this data? Well, so I'm working I've been working on this open source project called Dataset, which is a application that takes a SQL-like database and gives you a UI for exploring it and it gives you an API for getting the data back out as well. And it's the perfect application for cryptozoology. So what I've done here is um this GitHub repository has Travis set up to run a script every time I commit anything. thing and the script reads in the GeoJSO and writes it into a SQLite database. It's all of what? 123 lines of code. There is not a lot to this. So it builds a it builds a SQLite database with all these cryptid It deploys it and then based on this I can build out an API. So here is um the nice thing about dataset, everything's SQLite, so the API is just a SQLite. SQL query. Here is a SQL query that um selects details of

19:00

Speaker 5: cryptids where the geometry overlaps the um where within geomtext, so where the geometry overlaps a point. Here's our current latitude and longitude right now. And if I run that query, I get back the Bigfoot. It turns out America has a lot of Bigfoot sightings. This is now a JSON API which you can feed latitudes and longitudes to and it will tell you which cryptozoological creatures are you are within range of. Here's Travis. Oh look, Travis is uh building the drop bear right now. So if we're lucky that'll be built and deployed in just a moment. I'll talk about Bigfoot quickly. It turns out there is already a database of Bigfoot sightings run by the Bigfoot research organization, um, who are banned from Twitter. If you try and tweet a link to BFRO. net, you get an error message. So clearly Twitter are part of a cover-up

19:47

Speaker 5: conspiracy trying to hide the um Trying to hide the existence of Bigfoot. But they've got three thousand sightings. They publish a KML file. A KML file has latitudes and longitudes in. I happen to know that the range of Bigfoot is fifteen miles from a conversation I had at the Bigfoot Discovery Museum outside of San Francisco. Santa Cruz, thoroughly recommended. Um if you take those 3,000 points and put a 15 mile radius on them, you can see that Bigfoots have been sighted across much of the United States, everywhere in Florida. It turns out everyone in Florida is seen a Bigfoot and here in San Diego. So when I ran that query with our San Diego coordinates, um I did get back the Bigfoot. I've got a IOS shortcut that I that isn't working at the moment, so I can say hey Siri Check for cryptids, that's very useful. Um and if you want something a bit more useful, I do have a version of this that works for time

20:34

Speaker 5: zones instead. Um so please take a look at the project. Draw maps of cryptozoological creatures, since they probably don't exist. You don't have to be very accurate. And um and let me know how it goes on. And before I get hugged, I'm just gonna quickly see if Russ's drop bear has deployed yet. Um unfortunately it looks like the drop bear did not quite make it, but in a couple of minutes there will be a drop bear. So thank you very much. All right. Uh

21:07

Speaker 6: begin.

21:08

Speaker 7: Thank you. Okay, this is the first presentation I've given In a very long time. Uh I'm this was from something I did literally eight years ago, so I'm gonna try and do this really quickly. Uh in 2010, I was at a place called the Santa Fe Institute for a summer program, and they asked us to like create various research projects. So me and some friends stuck something together in three weeks, which was a Django app, which you'll see. Uh four years uh that should have said yeah, four years later I got diagnosed with a really complicated situation which is why I've had three surgeries and why I can't have this part of my glasses on that side of my face. Long story. Still managed to get a PhD in sociology now I'm trying to figure out what to do. Um it went so well my most recent surgery in June that I'm here. and I'm really really happy to be here.

21:58

Speaker 7: And uh my GitHub name is Spool. If any theater nerds who know Craps Last Tape get that reference, please see me out after because very few people do. Right. Okay. So the project was to try to study how people create collaborate creatively and how they might respond to each other and make But basically like if we're working together on making a design, um what do we how do people respond to each other's things? Uh people may be aware of the whole Reddit project that came a couple years ago. This came before that technically in 2010 and you'll see some differences in the

22:44

Speaker 7: So the idea was uh we we printed a t-shirt at the end. It was gonna be a a black uh t-shirt by default, and then we were gonna Put some designs on it. Uh we wrote it in Django and a a language called processing, uh which is really cool for doing some user interface and sound experiment stuff. There was a processing JS back then, which was really slow. There's now P5JS, which I really recommend people try if they like to. What we did is we created a a grid with 64 cells or squares that people could do their own designs on and that whole thing became the the canvas. Um and you could only see your Moore neighbors. Can I see a pair of hands or hands for anyone who knows what a Moore's neighbor is? Okay. Oh, there's one dude. Do you want to tell everyone what that is?

23:36

Speaker 7: Fine. Okay. That's cool. Um you'll see it in a second. So this is just An example of how we've run it in a museum recently. So you get assigned a square in this big grid, and you can only see the edges of your neighbors. So um If you s s there's the the ones to the left, right, top and bottom, uh and then there's the ones on the corners. So those eight are the Moore's neighborhood. Um so you can press spacebar. Do a design and then press spacebar at the end and you see how your design fits alongside the designs of everyone else. Okay. Um Press play again.

24:22

Speaker 7: So that was that um so uh that was the uh You had you could log back in. So when we ran it at a museum, you could only do one design and then see the whole canvas. The way we read it ran as a website is you could log in, you could only see your neighbors. Then if you logged in the next day, you might see how your neighbors changed and then you could respond Um so we had a couple of prog in problems. If someone logged in and then left it for an hour and then logged back in, their neighbors might have changed and then they could they would log in and We ended up they would accidentally not respond to the most recent changes of their neighbors. We should have done that with uh WebSockets. We also didn't take into account time zones, so someone's laptop was on a different time zone and they kept overwriting everyone else's and we didn't know why.

25:09

Speaker 7: That was really frustrating. Um so uh this is what it looked like. So each of those cubes are individual people's designs? Oh the music. That was when a dude tried to do his own and then accidentally overwrote everyone else's. And it's a torus, so the people on this on each side can actually see each other and the people on the tops can see the bottoms.

25:55

Speaker 7: Anyway, thank you very much. So that was the final thing. If anyone's interested in the music you just heard, that's by a really incredible composer named Julius Eastman, who sadly died in poverty in nineteen ninety. Look them up. Thank you.

26:23

Speaker 8: Hello, uh I'm Ernest. Uh I really like to tweet. Um you might think that tweeting is my full-time job based on this incredible brand engagement, uh, but in reality I work for the Python Software Foundation. I also tweet a lot, and so if you want to follow me, uh that's where you do it on Twitter. But yeah, the the Python Software Foundation. So uh the Python Software Foundation, if you're not aware, is a nonprofit that controls the uh intellectual property, uh copyright, trademark, et cetera, for Python the language. Uh also it is a grant-giving nonprofit. And so we raise money and we pour that money back into the community in order to support uh events like this, meetups, uh, and smaller events as well. There's a larger event associated with the Python Software Foundation called PyCon.

27:08

Speaker 8: If you're not familiar with PyCon, it was in Cleveland in 2019. It's gonna be in Cleveland in 2019. twenty uh eighteen. It's gonna be in Cleveland in twenty nineteen and uh I want to talk briefly about that. Um currently the call for call for proposals for Python is open. You can check that URL out to go submit a talk, tutorial. education summit pre presentation poster or a Charlis or a um a talk in Spanish. Speaking of talks in Spanish, the PyCon Charlis uh came out of the PyCon hatchery program, and so this is kind of what I really want to pitch to you. The PyCon hatchery program is a uh a way for your ideas to be realized in PyCon. So if you've ever been to PyCon and not seen something that you wanted to see, uh this would be the way that you would tell us uh

27:55

Speaker 8: What you want to see, and perhaps even propose to do the work to uh make it so. Um so please check out the hatchery program and read more about that. So you might be saying, okay, like So what about Django? The Python Software Foundation loves Django. That run on python. org. Out of that, seven of them are currently written in Django. The PyCon website is written in Django, and uh I love Django. Um Admission, I I haven't always loved Django, um but it but it's gotten much easier in the past um

28:41

Speaker 8: few years. And so being here and being among uh people who are interested in or experts in Django has been really exciting. And so I'm also going to come up here and ask for your help. I frequently tweet and sometimes I tweet about asking for help in Python or in Django. More often than not, when I'm tweeting about Django, it's asking for help. So uh help, please. And when I say this, I I mean this sincerely. Uh if you are in this room, it is probably because Uh Django has either made you feel like you have superpowers or you want to feel what it feels like to have superpowers. And I want you to be involved in uh Django in the PSF and PyCon. if any of that's true.

29:26

Speaker 8: Uh you might just be starting out and I'd love to work with you to get you started. You might be an expert and I would love to work with you to get a little bit more of that expertise. And so that we can all share and grow. So, there's also a bonus. Um, it's not Django, uh, but uh PyPI is a piece of software many of you in the uh room might be familiar with. And if you're at all interested in contributing to PyPI, we have stickers now. Uh you can uh you can check out uh a m micro sprint that's going to be occurring at lunch with myself and Dustin Ingram. And uh that's it. Uh I'm Ernest and you can follow me on Twitter there.

30:14

Speaker 5: Okay, so I'm going to talk about you get folks about one plus one equals one or record duplication in Python. This is a 45 minute talk. I

30:23

Speaker 8: will make it in five minutes. Not sure if it works. Uh so real world data is a mess. Probably you dealt with data like this before. Uh those are restaurants, uh restaurant records and you You see here that clearly those four records here from 0 to 3 are duplicates because the name is quite similar, the address is similar, CDK and Varai. So we have duplicates the here. Uh real world data is a mess. We don't have unique identifiers. And the solution is to perform the duplication. also known as record linkage to join records in a fusy way using data like names and addresses. Mostly we will deal with those kinds of dates, but it can be other kinds of dates And to solve that we should do some fusy comparison of strings.

31:11

Speaker 8: We can use uh algorithms like Jarrow Winkler similarity. If I compute the general similarity between those two similar strings, I get this high number. And with those different strings, I get this. lower number. So I can use that to as a tip for me as an indication of similarity between records. I can do also fusic comparison of addresses. And the trick is to geocode them to latitude and longitude and that'll allow us to clean irrelevant address variations, like a small variation on uh on the street a number or something like that and to enable the calculation of geometric distance using latitude and longitude because we want to group and match things that are close together.

31:58

Speaker 8: And if I uh geocode those two addresses here, you can see that although they have variations and even typos, the latitude and longitude is the same, and the zip code is always is also the same I can grab this from Google Geocoders, for example. Okay, now into the process of uh deduplicating a data set. First we need to pre-process it. We will use the restaurant dataset. It contains 881 restaurant records from the folders and Zagat's guides. And it contains 150 duplicates, and we want to find those duplicates. The dataset looks like this. So it comes with the cluster column, which is the truth about this data. Of course, we will remove that. And we will also remove the phone column

32:45

Speaker 8: because it will make things very easy for us. And we left only with name, address, and CD. And we want to the duplicate using only that. First we clean uh just using some regexes to clean the name. Uh we'll geocode all the addresses so we get the postal lot to the longitude For uh the addresses. And then we can move to the next step on the record linkage process, which is indexing. We will use the library record linkage, also known as Python record linkage tokit And we have the cleaner records. Now we want the pairs to compare to find matches. To produce the pairs, we could do a full index. We could compare our records against our records. Of course that's slow, but uh

33:31

Speaker 8: we don't have enough time to think about a smarter way to index, so we just produce uh our records against our records. And the pairs looks like look like this. Uh we compare zero with one, zero with two and there it goes. Now running the comparisons. We want to compare the pairs to get a comparison vector For each pair. So the comparison vector looks like this. The names are similarities 0. 5, the address similarities 0. 8, and that's our comparison vector. And to compute the similar the comparison vectors, we define similarity functions for each column. We can do that with the record linkage toolkit. Uh we use Gerald Winkler for name, address postal, and an exponential decay

34:17

Speaker 8: geometric similarity between latitude and longitude. And that's what we get. If we run that. Now with the vectors, we want to explore different ways to classify them as matches and no matches. And we can do some simple threshold based classification. We can compute a weighted average over those uh vectors and by looking at data we see that those three are matches so we just consider anything with more than zero point nine uh of uh score as a match. If if we do that and compute from the truth about this data, we see that we got 128 true positives and two false Positives, only two false positives and 22

35:03

Speaker 8: false negatives. So it's quite good performance, but are there smarter ways to solve that problem? Make sure to check active learning classification. It will help you a lot because it will allow you to build a training set for your dealer. Thank you very much.

35:23

Speaker 6: Thank you

35:29

Speaker 5: Okay, so um as I 'm Felipe, I'm partners at Vinta. Um and let's talk about parks So I have an image here uh of two layouts for a park trail or something like that. And I want you to think about which one of those you think is a more pleasant uh trail for park So um to be honest that this is a very trick uh question. I didn't give you enough information to answer that. And the reason is you don't know nothing you don't know anything about the terrain, you don't know anything about if there are trees around, you have absolutely no idea of what what's going on in that area. So

36:14

Speaker 5: to answer that kind of questions, you need something like this book, which called A Pattern Language from Christopher Alexander, and it's from 1977. And this guy he defined a series series of partners and and uh things to help architects to uh build and design good and do good architecture Uh so for example you can take the pattern 120 uh which is uh about paths and goals and it says the the out of path will seem right and comfortable only when it's compatible with the process of walking. And the process of walking is far more suitable than one might imagine. This is very interesting, uh especially because it can be

37:00

Speaker 5: be visualized through this thing called uh desire paths. Desire paths are like in this image, they they show how people interact with the place they live uh through like these patterns in the terrain. Um so for is for instance here we are seeing there are two paths going one that goes straight to the door where there is a fence and the other one that go that goes around so at probably at some point that fence didn't exist there. uh and people just changed the way they they went uh inside the house uh after that also this other example so in this case people are going around the trees And and here is very interesting

37:45

Speaker 5: because like they got the desired pattern, the desired path, and actually made it a fixed uh path and a proper path. So um One thing to remember, and this comes from this idea of patterns and paths, is that when you see a big street, mainly when it's an old one, uh main avenue in a city, uh Probably at some point there was a bare road in the woods. And maybe before that there was just a path where people and animals passed by. And so those are patterns and uh this this is from Christopher, uh the the author of the book, and he says

38:32

Speaker 5: This idea comes simply from the observation that most of the wonderful places of the world were not made by architects but by people So the the book is really uh very based on this on on how architecture is built by uh is made by people and and construyed naturally. Uh so the book has other partners and Uh for example the path shape one, uh there is seat spots. So uh when you um sorry, I'm gonna go It's missing. Okay, so there's another part that's missing here, well, which is about language. So it's a pattern language. Uh we talk about pattern and language. Uh it's just like when you have words. You have words that have separate meaning by e each of them

39:18

Speaker 5: and when you group that together you get a lot more meaning through language. So language is that's a group you you group words to convey a lot more meaning. And uh that's where we get to design patterns So Design Patterns was actually created the book we know, uh the software book we know, was actually created based on uh Christopher's book. And I don't have a lot of time to talk about like design patterns here, but like it's just uh that uh that idea that comes from so let's just uh jump to some takeaways. So first thing design patterns are not created, they

39:55

Speaker 8: merge. So just like the Christopher 's patterns, our design patterns are just observation from how people code. Um design partners are a tool for communication between programmers, uh free speech and code. Uh they are not to brag about, they are to help you communicate with your teammates. Uh and the last thing I want to leave to you is that that quote that says good architecture is about improving people's lives and most of the time

40:21

Speaker 7: this means that what feels more natural or m more pleasant to us. And this applies to both to architecture and to programming. Thank you.

Questions this talk answers

How can asking for help change an addiction problem or another major personal problem?

Tim explains that he could not overcome his alcohol addiction alone; asking for help led him to rehab and recovery. He encourages people to identify the one change they need to make and seek support, because that change can transform their lives.

Discussed at 6:16

How can I automate screenshots of a web application for different screen sizes, languages, or themes?

Use Selenium with headless Chrome and pytest, defining fixtures for test data, login state, screen resolution, and Chrome options. Parameterized pytest tests can then capture screenshots across locales, themes, or other configurations.

Discussed at 9:25

How can I build a map and API of cryptid sightings or ranges?

Represent each creature’s geographic range with GeoJSON in a GitHub repository, then use automation to load the data into SQLite. A tool such as Datasette can expose SQL queries as an API that accepts coordinates and returns cryptids whose ranges overlap that location.

Discussed at 16:27

How did the collaborative Django art project let people create a shared design?

The project divided a canvas into 64 cells, assigned each participant a cell, and showed them only the eight surrounding Moore-neighborhood cells. Participants could design their square, see how it fit with nearby designs, and later return to respond to changes.

Discussed at 23:22

What is the PyCon hatchery program and how can I use it?

The hatchery program lets the community propose ideas for things they want to see at PyCon and potentially help make those ideas happen. It has been used for initiatives such as PyCon Charlas.

Discussed at 27:48

How can I find duplicate records in messy real-world data using Python?

Clean fields such as names and addresses, geocode addresses to compare their locations, generate candidate record pairs, and calculate fuzzy similarities using a record-linkage toolkit. A weighted similarity score can classify likely matches, while active-learning classification can improve the process with labeled examples.

Discussed at 30:23

What are desire paths, and what do they teach us about designing places or software?

Desire paths are the routes people naturally create through a landscape, revealing how they actually use a place; architects can formalize those routes into proper paths. The talk applies the same lesson to software design patterns: they emerge from observing how people code and serve as a communication tool between programmers.

Discussed at 37:00

Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.

More videos by Ernist W. Durbin III, Filipe Ximenes, Flávio Juvenal, Griffith Rees, Katie McLaughlin, Raphael Michel, Sarah Heim, Simon Willison and Tim Allen

Thumbnail for video: Lightning Talks Day 2
0:40:50

Lightning Talks Day 2

Published November 14, 2018

Thumbnail for video: Tasks: you gotta know how to run 'em... by Filipe Ximenes
0:21:26

Tasks: you gotta know how to run 'em... by Filipe Ximenes

Published September 8, 2017

Thumbnail for video: Mixing reliability with Celery for delicious async tasks
0:24:57

Mixing reliability with Celery for delicious async tasks

Published November 22, 2023

Thumbnail for video: Why large Django projects need a data (prefetching) layer with Flávio Juvenal
0:39:31

Why large Django projects need a data (prefetching) layer with Flávio Juvenal

Published November 3, 2022

Thumbnail for video: "Normalize until it hurts; denormalize until it works"... by Flávio Juvenal
0:24:15

"Normalize until it hurts; denormalize until it works"... by Flávio Juvenal

Published November 8, 2018

Thumbnail for video: Lightning Talks Day 3 by Kojo Idrissa
0:42:19

Lightning Talks Day 3 by Kojo Idrissa

Published October 25, 2019

Thumbnail for video: Scheming with CSRF: When platforms manage to break things with Katie McLaughlin
0:25:02

Scheming with CSRF: When platforms manage to break things with Katie McLaughlin

Published November 3, 2022

Thumbnail for video: KEYNOTE: What should you have to worry about
0:45:05

KEYNOTE: What should you have to worry about

Published October 14, 2022

Thumbnail for video: What is deployment, anyway?
0:30:32

What is deployment, anyway?

Published October 20, 2021

Thumbnail for video: Fast on my machine: How to debug slow requests in production
0:23:03

Fast on my machine: How to debug slow requests in production

Published July 11, 2024

Thumbnail for video: Building and scaling a live event platform with django-channels
0:29:01

Building and scaling a live event platform with django-channels

Published June 7, 2023

Thumbnail for video: Security strategies for multi tenant applications - Raphael Michel
0:24:32

Security strategies for multi tenant applications - Raphael Michel

Published September 30, 2020

Thumbnail for video: How to design and implement extensible software with plugins with Simon Willison
0:44:19

How to design and implement extensible software with plugins with Simon Willison

Published December 6, 2024

Thumbnail for video: Increase your productivity on personal projects with comprehensive docs and automated tests
0:25:09

Increase your productivity on personal projects with comprehensive docs and automated tests

Published November 3, 2022

Thumbnail for video: State of Django Panel by Simon Willison
0:48:35

State of Django Panel by Simon Willison

Published November 10, 2018

Thumbnail for video: Packages! Packages! Packages!
0:40:08

Packages! Packages! Packages!

Published July 19, 2024

Thumbnail for video: Don't Buy the "A.I." Hype with Tim Allen
0:26:26

Don't Buy the "A.I." Hype with Tim Allen

Published November 22, 2023

Thumbnail for video: Miracles in Anarchy with Timothy Allen
0:44:12

Miracles in Anarchy with Timothy Allen

Published November 3, 2022

More videos from DjangoCon US