How we Used NLP and Django to Build a... by Vince Salvino

Speaker 1: Come on, no.

Speaker 2: Hi everyone, thank you. Um

Speaker 3: it's my first time speaking at DjangoCon, so I'm very excited to be here. Um yeah as uh This is my email, so if you have any questions or anything, please feel free to reach out. And real quick, if anyone likes to follow along, the slides can be downloaded at that URL Um in case any it's just a PDF in case anyone is looking for that. So real quick um to to give you an overview of what we did is um this is gonna be sort of a talking about a project we implemented. So we worked with the Cleveland International Film Festival to um to build uh a little bit of software for them. This was sort of a hobby project. that we did. And um

Speaker 3: real briefly about the Cleveland International Film Festival. It's a two-week long event, you know, similar to like Toronto or uh um The Sun Sunbeam Sun the other film fest and um they have a lot of independent films was held in April this year And it's a pretty big event for Cleveland, so we were really excited to do some stuff with it. I myself am not a film buff. I'm sure many of you in the room are. Shout out to a movie I saw there, Morse from America. It was an independent film, very good. And I'm not affiliated with the Cleveland Film Fest, so just that's my disclaimer And

Speaker 3: about the project, that's our little robot guy there. Um so we built the movie recommendation engine. You know, there's about over 400 films at this festival, and they're all pretty much all brand new releases, independent. They never they've never been seen, they've never been reviewed. The only thing we have on them is a disc a title and a description. So when you're deciding which one to see, especially if you don't know the producers or don't know who's making the film, you know it's kind of intimidating for an average uh person. So we decided to take the publicly available data that they published on their website and do a little bit of natural language processing on it. to identify which films were similar to uh the other films so we could know

Speaker 3: have a b uh guide as to what to watch. And we also did the Twitter bot too as sort of a fun uh aspect of it that would tweet out uh during the show times it would say um you know this film is starting and then when the film was over it would say this film was ending. If you liked it catch this other film that's similar and it was all done through uh it was all computer generated uh statistics about that. So it was a quick thing we did in a couple days. It's 100% Python except for the use of cron. It was all built with open source software and once again the data was all publicly available from their website. So what we're gonna look at in this talk is we're gonna put on our search engine hats first of all. and figure out how to scrape the data from their website and get it into our uh

Speaker 3: Django database, into our models The second thing we're going to do is look at some Django management commands, which are pretty simple, but maybe new for a few people. They're very useful. And then we're going to get into the meat of the talk and look at some concepts in natural language processing, just some basic stuff, and then explore functionality in the NLTK, which is the natural language toolkit. It's an open source toolkit. It's all in Python. It's very easy to use and it's a great way to learn natural language processing or entry-level AI if you're interested. And then we're going to look at using the Twitter API to make sort of a dumb Twitter bot. We're not making a uh a Microsoft Tay or anything like that.

Speaker 3: And um we're gonna use the cron job to sort of uh give life to our Twitter bot to tweet out at appropriate times. And if we have time, I'll go into the whoosh and the haystack uh searching of the movie database. So the first things first is we need to identify, you know, we're basically starting from nothing. We need to look at their website and identify what's the data and how what do we need to do to represent it And we need to make some models. So the most obvious models are the movie model to represent the film itself and the showtime to represent the time that the film is playing because they some of them will play um two or three times over the course of the of the film fest. So I will just

Speaker 3: go to their website real quick And they've got a whole list of films. Now this may have changed a little bit since it's all over, but Here is a film on their website and as you can see they you know it's it's a very nice looking site. Um It has all the information laid out, but once again putting on our search engine hats, how are we going to identify what is what on the page You know, we know that this is a description, but how will Python know? And we know that this is a title. How will Python know? So for that we use two libraries.

Speaker 3: One of which is URL lib, which I'm sure many people have used in here, and the other is BeautifulSoup. So URL lib basically just gives you the ability to do requests. um sort of network type things. Beautiful soup is it sort of builds the HTML document in the markup unless lets you query that. So if you've ever used uh jQuery, BeautifulSoup is sort of a Python way of doing a similar thing, querying the different HTML tags on the page and identifying what's in there, parsing the data out of them, etc. So I did want to open source this, but because we could potentially DDoS their site from crawling it way too much, uh I didn't, but I'm gonna show you the code

Speaker 3: So basically URL lib is a pretty simple uh tool that will let you oh that's not showing up on there. Yeah. So uh URL lib basically just lets you make a request to any any web any you you know resource out there. It's very straightforward. It reads that then into memory. And from there we will take what we just read and parse that with BeautifulSoup as HTML. So now we have this whole soup object which is essentially the HTML document.

Speaker 3: And from here we can query through it. So you would the the obvious thing is okay, the H1 is probably gonna be the title, right? So we go through and we parse some of that stuff, but as you will as you have seen in your careers probably web pages aren't always formatted uh in a way that makes sense especially when you get uh JavaScript and stuff going on so This ended up being a little bit more complicated than we initially thought to try and find, okay, where is something existing? You know, we have to look through different columns on the page, etc. So naturally this is a little bit hard-coded more than a regular search engine would be. But you can basically just loop through everything on the page and parse out that info.

Speaker 3: Using uh jQuery queries similar to like jQuery, like right here you see. We're gonna find all divs that have these certain classes and then within there find the paragraph tags So um and then right there you've got an object that you can play with and get the content out of. So that's beautiful soup. It's really great if you need to parse any uh HTML. So we basically go through that painstakingly until we're able to identify everything on the page and then loop through here and we we just create our Django object right from that So this is our movie object, our model, I should say, and we we just create it from this stuff that we parsed, and now we have a database full of data.

Speaker 3: But we need to run this because this is sort of a script, right? It's almost like a migration, you could say, because you have to populate your database before we can even do anything with it. And then the other quick thing I'll note is that do pay attention to time zones if you're ever dealing with dates because that can really throw some things out of whack. So here we are using the US Eastern time zone So just always something to be cognizant of. So we've got our big long function there that we've got our big long function that Takes care of requesting the pages from the website and then identifying the data in them. But how are we going to run that? So Django Management Command

Speaker 3: They're very simple to do. They really are this easy. They may seem a little bit cryptic, but when you in a Django management command is just From my virtual ENV, I can just do, you know, Python manage, you know, run server. That's the obvious one. But if you just do help, it will list them all. And you can see down here that we have a few of our custom ones. And the one we're interested in is Scrape Movies. So all that is to make a Django management command is you put a file And within your project, my project is called Web Management Commands.

Speaker 3: It's as straightforward as it can get. And then the name of your command in its own Python file. And it really is that simple that all you have to do is call a Python function from inside your handler. So creating this one simple file will give you the Django management command that shows up right down there. And if you notice I had a little help string in there, so if you do um help and then uh it will output my help command right up here and all of the standard uh options. So it's it's very easy to make a Django management command and then all I have to do is run that

Speaker 3: Which I'm not going to run now because that will make hundreds of requests to their site to scrape everything. But it's as easy as that to do. So I would run my migrations, make my database, and then run this and it will populate my database. So now we've got all that stuff taken care of. And that's code tip number one. And once again you see it just gives you the management command right there. So that part was in principle pretty easy, but it took a little bit longer than expected to once again find the HTML. It's sort of an SEO thing almost. Now into the good part, which is the natural language processing. So before we um

Speaker 3: Look at any code for that. I'm going to go over a few concepts. So as you as you know, natural language processing is is basically understanding uh text in a way that a a human would, but you're doing it with the machine. So in our case what we're trying to understand is how similar are two movies You know, if we've got one about war and one about love, how is the computer going to know that those are even similar? And the most obvious way is look at the words that are in the document So that's what TFIDF is. It stands for term frequency, document inverse in frequency. And basically what that means is it looks at both of the documents and says which terms show up most frequently.

Speaker 3: within those documents. So if you have three documents and one of them has the term love in it, you know, repeated ten times and another one has the term love once and the third one has the term love five times, then document one and document three will be determined to be the most similar out of that group. So it's because of the simplicity, it's relatively easy to implement. But the problem is it's not very smart. Because it it only looks it literally just counts up the words and says which has the the most in common. Um and one other thing I should mention is that when you are making this comparison you remove what's known as stop words, so like the, a, and, you know, your prepositions, that type of thing.

Speaker 3: Um so you're really only looking at words that actually have uh context or meaning rather than just uh Words that are there for structure. So let's look at these two sentences, right? I went to the bank to deposit money compared to I slid down the bank by the lake. TFIDF would say, oh, they're both similar, you know, they are short and they both contain the word bank, so they they must be similar. But we know because we are humans that The word bank has two completely different meanings in this context. And the one of the more traditional examples that you see in textbooks is, you know, I gave my dog a bone or the sailor dogs the barnade

Speaker 3: Right? One is being used as a verb and means to to pester, and one is being used as a noun and means an animal. You know, completely different meanings. So TFIDF really isn't that good at solving this, but half of the time it will get pretty accurate because our language isn't that diverse. So what's a better way to approach this? A better way is to set look at the word sense disambiguation. And what this means is that you actually look at the meaning of every word based on the context that it's in. So once you understand the context, you will know that, okay, this word refers to something completely different than it does in this other context, therefore they are not similar.

Speaker 3: And the way you would go about this is you can't just sum up all the words in the document. You have to look at it sentence by sentence so that you can get the correct context. So once you determine the meaning of each word within each context, you would take another step and look at the lemmas. Now a lemma is basically like an adjective, but the true way of describing what a lemma is is it's the it's the meaning of the word when it's in your head before you actually speak the word That's the true definition. It's hard to describe, but it's basically a synonym, so it's the true abstracted meaning of that word. So we will look at each sentence, identify the meaning of the word within that sentence, and then look at lemmas or synonyms of that word within its context

Speaker 3: And when we do that and compare two documents, we'll definitely have a much better sense for how similar they are. So here's our our same example once again. I went to the bank to deposit money. Here we identify bank as a meaning, and it is in this case a fine a financial institution. that accepts deposits and channels the money. The lemmas would be bank, banking company, financial institution, there's probably more. And then in our other example, I slid down the bank by the lake. Here we identify the meaning of bank to mean sloping land, especially besides a body of water And dilemmas would be slope, curve, side, edge, shore, shoreline, etc.

Speaker 3: So If we were using word sense disambiguation to compare the similarity of these, we would not even put them in the same class whatsoever. And then the third topic on natural language processing that we used was sentiment analysis Now this is something that can be very difficult and things such as the IBM Watson and many other big AI programs, you know, really try to pinpoint this down, but there are some very simple ways of doing it But basically what sentiment analysis means is determining the feeling of the text. Is it positive or is it negative? That's the most common implementation of it. But when you combine it with other forms of NLP, you can get a

Speaker 3: deeper level of what kind of sentiment is actually being expressed. um you know product reviews, negative comments on things, um, you know, they might indicate different uh sentiments. within the realm of positive and negative. So for this example we're just going to do positive or negative, but we will be determining that on every movie description. So now to get into a little bit of the code. So TFIDF We're going to use NLTK and we're also going to use Scikit for this because TFIDF is sort of a mathematical thing. More than anything, you're summing up the words and comparing the vectors of each. each document. So this is the code that we use to do that.

Speaker 3: It's very simple when you use the scikit. So scikit gives you a TFIDF vectorizer. And it also gives you the uh and from there you can basically do a a fit transform So you the first thing you need to do of course is clean all the stop words punctuation etc out get it into a pure uh uh pure list of words essentially And we're doing that on our film descriptions. And then we just run them through the vectorizer. And this here is basically building a matrix because you're comparing every single film to every single other film. So if there's 400 films, it's basically big O of n squared

Speaker 3: Um you're not comparing it to it itself because that would be 100% match, so it's big O of n squared minus n. But you what the result is is a huge grid of you know you can think of all the movies on axis y and all the movies once again on axis x and how well do they compare to each other So it's a big amount of data. And this link here, the sidekit's pretty well documented, so if you're interested, definitely check it out. It's a little bit difficult to install in pip because it requires SciPy as a prerequisite and doing all of those requires a good amount of compiling and C libraries and stuff, but it is just a pip package, so. You can definitely install that.

Speaker 3: Um so it's very easy to do TFIDF. Word sense disambiguation, a lot more difficult. So we'll dump we'll dive into the code a little bit for that. So you could see that my um our TFIDF function. You get to can't see it? Oh So the TFIDF function, this was the exact same code that was just just in the slide, relatively simple. You know, you basically use uh sklearn

Speaker 3: to build a uh a vector for that. Word sense disambiguation is a little bit more involved because you basically want to do the same thing. You want to build a matrix to see how similar are is is each movie to each other movie. But you're not looking exactly at the words in that description. You're looking at the meaning of each word and then breaking that down and to look for similar lemmas. So if in the case of the bank example If one document contains the word bank, and we know that it means a financial institution, and another document contains the word finance. We will still count those as being similar, even though they're different words. So to do that, there's a lot of looping and a lot of good stuff that needs

Speaker 3: needs to happen. And in this implementation I sort of did it manually. We did not use the TF-IDF vectorizer. So basically loop through each sentence here , tokenize it based on the sentence level, not at the word level. Look for the important words, which is basically removing all the stop words, you know, remove your the's, ands, etc. and then use the LESC word sense disambiguation algorithm, which is built right into NLTK, and it's the LESC algorithm. I'm not exactly sure about uh the details of what these algorithms do but it's built into NLTK and this what this does here this is really the magic uh function call

Speaker 3: you give it the tokens from the sentence and the specific word that you want to identify the meaning of it in the sentence. So in this case word would be bank and the sentence would be I deposited money in the bank. And it will give you the syn set in return, which is the list of all the lemmas and the synonyms. So what I'm going to do now is add all of those synonyms and everything to my words to check for. So the words that I'm checking for might actually be larger than the document itself because I want to find everything that's similar to it. And then from there, we basically now we have a complete list of words to check for based on the meanings and the lemmas, the synonyms. Now we're going to do the uh the nested for

Speaker 3: loop where we net where we look through and build that matrix and cross-compare every document. So the results of this are sometimes very similar to TFIDF and sometimes very different. And I will show you It's on. uh the comparator object was what I'm using to uh compare the similarity of two movies. So here you can see one

Speaker 3: one film named Zalos, one named You Carry Me The TFIDF, which is basically seeing how many words that they have in common, it scores 0. 013. 0 means zero in common. 1 means 100% in common. So here it's 0. 01, which means not they're not very close at all. The word disambiguation actually scores a little bit higher because there probably are some synonyms involved in that. And if you look at a few others you can see that there there are some differences. Usually the uh word sense disambiguation scores them more similar than uh turn frequency, document, and

Speaker 3: So we use both of those metrics to when we're deciding, okay, how similar really are these? And then the third, uh the kind of the fun one here is sentiment analysis. And this one's also really easy to do. of NLTK. So we're using the uh Vader sentiment analyzer which is once again an algorithm and it's built into NLTK. It's particularly relevant in our case because the way this uh algorithm works is it was trained from a data set. And this data set consisted of 10,000 tweets and 10,000 movie reviews. And I forget when this was trained, but I want to say it was uh

Speaker 3: maybe like 2011, 2012, sometime around then So it may have changed a little bit. But um in each one of those uh tweets and movie reviews was tagged by a human as saying, okay, this one's positive or this one's negative. You then feed those into the into the uh computer, into the algorithm, and it learns based on the syntax and the grammar and the and the word choices, what a positive text looks like and what a negative text looks like. So from here, you can basically feed anything you want into the Vader algorithm, and it will scale it from negative one to positive one. whether it's 100% negative or 100%

Speaker 3: positive, based on its um learning data, its training set. So because we it was trained on movie reviews, it was pretty pretty relevant to us, I think. And once again the code is very simple for this because it already exists in the NLTK. You have a sentiment intensity analyzer object. Really couldn't get any easier than that. So all you have to do is in our case clean up the data a little bit, run it through the polarity scoring function. And what we're looking for is the compound score. It does produce a lot of other scores and metrics. The compound score is sort of the overall, the average, just you know

Speaker 3: Which is what we wanted in this case. And uh and that's really all you have to do. So we just loop through the movies, run it through the uh NLTK uh functionality and it gives you that. So it in in some ways it makes you seem a lot smarter than you actually are. And I'll show you a quick example. I should have started out with this too. But This was the uh the site that we made and it used to show you now playing and recently ended. Um obviously it's all over but there are a few lists that we can pull from And we represented that as a dark plot or a upbeat plot. Um because if pertaining to movie reviews, that's what or not reviews but movie descriptions, that's what it it most uh closely corresponded to.

Speaker 3: So this chart here just goes from negative one to to positive one. And it varies per it varies per film. And I I did read through a few of them myself to kind of spot check. And in in general, it's pretty accurate. There are a few that were kind of off But uh in general it was pretty accurate so very good functionality right there in NLTK and an NLTK is primarily designed to be an academic type thing just to learn and to play on It's probably not very good for doing any real, you know, if you're looking at a business functionality or something, it would not be good for that. But for projects like this, you know, it's great. So so that's what we did. And um

Speaker 3: So now that we have established three different forms of of National language processing the descriptions of these movies, it's time to crunch the numbers. So our crawler pulled in 300 and uh 436 films from the Clevelandfilm. org website We did TFIDF on each film compared to each one, so that was a big O of N squared. And we also did the word sense disambiguation on each one. So we um Broke it down into every sentence of every review and cross-compared all that. Once again we ended up with 189,000 comparisons by the time our database was fully populated And as I mentioned, we use the comparator model to store the uh

Speaker 3: the comparison between each each film. So the actual movie model itself doesn't contain any of that, it just contains the movie information And how we did that was through a set of management commands Okay, and I'm going uh way too slow here, so I'll speed it up. That was a Django management command. And then the last good part of this was the Twitter API. So Doing this is very very simple if you're just doing it with your own Twitter account. This is what we did, which looks a little bit complicated, but the simple form using Python is super simple. You just create a Twitter app

Speaker 3: at apps. twitter. com. Since you're the owner of it, it has access to your account and you can access all of your API keys right there at apps. twitter. com You basically plug those into Python and you can send a tweet. So I will send a quick tweet here just to just to show it off because it's kind of cool And I made my DjangoCon tweet function here. It's literally as simple as doing that, calling uh Python

Speaker 3: twitter. update status. And if I run my management command I get a Django 1. 10 uh deprecation warning, but then I also get I also get my print line uh tweeted and if you check out the Twitter site Right there. So all we did was use a cron job. Um which is basically a one-liner where you in your cron uh entry you enter the virtual environment.

Speaker 3: that your um your project's running in, you run the management command, and then you just specify to log it So we just ran that every five minutes and it would run through and and query the database and uh tweet out what was coming up. So So that's uh that's the bulk of our project. I hope you got something out of it. I don't know if we have any time for Q<unk>A, a few minutes maybe

Speaker 4: Command preventing a lot of the Uh thanks for the talk. I was just going to say the website looked really nice. I was going to ask what sort of front-end frameworks you use for that.

Speaker 3: So for this one we just use materialize. css. Uh it's basically a clone of the uh Google material design It's very simple, it's pretty good. I found a couple little bugs in it. We normally use bootstrap for most stuff, but we we wanted this to have a little Android feel since it's kind of robot themed. So yeah, materialize. css

Speaker 5: Have have you thought about the possibility of evaluating your comparison algorithm by say piping a front end into something like Netflix where there's a wide uh where you ha already have a wide knowledge of of the movies that are out there. and you would be able to f you you could you could make a kind of a qualitative analysis of of how well your engine performed with movies.

Speaker 3: That would be yeah, that's a great idea. And um once again enlarging our data set too would be great because um The sentiment analysis was just off of a small subset of 10,000 reviews and our own comparisons were just programmatic. They were not going off of any existing data. So yeah, probably for version two we would want to compare that to an outside source. source of data for even more uh accuracy.

Speaker 6: Hey um So I'm not terribly deeply familiar with NLTK , but I wondered the previous question, did you test how well it performed and how could you do that? was was good. Um I also wondered uh why you uh did the parsing uh with Beautiful Soup. more by hand than using a tool like Scrapie that is probably um probably heavier, you know, just 300 movies. So but I think the biggest thing I wanted to mention is that it turns out looking, um Googling on NLTK while you're doing the presentation, since I'm familiar with open NLP and

Speaker 6: you know Java stuff. Is that it was developed at UPenn. And the other comment I wanted to make was that Because this was, it took me a moment to understand. I had to sort of think about it myself while you were talking. Because this was for the Cleveland Film Festival, there wasn't any need to do a search function, you know? People were already there, they were looking at the schedule, what do I go see next? You know, and I think if you do this again, that would be a good thing to point out because people are probably familiar with you know how you would do this with search, just pipe it into solar or something or Elasticsearch.

Speaker 6: You know.

Speaker 3: Yeah. Uh and actually I I if we had a little more time I I went a little too in detail, but I did um There if you want to download the slides, there are a few additional ones with uh Haystack and Woosh just to do a really simple search. Um and it wasn't really relevant to any of the NLP stuff. It was just part of the website that we made um, you know, to to search through that information. It just made a simple search up there. So once again not relevant to the NLP or the Twitter part, but it was part of the whole project. So there are a few slides on implementing a really simple haystack search in like three steps. So if you are interested in in Some searching.

Speaker 7: This is naive. I didn't understand why you use the manage pie interface instead of just calling the Python code directly.

Speaker 3: So the reason we use that was because every all the Python code needed to run within the Django environment. So that's yeah, so that's fine. I guess you could drop into the Django shell and call it.

Speaker 7: Yeah

Speaker 3: But the management command provides sort of a simple way to do that rather than having to drop into the management shell so that you can access the models in the environment. Any

Speaker 7: more questions? This would be the last one.

Speaker 4: Since you did the work to have the word sense disambiguation, why did you keep the uh the um original method of just the word count comparison?

Speaker 3: Um we sort of kept it almost as a crude benchmark to see is there really a difference. Um and originally when we had our when we had the uh the website where you it would show you the similarity, um You know, this is just making a simple Django query to pull the most similar ones, just ordering top to lowest. Um we we we did it with TFIDF and Then some of them were not very similar, so we're like okay let's try a better way. So we did uh word sense disambiguation and the uh similarity results changed when we reloaded the page basically and um they're a lot more accurate. So we have both in there for reference, but we primarily looked at the word sense disambiguation first ,

Speaker 3: GFIDF second

Speaker 7: All right. Thank you everyone for coming. Another hand for Vince