Demystifying Natural Language Processing across several industry verticals

Hi everyone, I'm going to be talking about demystifying natural language processing across several industry verticals. My name is Jyotika Singh and I work as a VP of data science at I6 Media, which is a content and audience intelligence company based in Washington, DC. I am further attaching my social media handles here just for reference as I'll be posting the slide deck after the conference on my Twitter account. And also in case anybody has any questions that you are unable to uh ask during the conference, please feel free to reach out to me on Twitter I also have an upcoming book in summer to fall of 2022 with CRC Press titled Natural Language Processing in the Real World, which will contain a lot of text processing analytics and classification details.

and application across with practical implementations across 15 plus industry verticals So let's get started. What is NLP? NLP is natural language processing and is essentially the way a computer understands language that is spoken or written by humans Humans have been using different ways of communication over time and the ways have naturally evolved, such as the kind of words we use, the language. A good example would be abbreviations such as L O L and BRB were not prevalent before the 2000s. So this is something that evolved over time with humans continuously using language. And that language that gets evolved over time that is used by humans to communicate is called natural language.

Processing of this natural language, when we say natural language processing, that really means uh anything to do with the computer trying to understand this language. Now that can be something from as basic as just counting words and word frequencies using a program or something much more complex such as a voice assistant being able to hear our voice understand what we're asking for and also generate a response and answer us to maybe a question that we asked You have all seen natural language processing applications around you multiple times and some of the examples include home assistants such as Google Home, Alexa. Anytime you're able to ask any questions and you know they come back to you with answers, that's an example of some of a technology that is powered by natural language processing.

Other examples include these automated text recommendations that you get when you're typing emails these days. So something like this: Hi David, hope. When you say hope, you know it tries to autocomplete And guess what you're trying to say in that particular sentence. So helping with any automated replies and quick reply features is also powered by natural language processing. Furthermore, you may have noticed that in your email there are the there are so many different categories and classifications of email folders, one being spam. And that is also powered by natural language processing is being able to detect email that is more likely to be spam into a separate category. And this is something you mostly don't see in your inbox.

The technology does pretty well in removing our spam emails successfully Other examples include any technology such as language detection, language translation, and also you may have noticed when you are browsing a certain type of content, you get all these other content recommendations that are all related to. what you searched for or what you were watching or the kind of post you're interacting with. So all of that also happens with natural language processing backing the whole process Throughout this entire talk, we are going to be talking about how different industries leverage NLP and where they use it. So these industries are going to include e-commerce, finance real estate, social media, marketing, and email.

We're then going to be talking about some practical NLP applications using Python, which include sentiment analysis and then consumer service agents, also called chatbots. Let's talk about NLP in e-commerce. E-commerce is essentially buying and selling of goods and services on the internet There are different types of e-commerce businesses. It's generally classified into three categories. It's business-to-business, such as Shopify, where businesses are selling to other businesses. business to consumer where businesses are selling to consumers such as Target and so on and consumer to consumer such as eBay where consumers post about selling their product and other consumers are directly engaging with that

to buy that. Let's look at this. We are on Target and we search for Tumblr with the wrong spelling that is still able to realize what kind of Tumblr we might be talking about. So in spite of the spellings it is returning the right tumbler. Let's say you're interested in a tumbler with a straw. So you clicked on that. Now if you see in the more to consider section you see a lot of other tumblers and most with a straw because that's what uh the kind of Tumblr image you first clicked on and the product you first viewed. This is an example of NLP that we have seen uh in a particular e-commerce setting Other examples include something like this. Let's say we're searching for formal pants and then we click on something.

There are a lot of reviews that are categorized for us and a lot of reviews where even customers categorize them, but then there are categories that are not filled in by the customer, such as fit, size, color. But these comments are categorized at the back end so that the consumer is able to get the right view of the comments that they might be interested in. So if I'm interested in fit This is where I can see all the comments that are talking about the fit of this product. This is also an example of NLP where your content, in this case comments, is categorized into different categories such as fit, colour, and so on. So, those are some examples of how we see NLP and how NLP is used in e-commerce setting. Other uses of NLP in e-commerce include intelligent search functionality.

Product recommendations, as we have seen in the previous examples, customer review analysis, not just categorizing customer reviews into different buckets, but also analysing the sentiment to understand which product may be selling low because of particular sentiment of the product of the consumers buying the product, whether it's negative, whether it's positive What is it really about? So really analyzing comments of consumers around certain products. Chatbots is another example. You may have noticed even on Amazon or Target, many such places There is an option for using chat to interact with a customer care representative. When you start the chat, typically there are a few questions that the chatbot asks you and tries to give you automated answers before it actually transfers you

To a customer service representative. If it's something really basic like order status or return status, all of this can be returned to you without actually connecting you to a customer care representative, a person itself. So that is chatbots, which really helps in customer service and creating quicker solutions where some of the answers can be presented to the user without really human interactions. Virtual assistance is a is a similar but a little bit different experience where in some locations when you enter a a shop or even when you're using an e-commerce website, uh there's an agent which is chatting with you and trying to recommend products to you and maybe ask you questions like what are you looking for

uh something adventurous so these are the products you may like and these are the places where you can see them or check them out uh so very similar but just a way to interact with the uh the people, the consumers without actual human interactions, but the experience is set to be close to how humans would interact with each other. Customer service analytics is another use case where any calls or charts actually happening with a customer care representative are analyzed so that they are able to serve consumers better and they're able to analyze when a user may be frustrated or when a user may be happy and then see why that is happening and really optimize how the interactions are done with the consumers. Other use case includes translation

for more of a global reach because there are so many different languages across the globe and then there's some businesses which are globally set as well. some e-commerce businesses. So being able to do language translations accurately is another application where we see NLP in e-commerce. Moving on, let's talk about NLP in finance. Finance industry includes a really broad range of businesses. That is everything to do with managing money in some way or the other. Examples include banks, credit unions, credit card companies, insurance companies. Individual managers, investment funds, stock brokerages, and so on. When we are thinking of finance, one particular use case of NLP comes to mind, which is about stock predictions

News in general influences people around the world, especially when the news is related to stock. It influences people who are investing in stocks. And that makes them think any such news related news makes makes them think whether they should buy or sell some kind of stocks. And this entire thing in turn, uh the basically decisions taken by the investors in turn have a Either a positive or a negative impact on the price of stock trading on exchanges. A popular example includes this one time when Elon Musk uh tweeted about uh using signal and encouraging his followers to use signal. This actually led to a surge in a unrelated stock with a very similar name So news, tweets, a social media post

really do impact how stock market changes and how some stock price varies in a positive or a negative way The uses of NLP for finance range from a broad category of first just finding financially relevant content on the social media because people are posting about so many different topics. Finding what the relevant uh topics about stocks would be, for example, is one way uh NLP is used to just being able to correctly identify content that is related to stocks. Secondly, any sentiment analysis for tweets and identifying entities to relate to which entities are being tweeted about or which companies, which stock prices.

which stock names are being tweeted about and how people are reacting to it. Is it like in a positive way or a negative way? All of this together combined with analysis of the social media post helps in stock price predictions algorithms. Furthermore, there is a lot of other places where NLP is used and they can be in any voice-to-text or text-to-voice applications. any real-time translations of any content from one language to another and especially agreements, legal document review to review mass amounts of documents and trying to find anything that would raise any flags. in a faster manner, in a more automated manner, and to reduce the amount of manual labor labour involved in going through all the documents.

Furthermore, chat sp chatbots, just as we spoke previously, they are to help people with any uh you know concerns, any basic questions before actually connecting somebody to a customer care representative. So chatbots, customer support, that is also a big use case in finance as well. And then also customer call record analysis so that they are able to serve the customers better by analyzing how people may be reacting to certain uh traits and you know how they're communicating with the consumers. Next, we talk about NLP in real estate. Real estate is a really broad term and essentially refers to a property that consists of any lands and improvements that are made on top of the land.

So this includes many different types of real estate that we have seen around us majority of the times. Land is just undeveloped property. residential where people actually reside, so apartment buildings, condos, townhomes, single family homes, commercial, which is more uh on the side of office buildings, parking lots, shopping malls, and locations such as that. Industrial is more around factories and any mechanical productions, construction sites, any warehouses, all of that comes under industrial. Now those were the type of real estates. The real estate industry includes so many different branches as well, from developers

to brokerages. sales and marketing, property lending, property management, and then professional services. Let's talk about a full-service real estate brokerage such as Redfin. You see this is a platform where people post any listings. Along with photos and descriptions, price, important features such as number of bedrooms, number of bathrooms, the square foot area And then they leave in text as well about just mentioning the locality, the condition of the house, and all the things, all the features of the house that they want to highlight. An example includes like what you can see on the screen. Sometimes there's a text description where the user mentions a lot of details about the listing, but maybe does not fill out the sections that require additional filling such as sp

specific columns for number of bedrooms, number of bathrooms, or any other features that buyers may be interested in So a particular application of NLP includes taking the text that is filled in by the seller and extracting any useful information out of that that either helps in enhancing the information already present on the website to get more users uh to being able to filter by some of these features or just filling in any missing information and just enhancing the whole description and how these properties can be searched. So in this example that includes detecting things like swimming pools, three-bedroom, three-bathroom children's playground, energy efficient windows that are dual-paned, presence of fireplace and a lease solar system. So all of these may be really attractive to particular buyers

And if this is not explicitly mentioned, they can be extracted from descriptions that are provided by the sellers themselves. These really uh Help improve searchability of certain properties and for a buyer to be able to filter on certain parameters that the user may not have explicitly filled out. This also helps in checking any mismatches between description and the columns that the user may have filled out explicitly, such as if it says three bedroom but the description says two bedroom, well that's a particular mismatch There was a study that further observed that natural language processing analyzes hidden value in text descriptions, and doing so, this actually increases the property value between 1 to 6% on average.

And this is because the property gets so much more searchable and just has more highlights to it that otherwise would be hard to find if it's just all embedded in the text description only. Looking at the larger list of uses of NLP in the real estate businesses include autofilling of missing feeds from property descriptions and enhancing search, search engine optimization. So that there are keywords that are used that are highlighted properly, so a particular property is better searchable and more users, more buyers are able to find and search for it. Other applications include chatbots very similar to the ones that we've talked before to help a customer with a better customer service experience and being able to reduce the manual labor involved.

uh in dealing with different customers, especially for basic requests. Others include legal document review. So brokerages have millions of documents that are stored in their digital archives. And sometimes they need they all are important in a particular th type of setting or a deal and uh it would reduce a lot of manual effort when there are systems, there are natural language processing techniques that are in place to go through these legal documents and flag anything or highlight anything that might be of interest uh such as you know document summarization or finding certain keywords that may be what they're looking for in particular documents This also includes assistance in compliance with GDPR and California Customer Privacy Act, which

essentially means that if anybody does not want their information in a particular database uh they can contact the business and they have to remove their data. So just being able to make sure which user is being matched to the right one in their database. all of that uses NLP as well and furthermore sometimes accidentally or not but there may be some personal identifiable information embedded in certain documents that are made public or certain descriptions. So to be able to identify this personal identifiable information which is also called PII. So identifying it and removing it from their records. Next, let's talk about NLP in social media. People use social media globally.

Millions of posts going on across different social media platforms, and somebody who's in one location makes a post and people from several different other locations are able to view it, interact with it, whatnot There are so many different social media platforms where there is content such as posts, which is text itself, videos, images, and whatnot. Especially if we consider something like YouTube, there is video data there which has something visual and something audible, but also other things such as title of the video. description of the video that is filled in by the user. So all of this is a lot of text data and sources of text data on social media. Furthermore, other sources include comments that people make on videos.

It has a lot of text and a lot of potential in doing analysis to really understand how consumers are interacting with a particular piece of content. Places where you may have observed behavior that is powered by natural language processing include such as searching for natural language processing on YouTube. And when you do that, you see all the videos that come in recommended are very related on a related topic. They're about natural language processing, about machine learning. and all all related content because of the first search term that you that you filled in and the first video that you watched There are many uses of NLP in social media. One thing that we have discussed includes similar video and post recommendations, but there are so many others that include content categorization.

So being able to just take text and categorize it into whatever categories your business requires or your product requires. Topic modeling, being able to get keywords from text or get topics extracted from text automatically. Post and comment analysis which includes sentiment analysis and just categorizing comments into different categories based on what it is that you want to filter. Furthermore, it includes trend analysis which takes into account statistics on social media posts and what the post is about to do a trend analysis. Also, this a lot of times includes things like time series data where you're trying to analyze a particular popularity of a topic from one year versus another year or a month by month or a day by day as well

Further, social media is used for audience targeting, such as finding relevant audience just based on how they interact with social media. So, for instance, if somebody's Really interacting with all content related to NLP, well that person uh may get you know ads uh that are related to NLP, other NLP content, NLP articles, NLP books So audience targeting uses a lot of social media data as well to identify audiences that may be interested in a particular topic or content. Fake news classification is a very popular example as well where there's so much going on on social media and sometimes uh it's all not true stuff. So able to match a particular article or news with other sources to identify it as real or fake is another application

which uses NLP in the back end. Customer support, same thing, either chatbots or just customer support calls, a lot of that is powered by NLP and then also sensitive content filtering uh there 's there are many content sometimes posted um across which may be sensitive or for mature audiences and to be able to identify this content algorithmically so even if the user does not flag it It does not stay online for more time than it should or just being able to flag any sensitive content so that it's either the users are advised before they watch such content or otherwise. Some examples of trend analysis using social media include a shift in any video

engagements and really analyzing that over time. any changes or any correlations of that with any other data sources and really extracting keywords or you know key phrases from content and seeing what is what is it that is really highlighting from a certain section of posts or segments. Here I've attached this paper that essentially talks about Comparing travel vlogs between 2019 and 2020 and how the statistics of user engagement has shifted month over month between 2019 and 2020 and how that correlates with flight searches between the same timeframe and what people have been talking about in the timeframe of 2020 uh when they are interacting with YouTube

uh content and when they're leaving comments such as what kind of content they are watching. A lot of things surfaced up like hiking and beaches and a lot of things that people were more able to do in 2020 And then really what location names people were mentioning. So all of such analysis can be done using NLP as well that uses social media data. Let's go on to NLP in marketing Marketing is a really broad industry and it's essentially promoting or selling of products or services that include any market research or advertisement. We spoke about advertisement a little bit and how social media data influences ads and how basically these ads are uh targeted for particular consumer segments with social media data helps identify

and essentially advertising is a component of marketing There are four P's of marketing which are product, price, place and promotion. The four P's collectively make up the essential mix a company needs to market a product or a service. Neil Borden popularized the idea of this marketing mix, which is the four-piece , and the concept of four-piece in general in somewhere about 1950s. Marketing includes identifying ideal customers and drawing their attention to particular products and services. It applies to most industries because A lot of industries have either a product or services that they're offering and they need to be able to communicate what they do to their audiences or people who would be engaging in their content.

Talking about ads, has something like this ever happened with you where you're searching, let's say, for king bed frames on Google, and then later when you open Facebook, you have ads for king mattresses? Well that is essentially advertisement and everyone from New York Times to part-time bloggers can be considered as digital advertisers or digital publishers. Advertisers essentially want to reach their desired audience and publishers which the which are which basically publish ads, they use ads to monetize their content. The targeted audience is identified using cookies and IP addresses. So there's a essentially um

Text files in your browser that track information that you have searched for. So your IP address is kind of like your house address for your system that you're using for your computer And it shows where you're located. So then these segments combine uh with the kind of searches you use, combined with your IP addresses or you know what information your cookie gathers, uh they help advertisers reach you. So that's how you search for one thing in one place, but you're getting an ad in another place. All of that really happens by all of this. Google is moving towards blocking third-party cookies by 2023 from Chrome browsers. So that does raise other ideas now that advertisers are marching towards

Because the world is going to be kind of cookie-less soon. So without cookies, they would be more relying on search data, social media, and so on. There's so many components to marketing and all of that starting with the product or the service that they're selling and the kind of words or the grammar, just the verbiage around it they use uh to popularize it or to define it that also comes into marketing. How they define their audiences and audience targeting strategies And essentially once they run an advertisement campaign, how they measure it to see whether they it was successful or not. All of this is marketing and it uses NLP in the back end in some way because if you think about it uh you're getting ads based on something you search for. Well that is all text and that is all analyzed to identify certain brands,

certain products that you may be interested in and that information is how advertisers are able to reach you. There's several other uses of NLP in marketing. There's just so many more. It's a huge field and that starts with topic extractions for effective content creation. So being able to see what your audience may be interested in, the kind of words that they may be attracted towards, identifying that from existing content to create new content Sentiment analysis to understand how consumers interact with a product or a services that could be yours or similar to yours so that you can structure your own product or services accordingly. Audience identification for targeted messaging, we have talked about this.

Improved keyword detection for SEO techniques to basically increase searchability of a particular product or service. essentially a piece of content by including keywords that would get m good hits when users search for a particular thing Other examples include chat bots that we have spoken about before, trend identification, which is trying to identify certain trends using search history text, product descriptions, articles, social media data to basically identify trends that further help in marketing of a content. Furthermore, uh there are a lot of AI systems that are not being used uh in a few places where uh What attracts users is a lot of catchy slogans, so using AI to actually produce these slogans for getting ideas on how to market content

other things uh like voice search uh which is you know getting really popular a lot of people now use uh voice to communicate uh with any either their smart devices or any devices that are capable of uh working with voice data to search. So that is getting more and more popular and that's what people want to include in their products and services as well. And then creating buyer personas. In marketing campaigns, they basically, based on their product or services, define the kind of people that they want to reach in terms of their ads for that particular product. So personas such as uh kind of a definition of the people and their interests that they want to target such as if it's a product that is only for males so like males

uh if it's product for older males males of 55 plus Living in a particular area, interested in a particular sport. These are kind of audience personas or buyer personas that they create with the help of NLP as well. Finally, we talk about NLP in email. We have seen this before and keeping this very brief, but automatic spam detection, able to identify email as spam or not. You may have also noticed that in Gmail there are other folders that can be created such as Promotions Primary and Social, in which they try to categorize all your social related content into social. uh category on your email uh

promotions anything where you get ads for certain promotions on different websites or different e-commerce that goes into promotion and any other emails go into your primary. So identifying the kind of email, the the categorization of email That is one thing and other thing is autocompletion of email text which we have also seen before. There are many other places where NLP can be seen, especially in writing such as you know automatic grammar detection, just autocorrect in general, which has been so popular. Every time now we type something on our phone, you know, if we misspell something, it just autocorrects. So all of that also uses NLP. Next, we talk about implementations and looking through some of the very popular implementations or use cases that we have seen across the several industry verticals that we have spoken about.

First up would be sentiment analysis. You've seen sentiment analysis is so popular, it's been used across several industry verticals and it is highly powerful in analyzing how a consumer may be responding to your content, product, service. business anything or how a consumer really feels about uh any any listing that you have or even when we have stock uh spoken about stocks we see how sentiment analysis plays an important role in prediction of stock prices. What sentiment analysis in general is essentially the ability to categorize a sentence or text in general into a Sentiment such as positive, negative, neutral.

There's an extension to sentiment analysis would be emotion analysis, which further classifies us into which happy emotion, which sad emotion and which neutral emotion, uh other things like even sarcasm. But talking about a very basic use case. Sentiment analysis would just take in piece of text and be able to say whether this is positive, negative or neutral Let's consider a very very basic implementation and let's see what that would look like. It would essentially be defining a bunch of words that convey positive meaning. A bunch of words that convey negative meaning and then in your sentence removing any punctuation and then fighting for the presence of any of these words Also lowercasing so that you know in case somebody uses all caps or first letter caps of a word,

if you don't want to really miss a detection, just lower casing everything So lower casing your text, removing punctuation, and then seeing uh how many words it has in common with our positive words list or negative words list. If there are more positive words than negative, positive sentiment. If it's either equal or you know there is no word positive or negative word detected, then neutral. And if there are more negative words than positive, then negative sentiment. So we see in our examples that we were looking at in the first slide, we had I love ice cream, which is in this implementation detected as positive. This was stupid as negative because we have the word stupid in there. And then who wouldn't love a headache? That's a kind of an example

which uh has positive words, but then it also has words like headache. Which may not be in your negative words list. So in this basic implementation, that is detected as positive, which is incorrect Now it's possible to increase your negative word list further, so it includes more that you have seen, such as headache here. If we had headache, it would probably count as neutral rather than negative. But there are also other things that can be done to further enhance it, like weight certain words. So if there's their negative words and then they're positive words, do you want to maybe just assume it's negative? Depends on your use case So we see it's really limited though to the words that you define as positive or negative rather than being something more generic or something that is unseen

by the person actually writing this code But nonetheless, this is a very basic implementation of sentiment analysis. Other implementations could include building a machine learning model, so essentially having some starting with some label data that tells you certain sentences are positive, certain are negative, creating a very basic bag of words, count vectorizer, TFIDF type of features and then passing them through a basic classifier that can be something like logistic regression or knife base actually works really well with text. So building that would then further help you detect words that you may not have thought of and especially in that case if you feed in who wouldn't love a headache as a negative sample then it would accordingly

weigh certain features and essentially do more than what we as humans can create with an if else logic The good news is that if you don't want to do either of that, there are really great open source tools that can help you do sentiment analysis in Python. One of them is Vader, which is Valence Aware Dictionary and Sentiment Reasoner. That's the full form But essentially it is a lexicon and rule-based sentiment analysis analysis tool. And then it is specifically attuned to sentiments expressed in social media. So if you have text that more resembles how people write on social media, Vader may really work very well. Further in text blog is another tool which is which uses NLTK for a lot of its implementation and it just doesn't do sentiment analysis, it also does a lot of other text processing operations.

Such as part of speech tagging, noun phrase extraction, tokenizer, and also sentiment analysis Uh there there's a lot bigger list and there's a great documentation. I've attached the link right there. So if anybody's interested, please explore more. But essentially, let's pass in our sentences And see how these tools perform on those sentences. So we have I love ice cream, which is detected as positive. This is stupid detected as Negative, the highest score is negative, and then who wouldn't love a headache with also the highest score as negative? Here we see for Vader There are other scores as well, which is compound. And the compound score is essentially just a metric that calculates the sum of all the lexicon ratings,

which has been normalized between minus one and one. So if it's minus one, this is most extreme negative and plus one would be more extreme positive. So that is with Vader. So now we see that you know the examples that were doing not so well in our very basic implementation are doing well in uh Vader. From if we consider text blob, we have other kinds of scores like polarity and subjectivity. So polarity is essentially if it's between 0 and 1, it's positive, 0 is neutral, and any negative number is more leaning towards negative. The higher the number, the higher the strength of the sentiment. And subjectivity is essentially for a value between zero and one where zero is very objective

and one is very subjective. So in this case we see I love ice cream has a polarity, positive polarity, so it's positive sentiment text. This is stupid as a negative sentiment test And who wouldn't love a headache? Well it it says it's positive, but you know, we know it's a tricky uh sentence as well. So if you actually imagine your data set that you want to do sentiment analysis on would be a little bit, you know, using language as such, then you know you choose the tool accordingly and really weigh the tools that you have uh for your own data set. A very popular practice is that if you have a particular data sets that you want to do sentiment analysis on, maybe just manually label a hundred and pass them through these tools and see which ones are matching your manual label more.

that may be a better tool. Here I've attached further many other reading resources in case anybody is interested Further in exploring the field of sentiment analysis. There's great research that is happening using deep learning, and I've attached links to certain articles and certain papers that I found were very helpful. Next, let's talk about chatbots. We have seen chatbots has been a very popular application of NLP across so many, almost all industry verticals that we have discussed. So let's really talk about chatbots. what they are and you know some basic implementations and really behind the scenes of what kind of components it involves. Now chatbots are essentially a computer program that simulate human conversations

So they can be either voice-based or text-based. There are so many other names that are given to chatbots. They are also referred to as conversational interface, conversational UX, conversational app, voice bots. chatbots, conversational experience, and many others. At a high level, there are two types of chatbots, rule-based and AI-based Rule-based are very simple and essentially like decision trees. Basically you know what the what the customer is gonna ask for and you only can handle a particular amount of or kind of questions or statements They do lack the ability to answer any unfamiliar questions. So if you have a more complicated use case, then rule-based may get a little bit too complex to build and may not scale well.

AI-based chatbots are more complicated, they use natural language understanding, and there are so many existing NLU services and platforms that can be leveraged if you want to build something like that. Natural language understanding is essentially also called natural language interpretation. It's a subtopic of natural language processing in artificial intelligence. It deals with machine reading comprehension Essentially, natural language understanding is considered as an AI hard problem. It uses computer software to understand input in the form of sentences using text or speech. NLU , which is natural language understanding, it enables human-computer interaction essentially. Let's talk about rule-based chatbots.

Rule-based chatbots are very basic and simple where you know what input to expect. So consider this example. where you have an input phrase, if the phrase is take call or answer call or accept call, there's an action taken to accept the call or pick up a call. Let's say this is on your phone And if none of them, then you know the phrase is not understood and there's no action taken. So this is a very basic rule-based chatbot example where there is a particular kind of phrases it understands, a particular input that it understands, and then it executes an action if it's able to identify the input. If not, then there's there's other either s message shown to the user or just no action taken at all. Now let's look at an implementation, a very basic implementation of a rule-based chatbot using NLTK and RegX.

So we import this data set from NLTK Corpus, WordNet, and then we import re. And then basically the high-level goal of this chatbot is to greet a user, ask for what they want. And if they want to know the store timings, tell them the store timings. And if they're done, just say bye. So there are three essential phases of this chatbot that it needs to understand is a simple hello, timings, and bye. But you know, hello can be said in so many other ways, like hi, hey, and bye can be said in so many other ways, timing can be said in so many other ways, like time, times, timing, timings. So there's all those combinations So what we use is we'd use WordNet

to get synonyms of these words, hello, timings, and bye. Once we have the synonyms uh we create a map of the intent to any cany keywords that associate with that. So for example, we know hello is great And anything that also conveys hello, like hi, hey, any synonyms also get saved to that map. So we make a map where we know what is a greeting, what is timings, and what is an exit Q, which is a buy So based on the synonyms that we get from WordNet, we build this intent map Once we have the intent map, uh the next step is to actually have a dictionary of responses. So

if the the intent is greeting then how do you want to interact? How do you want to reply to the user? If the intent of the user is to ask a timing, what do you want to say? And if the intent is to you know do nothing, just exit, what do you want to say So defining that and a last thing is if the user says something that is neither a greeting nor a bye nor timing, then you want to have a fallback response. In this case, I am unable to answer that. Please call this number for more information. And then timing, there's fixed timing of the store, so that is just a static response and everything else is static as well. So let's say we have this responses dictionary. The next thing is just to create a script that accepts user input.

identifies the intent. If it can match the intent, it is going to respond in the way we have defined the responses. And it's going to keep waiting for customer input unless the customer says bye or you know just exits out of there. So let's see how that works. We run the script and anyway the first message is welcome to Joe's store, how may I help you? User says hi, the response is hello, how can I help you? Hinter me store timings. So there are store timings that populate on the screen. Then do you sell cupcakes? Well that's something that our chatbot does not identify. So it's gonna go to the fallback response I'm unable to answer that. Please call this number for more information and then it exits out. So this is an example of a very basic rule-based chatbot.

That can only identify particular context and answer based on that that we have defined in a very very particular way. Now let's talk about AI-based chatbots. These are a lot more complex than the simple example that we just saw. Imagine this example where let's say we have Django Khan's pizza parlor and then you have a user called Joe and Joe is like, hey, I want to order a cheese pizza for pickup at 6 p. m. The chatboard goes like sure what toppings would you like? Garlic, chicken, jalapeno. Now the chatboard has everything that it needs to know to place that order. It knows when to pick it up. It knows it's a cheese pizza with all those three toppings. And it hits the database, calculates the total, presents it to the user, asks if they want anything else.

User says no, that's it. So all of this interaction without actually a human writing any of these messages, just using a chatbot, is an example of an AI-based chatbot, which is actually doing a lot more in the background than a very simple rule-based example that we saw. Let's see what we mean by that. Like what what is exactly complex about this? So when the user says they want to order a cheese pizza, in simplicity's sake, let's say the parlor, the pizza parlor only offers cheese pizza So we know that okay the intent of the user is to order pizza. What else do you need? To complete the order, you need the pizza, the intent to be able to make sure that the user wants to do this. When does the user want to pick it up? So

the pickup time is important. And to complete the order, you also need to know the toppings. So in this case, the user says 6 pm. So okay, that is identified automatically as a time of pickup And ordering a cheese pizza is identified as the intent of what the user wants to do. What is missing is topping. So the chatbot goes like sure, what toppings would you like to get the information that it doesn't have to complete the task at hand. So intent is essentially actions that your user wants to take. It's not as simple because your user may say it as pizza please or can I have a pizza or in so many different ways and it really depends on the user style of uh essentially talking uh so There are different ways this can be said, and you want to be able to identify all of that as the same intent, which is to order the pizza.

So essentially it's actions that the user wants to execute And then entities are more of the who, what, when, where. So things like any place, any date, times, in this example any pizza toppings, or if it was a more complex one, pizza crust, anything else That is actually a who, what, when or why where is basically entities. Let's consider building your own AI-based chatbot and then for that you would need various different things The first thing is to be able to identify the intent from a user's message. So in this case we just had one intent which could be ordering pizza. But let's say it's it could be several different things like just getting to know the timing of the store. or ordering pizza

or ordering a milkshake uh or you know all these different possible things that a business could be doing and wants to identify and wants the user to be able to execute using chatbots. So those are intents and there could be several ones. And there's there are ways to do that to build your own intent classification model. So like start with basic sentences as input for ordering pizza, like pizza please, can I have a pizza please? May I order a pizza? Can I have a cheese pizza? And even in cases where everything is in one message. So all the entities and the pizza. So can I have a chicken, pepperoni? and mushroom pizza for example. So essentially start with some basic user message and then go on to training a very basic bag of words classifier using the met the the first initial set of inputs that you have.

Other thing would be entity classifications, so what kind of things do you want to extract as entities? In this case it was pizza toppings and time of pickup So there are existing NER, which is named entity recognition models from Spacey and NLTK that can be used if your entities are what they offer. If not, you can also train your own entity classification models using existing spacey models and all of that as well. So you can train your own custom entities using that Furthermore, what you would need is once you know the intent and entities, you also need to be able to query your database fetch any information. So if you're out of pizza you need to be able to do that and if you have all the information you need to get the price of the pizza and give that to the user

In this case, you also need to know the state and context. Essentially, let's say state or context is the same thing here, but what is the current context of the conversation flow? So if you have asked the user, do you would you like anything else? And the user says no, you know that the no is for not needing anything else. And it's not for, you know, no, I don't want the order. So just to be able to know what a response really relates to, what's the context is really important. And then finally you want to also be able to generate a response that you can give back to user based on any information that you have fetched from any databases. for the product or service that the user is asking about. Initially because there is always lack of data, it's good to build, it's good to start off with a very basic model.

But as we go forward, you know, something like LSTM has shown has been a very popular one when there's more data and when you know you're not in your early stages anymore. Here's an example of a flow diagram which essentially once you have user message you have your intent and entities extracted, you know the state, which is also determined by the previous action that was taken, if any So this goes into the input of any database system or models that need to be queried or have this input run through to get your next action essentially what you need to do after that user message comes in. So that next action then goes and updates the state and then also generates your bots response and that's what the user sees. So these are the different components that are involved in actually building an AI-based chatbot

There are a lot of hosted solution options like Dialogflow by Google, there's Lex by Amazon, Lewis Chatbot by Amazon Azure, and then IBM Bots and Assistant. I have compared here the cost of each, the kind of channels, so essentially all of them are voice and text compatible, the integrations. and languages. So because you know it there are so many different languages that could be need needed to be integrated with the chatbot, there are different examples here for different rule Dialogflow is an NLU platform which is used to design and integrate a conversational user interface into mobile apps, web applications, devices, bots, interactive voice response system, and related users.

Bot building platforms go hand in hand with bot publishing platforms like Facebook, Twitter, Slack, you know, where where easy to build bots are launched for interactions essentially for businesses. Dialog flow, just from my experience, can provide easy and a quick way to create a custom and conversational AI bot. So it could be a good platform to start with, but Also, all the services do provide a free and a paid version and if you're subscribed or your business is subscribed to some other services already from Amazon or you know anything like AWS or IBM Cloud or Azure. then it would be better to just go with the chatbots that they offer. It may be an easier integration. Other options for chatbots could be something like Rasa, which is an open source machine learning framework.

And essentially what you can do is you can run it locally. Right, it's uh you don't need to have your data hosted anywhere or have any vendor logged in. Uh you can run as many as many instances as you want. You can create your own models and plug into Rasa. But the only thing is it's complex for beginners. So you need to really know about what you're doing and build a lot of things yourself rather than you know a hosted solutions. But that could really work for you if that's something that you're looking for. One other example of an open source tool in Python is called Chatterbot. Essentially, Chatterbot is a machine learning-based conversational dialog engine built in Python, which makes it possible to generate responses based on collections of known conversations.

The chatterboard uses a selection of machine learning algorithms to produce different types of responses. So essentially it has a Models where any user input is matched to existing known user sentences. And once it matches, it returns the response to that user message. So essentially it it matches the sentence itself. So it's a little bit different approach and if that works for you, you know that can work great. You can really build a basic example as well as shown on the screen where you just have very basic set of responses one after the other as in what you would want if a particular sentence is spoken as a response by the bot. But because of that, it also would not work well for some other use cases, but worth exploring if that's something that you're looking for.

Attached here is some reading resources which I found were very helpful. So if you want to know more about chatbots and other you know sequence-to-sequence based models uh which is more like encoder-decoder model that use LSTM which is long short-term memory uh for text generation uh you know from the training corpus. There are other articles and papers attached as well if you know you you're interested in further reading other solutions and other research that has been going on in the chatbot land Finally, I want to say thank you all very much for tuning in. It's been a wonderful experience being here. Hope to see you all next year