Chat RNIB Demo | How Wagtail CMS + AI helps people find what they need

Hi

Speaker 1: everyone, my name's Aidan Foreman and I'm the Director of Technology at the RNIB. So I'm just gonna move on. So the RNIB is the Royal National Institute for Blind People and it's the UK's leading site loss charity. More than two million people live in the UK with sight loss and every day another 250 people start to lose their sight, which is the equivalent to one every six minutes. The number of people in the UK with site loss is growing significantly and we estimate by 2050 this number will have more than doubled to four million. if the UK's current trends continue. And as an organization, our main purpose is to build the perfect country in the UK for everybody with sight loss. And our vision is a world where blind and partially

Speaker 1: sighted people can participate equitably. So as part of our service offering that we offer as the RNIB to our customers in the UK, we have a Sightless Advice Service which is there to support blind and partially sighted people as well as their friends and their families and their carers to help them navigate through their site loss journey effectively. So from the minute they're diagnosed, which you could imagine is quite a um emotional event. quite tricky in some of all the ways for thriving and being able to participate equally in society. So I would be really lucky that we've got a whole wealth really great trained professionals that are manning our services and in order to do that we have a knowledge base which is full of content.

Speaker 1: It's got around a hundred and sorry 1,150 articles. Which we have to keep updated constantly in order to provide that brilliant service. And at the moment, or previously, this was on a system internally called What's What. And this was just a standard knowledge base that we that we used to use. So from from that particular point in time, we were now starting to move over and were being powered by Wagtail for our public-facing website When ChatGPT was launched, and we identified really quickly that there was an opportunity potentially to look at an AI-powered interface for people to be able to type the question and get the answer that they wanted based on the kind of QA content. But rather than having to kind of browse through that content, being able to use the AI to maybe surface a better experience for our customers.

Speaker 1: We worked with Torchbox's innovation team to create a proof of concept, which we then wanted to roll out to our helpline advisors. which we've got some feedback around in a second, which we can share, which is really important. So just referencing kind of Brady's conversation around the legal compliance element. We're really keen to use this internally for a start before we went out offering any advice to our customers directly. As you can imagine, there would be lots of problems if the AI was to produce any kind of materials or answers that may be not given the right advice, especially if somebody was coming to R and IB in an emotional state. So what we did was we put this out as a proof of concept internally. So whenever whenever anybody rang out helpline They may have asked a question. We were then able to surface the answer for our helpline advisors to be able to give the best piece of advice

Speaker 1: that they could. And we got some really great pieces of feedback. So I'm not going to cover them all, but For example, this one talks about that it worked really well and it was much easier to use than their existing search. And this one talks about asking some really tricky questions and it was able to answer it in a really nice, good and tactful way, and in a really friendly manner too. And then there was a little, you know, a bit of contradictory feedback around the fact that it didn't always know all of the acronyms. And within the site loss sector, there are a number of acronyms, and we'll come on to that particular piece in a second. So just to contextualize this for everybody on the webinar, I'm going to provide a little demo here. So I'm going to jump into our What's What platform. Now just to be completely transparent, we've got a question here, which is what help is available for blind

Speaker 1: veterans? And now this this I know is an article that's within our knowledge base already. And of course that comes out of the top. But it also gives you a whole heap of other stuff, which for the end for the end user who's effectively providing that service to the customer, that can provide a very difficult experience when you're live and pressurized in that particular environment. And if I just throw another question in the mix, which is a little bit more difficult for it to answer in terms of what resources are available to help my person get back into work, and I ask that question. We 'll see that there's a myriad of results across across all sorts of different articles, which actually is really difficult for people to navigate and work round. So if we then go across into the chatter

Speaker 1: and a b tool, and I'm just going to log in to that now. We're provided with a completely different interface, which is really clean, really slick, also has a lot of accessibility features built into it, which again is all powered by and Wagtail, we were able to switch color modes and we're able to do all sorts of great accessibility features, which for our user base is really important. And if I if I dive in with asking exactly the same questions that I asked previously And I ask what help is available for blind veterans. What this does is it goes off and checks all of those great knowledge bases and it provides an answer in which it surfaces all the information it finds relevant. and brings that together in one place. That thus allowing us to be offering really world-class service really quickly.

Speaker 1: And we can then go to our sources and link back into any of the information we need. Now that was the more easy question. However, if we go back to the question which the what's what really struggled with, this is where the true power of the AI really comes in. So if I ask the question around how do we help a blind person get back into work, It uses all the knowledge that it has across all of those different articles and surfaces a really coherent answer with all of the great materials that we've got available to us within RNIP and again providing the different sources. So if I just dip into one of these sources, this is all stored. in the back end in Wagtail and it shows us all of the material that we've got available to us if we wanted to delve more into the the information available to us. So that's a quick demo of the solution. I'm now going to pass over to Dave who's going to talk to you about some of the behind-the-scenes parts that make this up

Speaker 2: Great, thank you Aidan. So I'm gonna share the back end, having Aidan having shown you uh the front end user experience. I'm going to talk you through how we've set up the Wagtail site to help answer those questions about site loss. And there are two key themes that I'd like to pull out for this. One is the transparency of the system and kind of what's going on inside it, and then also how we can keep on improving it and teaching it new stuff. So firstly, we wanted to make as much of the kind of the inner workings of the system open and visible to RNIB editors and administrators as possible. And I can show two of those from um from this this main page. Now I I'm logged into Wagtail, I can edit this main page.

Speaker 2: And from here As an admin, I can choose my AI model that's being used. That's the first thing. So we can at the moment I can easily switch between GPT-3 Turbo or GPT-3 but or we could add more models here including other non-open AI models in future. And secondly, is uh something that's come up a few times so far on on the webinar. And this is around the prompt. So the prompt that we're sending to OpenAI and something that came up in our user testing was that the prompt might need to change and we might want to experiment with different versions of it, tell it to answer questions in slightly different ways or with slightly different tone of voice. So rather than having that prompt managed in code

Speaker 2: we've made it into a snippet, which means that RNIB Wagtail editors can easily kind of change and iterate on this. So I'll just show you quickly what the what our kind of our current default is And this is yeah, this is our default prompt. It's very long. I'm not expecting you to read it all, but it's it that we're there's a lot in here around like how we want it to answer, what sources we want it to. use, what we want it to say if it doesn't know the answer, or if the question that's being asked is kind of is a bit irrelevant Because we found that early versions you could ask it questions like, who's the current president of the United States? And it would tell you the answer. So obviously we wanted to kind of to to limit what it would what it would do. As a quick quick demo of something, you know, obviously

Speaker 2: at the moment it's it's an internal facing tool thinking you know of a potential future where This could be an external facing tool. What might happen if we were asked if if someone was asking it a question and it didn't know the answer? So you might remember there were a few acronyms that Aidan shared I'm going to ask it, what does FFA mean at the moment? It doesn't know the answer to that question. And which is, you know, this this version is kind of fine while it's an internal facing tool, but you think, oh, maybe when this is an external facing tool actually it'd be it would be quite nice if we if we just could say something else uh when it didn't know the answer so i'm just going to show you quickly how easy you can just it's basically choosing another snippet So here is one I prepared earlier.

Speaker 2: If I edit this one, I'll show you the only difference here being that I've said, if you don't know the answer, then please just say, you know, please try a different question or you can call the RNIB helpline And yeah, that that would be um that would be an yeah a nice a nicer experience for someone if they were a member of the public asking this a question. So will refresh my page having republished it with that new snippet. Ask the same question. And you can say obviously instantly that you know the new snippet has led to this different behavior. So Yeah that that was a really quick demo of how you can change it, but obviously we could try different versions of the snippet to ask for a different tone of voice or a different length of answer or you know other variations. So that's the first thing around kind of how we want

Speaker 2: how we want it to be something that the RNIB team can kind of continue to improve the performance of. But the second thing is obviously teaching it something new. And one of the first things that we picked up from the testing, and I'm going to go back to that um that quote that Aidan mentioned earlier. People were actually asking it things that weren't even in the original source data. So we we were just using the source data from that what's what platform that Aidan mentioned. We went there and asked it what FFA is. It it didn't know. So then we thought, what uh what is there any other anything else that we could say what's an FFA procedure? I kind of it gets me some results, but it's just it's kind of uh matched on that keyword procedure. It's still it's yeah, there'sn't there's nothing in there about that topic.

Speaker 2: So obviously chat RNIB, as I just demoed, also doesn't know the answer. But with a little bit of help from Google and Wikipedia. . . I could find that it's actually a it is an acronym for a particular process, the kind of a testing process that happens for people with particular eye conditions. I'm not going to try to um to pronounce that word, but I'm just very quickly going to copy all of this Wikipedia content. and show you how we can then um add this to the knowledge base. So let's go over and teach chat RIB. I'm going to go back into the Wagtail backend where we have our chat homepage. As we drill down through the

Speaker 2: through the site's hierarchy, we have a questions index. And then we have all of our questions. So we've we've migrated all of these in from What's What , all of those questions that Aiden demonstrated. I'm just going to very quickly add a child page and this one is going to receive all of that Wikipedia content. I will just For now, give it that page title rather than phrasing that as a question. Publish that one. And now I'm going to go. Once that's published, go back to our chat interface. Now I'm going to see if I've managed to teach it. And I can go and ask what does FFA mean? Tell me the answer please And it's given me a quick summary and it's got a link to my page.

Speaker 2: But maybe more than just what does FFA mean? I might want to ask it more related questions. So what is an FFA procedure? And it 's also all of these, now that we have that that one piece of content in our in our database, it will start to tell me a bit more about it. But more than just obviously I don't just want to copy content from Wikipedia. I then had a bit of a dig around and found within RNIB's own content there's a PDF. on the public facing Wagtail website. And if I go down here and try to find, yeah, there's some content here in the PDF which I will copy in and create another Quest, so this is just another one. And this one was was content written for a kind of a public-facing leaflet that R and

Speaker 2: I B have. I'm just gonna maybe I'll just call this This like a bit of Lorosine Angiogram. I will publish this page. And now we've got two questions and answers pieces of content about that, about that kind of topic that until very recently um chat R and I B didn't know anything about. Now I'm going to ask it a slightly more complicated question. So what is it like to have an FFA procedure? And it you know to now it's going to tell me what what what's it going to be like to me or what might happen to me after an FFA procedure. So these these are the kind of questions now that we you know we in a in a couple of minutes there we've gone from a platform that didn't have a clue what this thing was with this with an acronym that one of our testers had

Speaker 2: um had picked up to now teaching it. from from some publicly available content, but also from the R<unk>IB content, what that thing is. So you can see it's kind of it's it's super iterate it's easy to iterate on on the knowledge base to teach more information, to change that prompt to keep making the answers more relevant. And I think we yeah, we've only just scratched the surface with what we can do. Obviously, you know, we've just kind of manually migrated some stuff from one R and I B platform and we've you know, had a look at some of those PDFs, but you know, there 's a huge amount of content there. So we've got some next steps. Um we'd really like to experiment with turning this from that single kind of question-answer interface into a multi-step conversation that preserves the context while someone's talking to it.

Speaker 2: And also, you know, we we're thinking about how this might enhance the regular on-site search experience by giving a natural language summary. to you know when people um use the search and I think yeah there are going to be really useful implementations of this kind of thing for any organizations that have a lot of kind of specific knowledge within uh within a Wagtail ecosystem that you know is available just in the web browser as normal. But yeah, this is a new way of accessing that. So yeah, really interested to see where it goes with RNIB, but also to see what what others are are doing with it. So I will wrap up there and we're really happy to take any questions.