Creating connections between stories and objects using AI

Speaker 1: There we are. It is five o'clock. After your talk, Trish, we'll try and have a couple of minutes for some questions as well, but if we don't get through them all, then I will save them out and send them to you after so we can maybe follow up with everybody. But I am delighted to welcome Trish Thomas, who is Head of Digital Innovation at London Museum. a partner that we work really closely with at Torchbox for work including rebuilding their website on Wagtail as part of their digital transformation. And today Trish is going to share how the museum is using AI to create dynamic connections between stories and objects. So I will hand over to you, Trish.

Speaker 2: Thanks Lisa. Hello everyone. I'm just going to share my slides and go to some full screen action Okay, there we go. Hopefully you can see that okay. So it's really nice to be here with you today. I hope everyone is enjoying the sessions so far. Oh, doesn't want to move on. Yeah, I'm from Wonder Museum. And to give you some context, we are a social history museum. Which means we tell the stories of how London came to be.

Speaker 2: Our collection covers about 450,000 years of history. and it contains about seven million objects and they're spread across 22 different categories from Roman archaeology to contemporary fashion. So we have the largest collection in the world relating to any single urban centre and currently around about 146,000 of our objects are available to explore online. So today I'm going to talk to you about three ways in which London Museum is breaking down barriers to audience engagement on our Wagtail-powered website, mostly using AI. The first project is about how we are dynamically creating lateral connections between our objects and stories to stop dead-end

Speaker 2: journeys. The second is about how our new AI conversational search agent launched on our website this summer. And finally I'll share how we used AI to create alt text for our collections images online. So starting with lateral connections then. Back in 2022, we commissioned some digital audiences research to understand how people were engaging with our website and collections online, and we kept hearing these three blockers from audiences. People told us they just couldn't find a way in to Collections Online. They felt it was inaccessible. Particularly for non-specialist audiences. And if people did find ways in, they hit dead ends really quickly because there was no onward journey signposting to related

Speaker 2: content that would be similar to the stuff you were looking at. And some people find the concept of a museum collection just hard to relate to and they couldn't see how it was even relevant to their lives today. So, to address the blockers around ways into collections online and making the museum feel relevant to people's lives today We realized that creating blogs and stories only about objects in our collection, the stuff we owned, was significantly limit significantly limiting our reach. We're London Museum, our subject is not just what's in our collection, it is the subject of London as a whole, which means we can talk about pretty much anything related to historic or contemporary London.

Speaker 2: as long as we can then find ways to relate it back to our collections and that might might be lateral relationships rather than literal ones. So we made a plan to create a new editorial layer that sat on top of the collection. And this would be made up of a thousand London stories about famous or infamous London people, places, major events and subjects and there would be both historic and contemporary stories. And the stories needed to be really well optimized for organic search in order to bring in new audiences. So the list of stories we chose to prioritize Creating was based on the thousandmost search

Speaker 2: terms relating to London Next, to tackle the blocker around dead ends, we thought about ways to engage meaningful um, to encourage meaningful onward journeys for users on the website The obvious way to create dynamic related journeys would be using taxonomies, so that's structured data But our collections taxonomies are quite granular and very specific to our collection. So you can see here the way that they are categorized. Which is great for specialists like academics and researchers, but not so great for everyone else So we needed to create a new taxonomy that would be more relatable for non-specialist audiences of curious browsers because we knew that was the group that we wanted to target.

Speaker 2: So we looked at the topics covered in our planning list for these new 1000 London stories. And we based on this did a giant card sort. And out of that came this new folksonomy you can see here, which is made up of these broad terms less granular and more relatable in terms of ways to label and group our content for audiences. The groupings are also broad enough to allow us to create sideways lateral signposting to other related content Then we take these themes and we map them against our existing collections

Speaker 2: taxonomy to create a structure. that would build relationships between collections objects and editorial stories. So if you take this example of the musician Stormsey You might start your journey on a website on a page about him and that could take you on a variety of onward journeys, for example, to learn about London subcultures or immigration or political activism. So much more lateral than literal. Structured taxonomies, though, could get us so far with signposting related content, but using those alone meant a couple of limitations.

Speaker 2: Firstly, there were lots of gaps and inconsistencies in our collections data because tagging conventions for our collections have only really been judiciously applied in the last five to ten years. And secondly, the conventional model for using structured data. was likely to generate very literal onward journeys. So for example, we're looking at a page about a gold coin. The conventional related content approach would signpost to other gold coins we thought that was boring. We wanted to explore ways to signpost more laterally. So if you look a bit more closely at the description of this five guinea coin

Speaker 2: For example, a more lateral connections approach could take you on new journeys about royalty with Charles II or James II as the starting point. Or maybe one about the British Empire and colonialism with Guinea as a starting point. So all much more interesting than just looking at other gold coins. This was a light bulb moment, so we realized that we needed to explore powering connections using unstructured data, not just keywords, but also contextual relevance. This meant looking at a large language model solution. So that's where we turned next.

Speaker 2: So we are dynamically creating those lateral relationships in two ways in Wagtail. Firstly, we're using an OpenIAI solution to generate embeddings from our content, which are then stored in a vector index in Wagtail. Embeddings are just numerical representations of text that allow machines to find relationships in content. And then Wagtail uses the vector index to find similarity matches in content and suggest related content for onward journeys that are more rich and lateral. CMS editors can also choose to override this dynamically suggested content if they choose.

Speaker 2: The second solution we're using is a tool called YAKE, which is used to extract keywords from Wagtail content and relate these to our collections objects which live in a different database. Yake is also using the mapping between the folksonomy that we created for our editorial layer of A Thousand London Stories and the granular taxonomy to support these relationships. So now the journey between editorial stories and collection objects for users becomes seamless. And here's how it looks on the front end. So this is a very rich London story about the big anti-Brexit protest in 2019.

Speaker 2: The CMS is surfacing laterally related editorial story pages about gay rights and anti-capitalism, for example. And as a social history museum, we have lots of objects in our collection relating to protest and activism. So it's servicing some of these, not just about Brexit, but this one, for example, that's about a march. against cuts to public spending. These are lateral journeys. So that is the first example Next up I'm going to talk a bit about Clio. In July this year we launched Clio 1. 0, which is a new conversational search agent

Speaker 2: on our website. And given the rising popularity of chat agents like ChatGPT and Claude CLIO is an experiment to help users explore our collections and stories more intuitively using natural language. Our CLIO agent is powered by Anthropic's Clawed Haiku 3. 5 language model. We thought that was most appropriate for our needs after testing a number of different ones. And the chat is available on all of our collections object pages, our London stories pages and our blog pages. So if you ask a question, rather than being signposted to a list of links, you get the answer to your query in paragraphs drawn together.

Speaker 2: information from multiple sources and presenting it as one summary. It's a much more efficient and naturalistic way to find information, I think. So with the chat, a modal appears at the bottom right of the page and users can choose to minimize or maximise it. The interaction starts with a triage question and the user has to choose whether they want to ask about the page they're on or a wider question. In this example, the user is on a detailed blog page about the Great Fire of London and selecting Ask About This Page automatically generates the simplified summary of the page. And it also suggests a follow-up question to encourage the user to continue the conversation.

Speaker 2: If you choose to continue your conversation, the chat also retains the context from your previous questions to improve the quality of its answers to your follow-ups. So here's an example of how it responds to my follow-up requestion and in this way it supports back and forth dialogue so it's more engaging for the user. And if I go back and choose the other triage option to ask about anything, I can completely change the subject while remaining on this page about the Great Fire. and it's using the triage to reset the context for the data it uses to answer the question. So

Speaker 2: the chat only uses London Museum's trusted data to answer questions. And early insights are very interesting. So Cleo is being very well used. It's most popular on our collections pages. Most users choose it to ask about the current page in that triage step and on average conversations include four questions going back and forth but some are longer. And the longest conversation we've seen so far had a whopping 98 interactions and it was uh it went deep um but it was focused around all sorts of themes relating to Irish identity. and nationalism.

Speaker 2: The data is absolutely fascinating and the museum will be able to use the questions that people are asking, it's all anonymized data, but we'll be able to use the kind of trends from that to spot the areas of interest. for our audience that we may be missing currently or that are particularly popular so that we might this might feed into exhibition planning or it might feed into content planning for blogs and stories for example. So it's super useful to us as well as to our users. So that is Cleo my second project And finally, I wanted to share how we're using AI to generate alt

Speaker 2: text for over a hundred thousand collections. images on a website to improve um accessibility overall. And of course generating alt text for that many images would have been an impossible human task. So it seemed like a very natural solution for us to start looking at how AI could help. So we looked at a number of image recognition, text generation solutions And in the end we chose a solution called Alt Text AI, which does what it says on the tin, and it is powered by Microsoft's Azure Vision service, which uses image recognition. So

Speaker 2: we worked with TorchBox and our other partner agencies and we created a script. That interrogates our digital asset management system image records and populates those with an AI-generated alt text every 15 minutes. One of the things that we really liked about alt-text AI as a solution is that you can Set the level of descriptive detail in the alt text and there are three options for doing this. So the most basic level is called matter-of-fact. And we found this option was quite cold in its tone and sometimes lacked detail, which didn't feel an appropriate fit for our museum.

Speaker 2: At the other end of the spectrum, there's an option called detailed, which has a much more engaging, friendly tone, but tends to lean towards flowery language, and we thought the risk Here might be that the AI starts to pad out the alt text using non-factual information And the option between those two is called concise and uh it sits with the right level of warmth in the tone and the right level of detail in the description for our needs. That is the one that we chose to go with concise, but you can see two examples of the text generated against images, example images here.

Speaker 2: So as you can imagine, we had lots of concerns about allowing machines to come up with descriptions for museum imagery. Generally, we were worried about three things. accuracy, so could the machine even recognise what it was looking at? We have lots of photographs of shards of old pottery in our archaeology collection, for example. We were worried about contextual interpretation, so could machines understand the nuances in an image And we were worried about language sensitivity. So when describing sensitive images, would machines know to use acceptable language? So we needed to get into the weeds and run some more tests

Speaker 2: to estimate the error margins we might be looking at and how the AI performed. So we chose a selection of problematic images where there was sensitive subject matter of different types so we could see how the AI interpreted these and what language it used and the results were surprisingly impressive So in this example, the image is showing an anti-race demonstration. There are all sorts of sensitivities around the language you would use to describe this. around race and subject matter, but it handles this in a really neutral, sensitive way Then we tested some images that you might call adult content.

Speaker 2: We have really big uh photography collections. And the AI was sophisticated enough to handle sensitivities around gender, for example, in the way it described images like this. So It it here it refers to the individual as a person as opposed to assuming the gender or using descriptive terms we might now consider to be offensive language. So it it handled all sorts of sensitive imagery really well. But we knew it wouldn't get the alt text completely correct 100% of the time. So we needed to just prove that the margin of error was low enough to be acceptable compared to the alternative

Speaker 2: of having no alt text for our collections images. And I'm pleased to report the error margin is low. So so far we basically see two types of error Firstly, there are errors where it doesn't fully recognise what it's looking at. So here's how it interpretes a photo interprets a photograph of the Queen on a train and it refers to her as an older woman in a hat and gloves Which is technically correct, but of course you'd want it to recognise that it was the Queen and name her. And here's another one. It thinks it's seeing a gold bracelet, but it's actually looking at the frame of a gold purse from above, photographed from above and I would argue that a human would probably also have made the same mistake with this one. And the second types of errors, for some reason, the AI doesn't read the image.

Speaker 2: It returns an alt text that says something like a plain white image with a few black spots. The good thing is because this is a specific recurring error, it's easy to detect and it occurs for about a hundred of the images that were processed. So there are two processes for correcting errors. For the detectable errors, it's easy to generate a list of affected images and manually update the text. And for the other two examples, harder to detect examples like that or errors like that. So we rely on spot checks and user reporting. and we built into every collections object page a form where users are able to report errors. And then we built a simple correction interface which pulls in the object image, the AI

Speaker 2: alt text and the physical description from our collections database and a human manually then corrects and overrides the AI alt text. And that is my three examples. Thanks for listening.

Speaker 1: Brilliant. Thank you very much, Trish. That was really insightful and we do have some questions for you. First of all, it'd be interesting to know have you got any insights now about how users are moving through those AI-generated user journeys that you've introduced.

Speaker 2: Uh that's kind of ongoing. So we're about to commission, so we're about to recommission that research that we did. So the objectives were around creating longer, more engaged journeys, but also around reaching some of the audiences that we weren't reaching before due to relevance, you know. So the the research that will be repeated is to demonstrate that we met the KPIs of reaching more. uh people under the age of 45 and um who are more ethnically diverse than the museum's current mix. and that they stay on the site for longer and that they are more engaged. The stats bear that out, but we would like to repeat the research to prove it.

Speaker 1: Brilliant, thank you. And Patricia would like to know how did you go about choosing an AI search agent?

Speaker 2: Oh, so as part of the process that we worked through with Torchbox We looked at a few and Torchbox built us a switcher for want of a better word which allowed us to, when we were prototyping the model for Clio , We were able to switch between a bunch of different agents that we wanted to look at. So we looked at some of the more larger and more powerful clawed models like Sonnet We looked at the smaller clawed models from Anthropic like Haiku 3. 5, which we ended up going with, but we also looked at ChatGPT as well and different kind of models within that. So the switcher allowed us to

Speaker 2: test different models against the same data set and see which gave the best quality results for our needs And the best performer, taking into account the cost as well of the kind of token usage that you pay for commercial models, best performer for us was Claude Heike.

Speaker 1: Brilliant, thank you. And Umer asks, when using the Ask About This Page option, did you choose to provide images, any PDFs linked in addition to the text on the page as context to the large language model?

Speaker 2: Um we didn't. So the m London Museum ha London Museum's AI policy. Um Requires that we kind of use the most use AI in the most sustainable ways and we consider the impact on our emissions as an organization. So we have chosen to use a text-only data for this for our model. And for those reasons we don't uh surface images or uh documents like PDFs

Speaker 1: Brilliant. And if it's working, then that's great. So now I've got Ahmed. What was the name of the AI tool that was used to determine related pages based on certain keywords? Is that the Claude?

Speaker 2: Uh that was Yake. So uh OpenAI was used to generate the vector index and then there's another solution called Yake, which is the one that that reads content in Wagtail pages and extracts keywords and also can look at the related relationships between our folksonomy in Wagtail and the taxonomy in our collections database

Speaker 1: Brilliant. Okay, I think when we share these videos on YouTube, we will include some resources and links to things that people have referenced as well so that you can find them all. And finally, Ralph asked, is there a threshold at which the tool relating to the alt text will report a level of uncertainty that leads to human intervention?

Speaker 2: There isn't, no. So it runs on an automated basis driven by the script. And we trust it enough based on our testing to allow those alt texts to go live. And then we're reliant on our own spot checks and users reporting errors. But yeah, it doesn't ever It doesn't ever get to the stage when it's running that script where it says, I I can't return an answer for this and need human intervention.

Speaker 1: Brilliant. Thank you very much. Thank you, Trish, and thank you everybody for your questions.