AI in Wagtail: responsible innovation for content editors

Speaker 1: Hi all, I'm Tom Musha. I'm an independent Django and Wagtail developer. Please find me here. You want to chat about everything anytime. A couple of years ago, uh where as I wasn't hitting the mainstream, I built this experimental package to see how AI could be integrated into Wagtail. That's Wagtail AI. More recently, I've been working with Torchbox and the Wagtail core team to bring Wagtail AI out of this experimental era and start building the future of AI in Wagtail. We knew that to remain competitive in the open source CMS space, like I needed an answer to how we kind of do AI. We wanted to make sure our approach aligned with the project's values. To do that, we came up with a set of guiding principles for how we approach AI. I won't go through

Speaker 1: all these now. I'd just catch up on Thibaut's data white health yesterday, if you'd like more background. But of note here are two. No fencing and white tail core. So we want AI to be all optional and provided by third five packages. And model provider agnostics. We want to be able to give you the option to use whatever model or provider you want, be it open source or uh cloud provider models. These guided a kind of three-tier approach to how we uh built features for Wetale. This is our stack. At the top we have Wagtail, CMS you all know and love. We didn't want to add AR features to Wagtail. So rather than doing that, our focus was to build ways to interact with Wagtail's UI components in ways that other packages can leverage.

Speaker 1: like ways to extract data from previews, APIs to interact with field panels, and quite importantly for me, documentation on the front-end components for this great type doc here that I found really useful when building AI features This means that as a package developer, you can start adding custom behavior to the box admin a lot more easily than you could before. In the middle we have Wagtail AI, additional package, not bundled with Wagtail, that hooks into various Wagtail UI components to add AI features like image captioning and content feedback using LLMs. These are things that we think are the right use of AI in the CMS context. They're not about creating content, they're about enhancing the editor experience. We'll see more of that soon in Tejo. As we're building that, we realize that a lot of these low-level AI things that are needed are

Speaker 1: specific to Wagtail or Wagtail AI. So the bottom of the stack, we have a new package, Django AI Core. This is a developer-focused package designed to provide utilities for common AI tasks for Django stacks in general. Right now provides three main things. The first is a unified interface over different LLMs based on Mozilla's any LLM package. So you can use this to talk to models from Anthropic or OpenI or Hosted ones on Hug and Face or Alama , a huge variety of models we can use by the same interface. Secondly, it has the ability to create vector indexes of your content This means embedding your content using an AI model and being able to find semantically similar content to that.

Speaker 1: This is useful for things like related content widgets on websites. and natural language search and retrieval orbit regeneration or RAG applications like the one you see you can see here on the London Museum website. Finally, it has ways to register custom AI agents that are given in the URL and some parameters. And then you can call them from your main website or from in our case what. ai. Um these things together provide a good baseline for building any sort of um AI integration into your website. Our plans that add features like streaming support and rate limiting and importantly tools to log that AIUC across your applications. So in this mock-up you could see, you can see an example of how you might

Speaker 1: utilize the emissions and costs and queries. not only the Waccile AI usage, but across your entire website and how AI is being used. Django AI courses and active developments. It's been changing all the time right now, still in flux. I recommend you give it a go anyway and see for how far it takes you. Watch out. ai itself, we've just released version 3. 0 of the Wiktop. ai, so it's not been badly production use just yet We want to hear your feedback whenever we can. We'll keep adding features to Wagtail AI. We hope that we're providing like a more powerful way to hook it into Wagtail Core itself and a powerful foundation, the Django AI Core. we'll see a wider variety of packages in the middle there that you build uh but can provide ai

Speaker 1: features to to lagtail With that, I'll pass it over to Sage, who'll give you a deeper dive onto how and how we approached that, the features in Wagto AI.

Speaker 2: Alright, thank you Tom. Let me share my screen. Alright. Hope you can see me. screen. Alright, hey everyone, I'm Sage. I work at Torchbox as an engineer in our Wagtail product team and I've been working with Tom on improving the Wagtail AI path package. As mentioned by Tom just now and explained by Thibaut in his state-of-fact of talk yesterday, we have our AI guiding principles and I will refer to these points a few times. Throughout the talk. So the first point: no AI dependency in Wagtail Core. So just to be clear, everything that I'm about to show you is built in this separate Wagtail AI package

Speaker 2: So if you want to have nothing to do with AI in your White Tail, well there's nothing to do. Everything stays the same, so don't worry. Now with that out of the way, I'm going to show you some of the features we've been building for Whitetail AI. Today I'm going to start with our classic bakery demo. As a word of warning, this is very fresh from the oven, so I just released the new version this morning as Tom just showed you. So what you're about to see hasn't seen um production yet, but it's available to try. So right off the bat we have this new AI Actions button that shows up on the page. title. If I click this, I have the option to generate a title based on the page content.

Speaker 2: This works by prompting the LLM to generate a title using the page content that is displayed in the preview panel. And when it gives you the result, it doesn't just immediately replace the page title. But rather it gives you this suggestion. And you can either use it, regenerate it, or dismiss it. And if you so if you don't like what you are seeing, you can regenerate it. And in this case, um If we see the result, it's a bit biased to the existing title because I already filled it out. But if I clear it so that White Tail replaces it with the placeholder title

Speaker 2: let's try it again and now it gives me something that's completely new Let's say I like the result so I used it, but I still want to tweak it a bit. One thing I noticed is that LLMs tend to generate page titles with a subtitle separated by column like this And in this case I don't really want that, so I can just remove the remaining parts and keep the remaining parts. And so yeah, this is an example of the human, which in this case myself in the loop of the feature. And the next thing that works similarly is the generation of meta

Speaker 2: descriptions, which are used by search engines like Google. It's also used by social media and other platforms to generate previews. So when you share links, sometimes it gives you a preview of that link. So it's very useful to have this built out. And a good meta description gives you info on what to expect in the page So a good way to write it is by summarizing the page content that you already have and that you, the human, have already written. Now rather than having to summarize the content myself, I can ask AI to do it. So again, it's the same interface and I can use it and make changes as needed

Speaker 2: Now to the developer side of things, so warning, I'm going to show you some code, but for those who don't code, you don't have to understand it. You can just say to your developers, hey, I want Wartel AI, and they'll take it from there. So the code. Yeah to enable the feature you can import AI title field panel and AI description field panel from WhiteLai. panels and then you can replace the existing panels in the page model. You can do this for all page models by doing it for the base page model, but if you want to only enable the feature on specific page types, you can do so with your own page models.

Speaker 2: All right. Next up, images So Debo mentioned yesterday that a common unfortunate problem with websites out there is missing alt text in images. Alt text is the description of an image that can be read out loud by screen reader software to assist visually impaired users. With the latest release of Wirtail AI, you can generate image titles and descriptions as you upload an image. So I'll pick this image of a whole wheat bread. So by default, Warktel just takes the file name, which isn't great, but that's all the info it can get.

Speaker 2: But now that we have AI, I can have a better title for the image freshly baked whole grain bread and yeah I can use that and maybe I want to keep it short Because it's just to name the image, but since Wagtail now also has a default description text for images, Wagtail AI can also help you to generate the description. And this time it gives you a more descriptive text for the image, which can be suitable for an alt text. So yeah, freshly baked whole grain bread slices on a wooden cutting board. so on. Yeah it's way more detailed which is useful for alt text.

Speaker 2: So yeah I'm going to use that and hit upload Okay, so that was default descriptions for images, but in some cases, like this one. um the same image can be used in different contexts so it might be useful to use that context when describing the image And this is an image of I think it's a type of bread call called knipe bread. I probably mispronounced it. But it is apparently the most popular bread in Norway. Can someone confirm that? And since this page is talking about Nordic red culture, it could be more useful to add that context into the image alt

Speaker 2: text or caption. So let's see what the AI comes up with gives me slices of traditional Nordic rye bread on a wooden cutting board, showcasing the dense dark texture characteristic of historic Nordic baking traditions. So yeah, that sounds useful, although in this case for some reason it gives me the title with a quote around it. So let's see if it can remove that. Oh well, maybe I'll need to tweak the prompt later, but in this case I can use that and then remove it myself. So yeah, it's not perfect, but it is helpful. And now for developers,

Speaker 2: if you want to enable the feature, um there's this um Wagtail images image form-based setting and you can use Wagtail AI's built-in image form to add the necessary fields So this is for the image upload and image edit feature, but for the contextual alt block , sorry, contextual alt text block. You need to enable the AI image block, which can be used as a replacement for Whitetail's built-in image block. But if you also if you already have custom struct blocks with images and custom field for the alt text, you can use this decorator.

Speaker 2: Alright, so next up, content feedback. If we go to the checks panel, we now have the option to get content feedback So another thing that AI can be good at that's not generating AI slop is giving you feedback. So let's see what it comes up with. We have two types of feedback. One is feedback for the overall content, and the other are specific suggestions for improvement. And it gives me the option to show where the field is. And it also, if I hover on the suggestion, it gives it gives me reason why it thinks it's an impression improvement.

Speaker 2: So if I want to use the suggestion, I can just copy over some of the results here. For example, in this case it suggests breaking the sentence into two shorter sentences. And yeah, if you don't like the feedback, you can just click dismiss or you can also um clear everything and then get new feedback We understand that getting reproducible and consistent results with AI is practically impossible. So you might want to tweak the prompt and you can do so in the settings.

Speaker 2: So for example, um in here I already have my writing guidelines. This is not following any specific format, like I just wrote whatever I want in here. But let's say I want to add another A rule saying that I want to use plain English and target a reading age of nine. So I'm I'm going to clear the feedback and get new feedback. And let's see if that affects the results So yeah, breaks a long sentence into short, easy sentences.

Speaker 2: And Trying to see. Okay, so in this case, um simplifies the time phrase and uses present tense to match the reading level But again, you get different results every time. So yeah, it's it's good to experiment with the prompts to see to see if you can get what you want. And this feature is enabled by default, so you don't have to configure anything in the code, although we tried really hard to strike the balance between controlling the prompts directly from the admin versus in the code because we want you to fine-tune the prompts to get the desired result the the results

Speaker 2: But also we don't want the features to easily break if you misconfigure the prompts in the admin. And in the case of content feedback, we write the initial prompts in code along with telling the LLM to use. uh structured response format and then we add the prompt that you put in the settings just before we pass the page content. We're still experimenting better ways to do this, especially for the other features that you can also configure in the settings. But yeah, so if you have any feedback, please do let us know And final feature we have related pages suggestions. You can now get suggestions for related pages in a multiple chooser panel.

Speaker 2: So in this example, it gives me the pages that are similar using Django AI cores factor. indexes and um I can also get different suggestions and if it cannot find any more results I can restart from the beginning And to enable this feature, let's go to the code. You can replace Wagtail's multiple chooser panel with the one from Wagtail AI. and then you pointed to the vector index that you built with Django AI Core. The UI could still use some improvements, but we had to work within the limits of what's

Speaker 2: reasonable to override and extend from WhiteL itself. So as we're building the package, we're also improving Wikileak core itself by adding extension points, UI components, and documentation that can be useful for building custom features. So even if you don't want to use Whiteel AI, the development of the package can still benefit you, especially since I've been seeing a lot of talks from the other speakers showing the amazing stuff that they've built using Whiteel AI. As an admin framework. Alright, so in the talk title and our AI guidelines, we mentioned our responsible approach to AI and also being model and provider.

Speaker 2: agnostic. As Tom mentioned, we use Django AI Core with the any LLM package to allow different models and providers. And in the case of Wagtail AI, you can configure the providers in the settings. So along with the provider model and API key, you can also change the base URL for the API. This means you can use any cloud platform. For example, you can use Scaleway. They provide open source models and they have also produced extensive research in sustainability.

Speaker 2: which is clearly in line with our goals. So yeah, you can use them or you can even host your own model. um so this definitely gives you a lot of flexibility and not uh just sticking with the big names like open ai And in the case of the content feedback feature in particular, we also explored other options. Such as the built-in prompt API in Chrome, which runs directly in your browser. And we saw that it was indeed possible to run the model locally with reasonably good results However, given that the API is still experimental and it's currently only in Chrome, we decided to disable it for now.

Speaker 2: So if you're if you like what you're seeing, please check out Marktel AI version 3 and let us know what you think. That's it for me. Thanks.

Speaker 3: Thank you, Sage and this is this is really great to see. having been involved in those discussions product developments for a while. I know it's not too easy to get to this point, but uh I think where we're at right now is really solid. And um there's been lots of enthusiasm in the chat. I don't know if you've seen just yet. I'll take you from of the through some of the questions and comments in there. Uh folks, we're a bit ahead of schedule. There's definitely time for more questions and comments, I think. And we'll get started. With a question from Yusuf reflecting on the content generation,

Speaker 3: what if you want to generate the content based on the content of other pages?

Speaker 2: Right. At the moment it's not possible with the built-in feature, but it's It's very easy to build your custom agents using Django AI core so you can write your own code to fetch the actual pages that you want and then use that as the API for the front end. And of course, we're still looking to add more extension points within Waiktail itself so that it's even more so it's even easier for you to do that.

Speaker 3: Makes sense, thank you. Um Lewis asks, um what about translations? So As far as translations, Kuhn has mentioned his separate Wagtail Translate package. Is there anything inside the Wagtail AI package specifically that's about translations or other like page-wide transformations?

Speaker 1: Not yet. No, we're still um I think with the more powerful control we have over websile components, we can start doing that sort of thing, but it's not there at the moment.

Speaker 3: And do you think you that makes sense to add at some point or is it like separate concern, separate package? I think

Speaker 1: it probably makes sense. It might be a model package type thing, perhaps, like part of localized something rather than part of this battle.

Speaker 3: Okay, great, thank you. I think this was clarified, but maybe you'll want to say more. Claudia asked in the chat with screen readers stopping to read out text as 100 or so characters Does the AI cap the suggested alt text that it provides at that limit as well? How does that work?

Speaker 2: So by default it will respect the maximum length you said in the code but if you want you can tweak the prompt so that it uses a custom limit and from my testing it seems seems to work pretty well. Maybe not to the exact number, but if you tell it say do it in less in fewer than 10 words for example, it works pretty well.

Speaker 3: Okay, great. I think you've answered this live, but I'll uh give you opportunity to clarify. Uh Riz One asked about using multiple LLMs, how that's supported.

Speaker 1: Yeah, so there's two the two layers here. So say showed you how to configure in your Dragon settings uh different models for different features. Also you can override the agents that we use with your own agents to use custom models and whatever layer you want. There are quite a lot of options there for how you customize what's being used.

Speaker 3: Great. And yeah, even AI in the browser. I think I want to hear more about that sage if we have time. Moving on to the next question. You showed us how the prompts are written and it felt a bit tricky. Can you let the AI write the prompts or help you in writing the prompts? Can you tell us how you make the prompts to start with?

Speaker 2: Um so not directly in the admin at the moment, but I can say that AI helped me in writing the prompt. So I I just open uh separate AI like chat GPT for example and then I ask it to write me a style guide for a blog that is easy to read for example and then you can try putting that as the prompt that's used by Wagtail AI. But of course, um there's lots of experimenting needed if you want to get the desired results. And even then, it's not always reproducible every time. So yeah, I guess my suggestion is to experiment and see what happens

Speaker 3: I like that. I'm going through the questions and I think we'll have even time for more than we have already, but let's get going. Yusuf asks uh for the related pages suggestions uh what it's based on does it scan the content of the page or just a title and uh sendra also asks if it takes into account statistics on uh users of the page views analytics for those suggestions.

Speaker 1: So right now we take the entire preview content of the page, embed that in the same way that you embed the other pages and compare those two together. So it looks at the whole contents and finds similar based on contents. We don't look at other things right now like stats or views. It's just based on the similar content to what you're currently writing.

Speaker 3: Okay, makes sense. Sébastien asks, uh any feature in the pipe generates the whole page, generate a whole page from a summary?

Speaker 2: That's not something we want to do to be honest. Um yeah, I mean if you really want to do something like that, I guess it is possible if you create a custom agent but you'll have to write your own code for that and that's not something we're looking to add in Wagdle AI

Speaker 3: Do you want to say more about which features might be in the pipeline that we might actually want to do?

Speaker 2: At the moment I think it's still mostly refinements to the existing features. But I think you, Divo, might have other things in the pipeline that you want to share.

Speaker 3: Well putting me on the spot. Refinements, definitely. I mean, I think it's excellent work And I'm really happy it's outside the CMS, but I'm also really happy to see it existing. I think those features are the ones that we selected that made the most sense to do right now, you know? And maybe when we have more experience, maybe when there's more advanced concepts we can tackle them, but for now, it's a solid place to start. Okay, getting to tougher questions. I like those two. Gregory asks, is it possible to limit the access to the prompt settings and the related content features? to a particular site in a multi-site environment. So for example, related pages

Speaker 3: only having suggestions from the sites that the page is inside of.

Speaker 2: That's a good question. That's something that we identified that needs to be made in core itself first because At the moment, Wagel doesn't expose the in uh enough information about the current page that you're editing for other packages to hook into. So in the upcoming release, we're hoping to add uh to expose more of such metadata of the page so that pages sorry so that packages can make use of it

Speaker 3: Yeah, yeah, that's definitely, you know, um a package, but a package that we've made some changes in Wagtail Core to better support. Louise asks, I love that question. Would the feedback also notes when users upload terrible images? For example, an image that's just uh a table with text on it rather than actual visuals? Can the AI complain about accessibility and other matters of taste like this?

Speaker 1: The AI only gets your HTML or plain text from the page previewed right now, so it can't see images, unfortunately. I think we'd go to see how that would work, but at the moment it doesn't do that.

Speaker 3: Okay, great. I'd love to see that handled at some point, but yeah, tricky. Getting to some really tough questions here. Ahmad asks, is it possible to set permissions? to decide which user groups have access to those features.

Speaker 2: So at the moment no, but I think that's a sensible suggestion. Yeah. So as I said, we are looking for feedback. So if you have great ideas for the features or like refinements to existing features on how they are used by users feel free to put an uh write an issue on github and then we'll take it from there

Speaker 3: Yeah, great. Sébastien from our core team asks, what if from the fourth zen of Wagtail We consider a prompt something the developer wants frozen. Is it possible to version the prompt into code?

Speaker 1: We discussed that a lot during the development of this process. This is a tough balance between can you customize prompts in the admin or is it in code? And we do have a somewhat in-between solution right now where you can define prompts entirely in code, or you can define prompts that are powered by like agent settings or via that prompts dashboard that came in earlier versions of Web. ai. So we've got middle ground right now, sort of flying a little bit, but yes, you can do that if you want to do that.

Speaker 3: Yeah, I can confirm this has been a struggle. Hopefully the current balance is uh decent. Yusef asks, can I use function calling or structured output? I have different fields and I want to scan specific fields only or should I build my own class?

Speaker 2: Okay, so I haven't used function calling yet, but um and I haven't seen whether it's supported by any LLM, but I think if it is supported by any LM you can use uh you can build a custom agent that does that because for example for the content feedback feature I used a structured output to make sure that the LLM gives a JSON result that we can um used to render the results in a consistent way instead of just text. So yeah, I'm pretty sure it's possible if the NE LLN library gives gives you power to do that.

Speaker 1: Yeah the AI core agents are kind of designed to be just some code you can write with these based with these low-level sources. So in theory you could write a function calling loop like you wouldn't any other like agent somewhere else. in there. Right now it will block everything streamless streaming support there or async supports, but that's coming soon, hopefully. So you can write these background agents that do stuff and then come back to you later on with some useful information, hopefully.

Speaker 3: Great, we have a few more questions. I think we might have time for one more possibly. Ries1 asks, how does title generation work with different languages? If a title prompt generates often English titles, but you want to use it on a site that's not English, how does that work?

Speaker 2: Right, that that's a good question. So at the moment it will respect whatever language that is set on the rendered content. So if your page is already in a different language that's not English, well it will uh so our prompt will say that please do respect the language. language but if uh it's it's also a tricky balance because uh we want in the case of contact feedback for example we want to show the feedback in the language of the editor So there might be cases where the page content is in a different language than the admin language. And we tried to with

Speaker 2: uh in in the prompt we try to tell the LLM Make sure that you use this language for the suggestion and this language for the feedback. And Yeah, so far it works most of the time, but we've seen cases where it gets it it got the languages mixed up. So yeah, it's pretty much experimental

Speaker 3: Two last rapid fires for Sage. Is it a Steam Deck behind you?

Speaker 2: It is, yeah.

Speaker 3: Great. And Tom asks, is there anything stopping me from using Whiteel AI 3. 0 right now? Are the docs up to date?

Speaker 1: The docs are lagging a bit. The next thing we're going to work on, I think. But you can use it right now. And some good examples in the bakery demo repo for how you might use this in the real site.

Speaker 2: Yeah, that's something that I will do right after White Toast Face. So maybe it will be updated like early next week. So Yeah, just a heads up.

Speaker 3: Excellent. Thank you both. I'm gonna cut it right there and see you all at the next talk. Have a good rest of the day.