llms.txt deep dive

Speaker 1: You ready with Okazoo?

Speaker 2: I don't know.

Speaker 1: You have to do it.

Speaker 2: Does it go first or do you go first?

Speaker 1: Uh we're live.

Speaker 2: Oh

Speaker 1: Well live already, Sage. Good to see you.

Speaker 2: You

Speaker 1: ready with your kazoo now?

Speaker 2: Yeah, maybe.

Speaker 1: Give us an intro.

Speaker 2: Mm-hmm. I'll I'll use a a different instrument next time.

Speaker 1: Oh, it's great. Love this intro. Welcome to this week's episode of the Waxtail Review, our very informal podcast. um sage it's good to have you here

Speaker 2: well it's good to see you

Speaker 1: yeah today we have um Special topic I prepared, so hopefully you still feel that way whilst we've been through it.

Speaker 2: Well, I mean every topic here is special.

Speaker 1: Oh that's so kind I thought it would be nice to do more of a deep dive into something I've worked on recently, which is lms. txt in the Wagtail docs And the reason that I uh you know I'm feeling ambivalent about it is uh it definitely has the promise to be very helpful, but also to mean that people um look at our docs less and less and spend more time with their AI robots instead. So it's bittersweet. How do you feel about lmst. txt? Have you used it yet?

Speaker 2: I personally haven't, but I've seen other people using it. And it does seem to improve the results that you're getting out of the LLMs. So if it means less hallucination from AI. I think it would be good. But also I I I saw the news um from Telwyn because uh where they experienced like a massive drop in the in the in the traffic to their docks that ended up in them having layoffs so That's really sad.

Speaker 1: Yeah. Yeah. I remember seeing this pull request opened for so long because um Well we use Tailwind really happy with it and they have a very specific triage process for feature requests where most things are discussed in GitHub discussions first and only brought to issues and PRs once they are kind of really ready so you could really tell someone something was going on with it um taking so long and um yeah I mean you know it's a big um stress test at the very least for open source um the change in how people uh engage with those projects engage with the docs um yeah i don't know if we see similar um

Speaker 1: change shifts for wagtail because uh analytics are like pretty basic like we have things that are not too invasive basically so we don't necessarily have access to analytics data over years and years and um I do expect something similar is happening though. Is it a problem for us?

Speaker 2: Um well given that we don't provide Wagtail as a as a well well Telwin is not really provided as a as a paid product, but they do sell Telwin UI. But we don't have something similar with Wagtail. So I guess we're in a different position.

Speaker 1: Yeah. Yeah, we don't rely on um income from the vast majority of our users. We do hope to sustain development with sponsorships, but it's not the kind of sponsorships you tend to get by people looking at documentation, it's more relationships and needs of big organizations.

Speaker 2: Yep.

Speaker 1: Yeah, um hope hopefully it works out for them. Hopefully they find a ways to change their business model for us. I think you know it's quite a promising format because uh we do want our docs to be as accessible as possible. Uh which is why we've invested into it. Yeah.

Speaker 2: Yeah, and we want Magnol to be as easy to use as possible.

Speaker 1: Yeah. Yeah. Um so yeah, we worked on this in the last uh latest release of Fragtail, which is due to come out in um uh about 10 days now and uh just because you know those files are tied to the documentation they're actually already available and ready for people to test So for the docs, you the developer docs, sorry, you you just have to know where to find the URL to those files. So you can access them. Or I think I added a note in the in the support page of the docs, which I'm not sure is the right place, but at least you know there's something written somewhere. that we can point people to.

Speaker 1: And for the guide websites, I built this new copy page. It's kind of a not an established pattern but emerging pattern across lots of subdocumentation websites where there's links to all the different versions. Um I don't know if you've seen this much on like AI docs sites where there's like Ask Claude about this in the

Speaker 2: Oh yeah, like a button on the documentation. Yeah, I think I've seen that. Yeah, sometimes sometimes it's also integrated in the search bar So instead of being a regular search, it also works as a prompt input.

Speaker 1: Yeah. Yeah, yeah. It's interesting you you bring up search because to me that's kind of um One of the killer features of those LMS. txt is just um AI chats are quite good if you give them the right information at searching through it and finding you the right answer compared to our site search which is really basic um yeah so um we did that because because useful and um Not gonna read through my whole blog post, but I guess a big hope for us is that uh it's encouraging people to use open source models and smaller models

Speaker 1: rather than focus on the one or two or three top providers. Um we I think know that Claude and Gemini and so on, they crawl the site's HTML on a daily basis. They have no problem extracting the contents from the HTML. But if we take some time to make the content more readily available, it makes it much easier to use with LMs that do not crawl your content like you know local model. or an open source model uh hosted in an independent provider. So I thought we'd try that together today, if that sounds fun to you,

Speaker 1: and see what happens

Speaker 2: Let's see it.

Speaker 1: Yeah. Uh I thought I'd uh give a quick plug to one of those providers first, which is the one we use for our AI experimentations in uh in Wagtail Core. Um I think there's like a lot of such providers out there. There's not necessarily um This one being like much better than the others, but it's definitely like exactly the right fit for us. So scaleway they are um based in Europe. the data center for their AI services is in Paris and we selected um them as a provider because um The open source models they provide, they're the right fit for us in terms of capabilities

Speaker 1: and I guess uh ethos, values, alignment. Um, and the data center being in Paris, it's close to as low carbon footprint as it gets, which really matters for us. Um so yeah today I thought we could try um Mistral Small, which is a 24 billion parameter model , context window of 128k. For context the guide websites docs like the full docs that's somewhere between 25 and 50,000 tokens and the uh developer docs that's about 10 times stats so 250 to 500 000 tokens

Speaker 1: um I don't know if you spent much time in your developer life sage dealing with different context windows and optimizing token counts to fit and so on.

Speaker 2: Not that much, but I do know that once you start having a long history of from it becomes less reliable.

Speaker 1: Yeah, yeah, yeah. Yeah. I I think when you reach the top of the context window, it already starts to suffer before you're at the threshold and then when you're after it loses the context. Thanks even more. Yeah, so I guess that's one of the big big things we wanted to do here is making sure that our uh LM files fit in smaller context windows and potentially producing multiple versions of the file so they're more likely to fit in those windows. Um yeah, and 128k, that's kind of um higher end for open source models that are kind of in smaller parameter sizes

Speaker 1: then there are bigger models that uh have much bigger context windows so quen 3 here 235 billion parameters um this active 22b so that means that when it's actually answering um you know your questions it has an architecture where there's only a smaller subset of those 235 that are actually being used. That's about double the context window. Anyway, let's see how it goes. I have a specific system for this in place and uh I haven't demoed this to anyone yet, so it's a bit of uh just hoping that it works well enough to be understandable.

Speaker 1: Um I'm gonna switch my screen sharing to another tool. This is my IDE of the moment. I'm in cursor, but really any other IDE could do just as well. Um I'm gonna make sure we have as much space as possible. Um how do I close this side? Yeah And we're gonna open Prompt Foom Have you heard of Prompt Foods Age?

Speaker 2: I have heard of it. Never used it though.

Speaker 1: Okay. I think we'll do the deep dive approach where I just give you a demo and we see how it works.

Speaker 2: Okay.

Speaker 1: And maybe we look at the docs later. So Prompt Foo is um a tool to to um make uh evals um I don't know what the proper term is, eval, eval suits, evaluations, I've heard all those things. I guess if I have to summarize, it's uh automated tests for AI prompts and AI agents. Which you know somewhat like unit of integration tests, but the assertions you can make have to be a bit more lenient to AI output being so variable. So top of the file here I give it which prompts I like it to use when answering my questions.

Speaker 1: Then providers, just which AI models, which AI systems we're using. So in this case, we have three providers. The Mistral model I just mentioned in Scaleway bigger GPT OSS 120B model and then the heavyweight uh Cloud Opus 4. 5 And I think the goal here is just to be clear is we expect we can uh with the right prompting, with the right context management, get the same all. better results with Mistral than you can get with a super heavyweight model like Claude Opus Yeah, and once you have that set up,

Speaker 1: you give it different questions to answer. So here there is three variables, which are the three different files. and um those are reused across all tests scenarios and then in the different test scenarios there's this user message there's gonna be the actual question that we ask the AI. So how do I create a new page? Kind of the bare bones. This is the minimum requirement for most factel users. They want to know how you do that. unsupported feature. So how do I recover a document I just deleted? It's a good test of how much DA I gonna pretend. to know more than it should

Speaker 1: or to do things that are impossible. Yeah, and then the assertions. Maybe I'll explain this with the results. So let's um view the results of an eval I ran previously. And yeah I switched to cursor because it has this uh browser window in the IDE feature that I thought could be quite nice for it. um yeah so here we're inside the the results view after the checks have run and um the idea here is that we'll be able to interpret uh which uh AI models did well with this or that configuration of the of the prompting.

Speaker 1: I think just to save a bit of space I might um how do I do that again? There is a way to hide the variables that I forgot about. Just make them smaller like that.

Speaker 2: Columns at the top, maybe? One of those options. Uh

Speaker 1: where do you see that?

Speaker 2: Um Up from where so columns, yeah.

Speaker 1: Thank you. Table, settings, and columns, exactly. There's always side by side and Yeah. So we'll just look at the question and then we'll look at the scenarios in the output. So how do I create a new page? I have a very small model here that I had you've been using in the past. on my own computer then we have Mistral Small GPT OSS and then Opus So we can see if I scroll that quickly top to bottom on the right hand side, Opus passes all of our checks. So it seems like it's doing really well and basically our job if we want to optimize for smaller models is to look at why the smaller ones are failing and then make sure to adjust.

Speaker 1: So I'll give you an example. If we look at the GPT-OSS answer for this question, here my assertion was based on the regular expression. So it looks like the contents have either this or that in them. And so what I try to do here is make sure that when the AI answers the questions It only refers to the real content of the real user guide, doesn't make up URLs on the spots. And if it does. . That's bad. So you would expect that with more refined prompting, we could get this out. Like maybe we could prompt it with a list of URLs and say like your answer better be mentioning one of these and nothing else.

Speaker 1: Um yeah if we look at um Another failure from another smaller model. So this was a regular expression test. This one is called it's a form of LM grading. So here you give your the answer from the AI to another LLM and you ask it to judge whether it meets this or that criteria. So here what I said is uh it's really important that uh when we tell people how Wagtail supports image resizing, that we focus on the features in the CMS rather than what developer can do. So here I think I my prompt

Speaker 1: was saying like oh it's important to um avoid mentioning this as a dev make sure that in the CMS it's clear it's only about um uh setting focal areas. Um supports image resizing generates multiple versions of it at different sizes That's kind of correct. How to set the focal area of an image which can affect how the image is scrapped and I mean that looks like a fine answer to me up to this point. So here maybe there is a case for our tests to be more lenient for this scenario. And yeah, this yeah is definitely a type of like developer focused information that's for a dev that's really important

Speaker 1: for CMS users. They don't really need to know about this. Do you want to try it with a new question, Sage?

Speaker 2: Oh. Are we also testing with the developer documentation or?

Speaker 1: We can do. I will need to adjust my setup a fair bit, but we could try. Give me a CMS user question first, and then I'll do the setup for a developer question.

Speaker 2: Okay um maybe how do I hide a column in in the page explorer or or image listing or one of those I guess

Speaker 1: You can do that?

Speaker 2: Well, we wanna see whether the other makes it up.

Speaker 1: Okay, so uh I'll go to my list of tests then and at the bottom. I think description we can just write uh Sage 's trick question So how do I add a column?

Speaker 2: Add or hide

Speaker 1: In the page explorer you were saying?

Speaker 2: Yep.

Speaker 1: How did you want to test that it answers correctly?

Speaker 2: Oh, that's a good question.

Speaker 1: So we can do like

Speaker 2: cannot

Speaker 1: yeah we can we can we can um not I'm not the biggest fan of using LMs to make sure that LMs don't LLM but we can we can try as a st as a start of point.

Speaker 2: Does it give you the output as well so that you can check? Right. So so if you have like an assertion whether the output contains the word cannot or can't.

Speaker 1: Yeah, yeah, yeah.

Speaker 2: Yeah, then you can but you can verify the actual output. can see

Speaker 1: yeah

Speaker 2: okay

Speaker 1: this is not supported let's start like that and yeah you know the types of assertions that I like to write is like oh it contains this very specific um word like for example how do i create a new page or it should contain add child page page type stuff like that um but we can start simple like this see what answer just come through and adjust later.

Speaker 2: By the way, I'm not sure the wordings that we use in the user guide, is it explorer or is it listings? Okay.

Speaker 1: Um Yeah, there's definitely a good question there which is that when when you do this you're testing many things You're testing the AI's ability to insert the docs, you're testing your docs content as well, and you're also testing your prompting. So You definitely need to look at the results carefully and be sure to, I guess, impar what what your findings map them to the corresponding place to improve. Um anyway, uh yeah, so we've uh created our scenario. I might just comment out the other ones so there's a bit less um

Speaker 1: noise in the outputs. Um

Speaker 2: Oh, that's a funny type of

Speaker 1: prompt fool. Yep, so um it's derived six test cases from your one question because um There is three different AI models and two different prompts. So it's gonna try to answer this with my smaller abridged file with the full lms. txt file across all three models Past two, four failed. Hmm, interesting. Um so we'll go back to

Speaker 1: our Evals results and um Probably using it wrong, but Um oh yeah okay uh did I make a mistake with the configuration there I don't know why the slash isn't coming through.

Speaker 2: Have you saved?

Speaker 1: Is it just that?

Speaker 2: I don't know maybe.

Speaker 1: Am I the fool?

Speaker 2: What?

Speaker 1: I'm I am the fool Let's give it another chance.

Speaker 2: Do you think it interprets the slash and question mark as something like a slash command or something?

Speaker 1: No, I think it's just me mistyping. I forgot to also remove the the default assertions I had for my other tests. So I just I think it's worth looking at this to be clear that it's what we want. So I had three default assertions. One, it should contain URLs to guide websites So this is because I expect the AI. In a prompt side, I expect it to always cite its sources Two the North Regex one I mentioned and three should provide a clear step-by-step guide or a clear summary. So I think here we should also amend and said

Speaker 1: Or explain if something is not possible And then we'll make sure that our actual failures match this scenario a bit more I only had one trick gotcha question so far in my test suite Okay, three f passed three failed. That feels like we'll So Cloud Opus doing well with both types of files we provide.

Speaker 1: Chat GPT OSS has one fail and Missroll has two fails. Okay, so Mistral with the abridged file says there's a column button. The column configuration feature in the pages section

Speaker 2: Click the columns button in the top right corner of the page listing next to the search bar.

Speaker 1: Save as default, optional. I mean this is lovely, but this is completely False? I'm surprised that it would lie to your face so much even though it clearly is citing the right pages Okay, so here complete bullshit conclusion we should refine our prompts so it's less risk to hallucinate Next one with the full file. Yeah, same.

Speaker 2: But what if you were a user and you ask it a question like You said we we should improve the prompt so that it doesn't give you incorrect results. But as a user, you would just write the prompt, right?

Speaker 1: Oh, okay, yeah.

Speaker 2: Yeah

Speaker 1: So I think um if if we expect there is like common misconceptions we can address them in the docs. So for example for uh um documents deleted documents recovery our docs explicitly say like this thing is not supported because it's quite crucial to know that you don't have this so yeah we have like option one update the docs to dispel common issues um option two um can i just open any url in here or is it gonna I don't use browsers integrated into IDEs too much. Yeah, I can. Okay. Um So this copy prompt

Speaker 1: thing that I have is my attempt at encouraging people to use um like a good default of a prompt if that makes sense So the idea here would be that you could copy paste this into your um AI of choice and it would have this default set of instructions that guide the AI towards hallucinating, yes. So here it says when a topic is not covered in or falls outside the CMS user scope, clearly state this without speculation. or developer level details. If we look at the results we got. This is definitely speculation

Speaker 1: to be polite. And this is definitely developer details So Mistrol definitely failing. Need better prompts and hopefully we can get it to work better. ChatGPT OSS A lot of thinking. Why does the thinking stop Okay, I think that's probably right about where we need to be looking. Hey Baptiste.

Speaker 2: Hello

Speaker 1: Welcome to our panel discussion. We're looking at the adams. txt format and uh testing the AIs with trick questions to see if they're following our instructions to answer questions about Wagtail based on the docs. That looks fun Chen requires developer to modify the project's Django admin configuration or create a custom view. Close but no cigar Use the exin columns to sort or filter pages, click the column header to sort, use bulk action checkboxes. Yeah, well there's definitely some elements that are correct in here

Speaker 1: but yeah again probably room for the prompt to be to direct beep direct the AI more and um for the prompt to um make sure that we refer to the whacked admin only. Let's look at Claude. What does Cloud think It's not something that CMS users can configure directly. What users can do That all seems very correct.

Speaker 1: Yeah, that's interesting that it does this. So there's um There's a big decision factor when we create those files. I mentioned earlier that, you know, it's important that um They are optimized to be smaller, to fit in the context that smaller open source models have, so that hopefully we get good results with less energy use. And release notes, it's something that's happened only and that kind of loses relevance over time. So it's always very interesting to me to see which questions might only be answerable with the contents of the release notes. So I mentioned to you like having this abridged and the full file. One difference is the abridged one.

Speaker 1: I removed all the release nodes to have a smaller file. And um I think this is even more important if you look at adms. txt versions of developer docs. That you decide, okay, do the do the release notes from Wacta versions five years ago, are they like something that we expect people to be using to answer their Wacted questions? Probably not, right? Do you look at release notes from like that that while ago to check?

Speaker 2: Well, I personally don't, but I can imagine if you're upgrading a really old wagtail. instance you might want to have answers from

Speaker 3: I've also in the past I think when searching for I can't remember what but the release release notes do pop up uh And it has been useful. I think Wactails release notes are quite good. There's obviously a lot of effort going to them. And I think sometimes it it it would help me uh understanding how a feature was used because oftentimes the release note explains how to change from the old way of doing things to the new way. And so as a side effect they do explain how to use things. And sometimes it explains them in a slightly different way than the reference documentation does, and it's enough to uh give uh like I don't know something quicks.

Speaker 3: In contrast, I think Django release notes often appear very high in the search. I think we've never figured out how to deprioritize them. And for me, they're rarely useful. They're the release notes for Django are really good at telling you what has changed, but I don't find any use for them for anything else Whereas Wagtail, I think you can get some some good information out of them. I don't know if that's a positive or negative. I feel that You know, your release notes, I don't think, should explain things that is not explained by the documentation, but I don't know.

Speaker 2: Yeah, we do have a problem of features being documented in release notes because we never actually documented the feature but Sometimes we go out of our way. So we end up writing undocumented things in the upgrade considerations of our release notes. So yeah

Speaker 1: Yeah, it's a super important design goal for us to avoid documentation that's only in the release notes and nowhere else. So I think I'll create an issue for us to actually write it somewhere. I feel like you know the really is not definitely meant to be a point in time most useful at the point point of the release and at the point of you considering the upgrade. But hopefully if your site has been upgraded you should never need to look at them ever again after Um Baptiste, do you want to try and trick the AI with a particularly devious question that might or might not be explained in our documentation?

Speaker 3: Oh , try to I I'm just coming off of a project where I did some quite heavy customization of the of the admin where I I had to give up a few times and go and read the source code rather than refer to the documentation. But uh What did I want to change that was hard? Ooh, I wanted to change the columns that were listed in a chooser. I had a custom chooser and I wanted to um

Speaker 1: so as a developer, yeah. Okay, um let's see if we can do that and uh that'll be fun for us to do together. So this this configuration for this piece of software right now it's set up specifically for the CMS user docs, but I don't think it will take us too much work to do a version of this that is for developer docs. And that'll get me a chance to teach you all how to do this, which I think you know definitely people who drink the AI Kool-Aid that benefit from actually having those kinds of automated tests. Sorry, not sorry. What do you think would be a good prompt for the AI to explore the docs? Do you have a good sense of how to prompt for this kind of information?

Speaker 1: Sage, do you have any

Speaker 2: I don't know. I I don't usually begin my promise with like you are uh helpful AI. I d I don't usually do that, I just ask the question. I assume it's already Well, yeah, I I assume with newer models it can understand questions just fine without having to have the intro like that.

Speaker 1: Yeah, yeah. Um I think you're right, but I think it's also, you know, definitely where you will get much better results from smaller models if you spend time to more carefully prompt it to answer in this or that way. But like flagship models definitely much less relevant. Okay, so we'll try um to reuse one of my existing uh support prompts. And I will adjust it a bit for the dev context. You are a professional support helper for Wagtail CMS developers. Use the

Speaker 1: LM CXT LMS full. txt files as the authorized documentation, help users look at the great information within, concise factual explanations, summarize relevant document. um provide code snippets Always provide real links to real pages of sections, that's definitely helpful. I don't know if our mm. txt file for developer docs supports this just yet, but hopefully. Uh D I I can try that. Uh retrieve page contents, yes. The CMS. Sounds reasonable.

Speaker 1: Um so we'll reuse this prompt Maybe I'll copy it to the separate separate file dev support. And so in our prompt foo config we'll use dev support. Um same models? So Baptiste, I don't remember if you were around when we discussed this, but um The point of this software is makes it super simple to compare the results of as many models as we'd wish. So one of our big design goals here is to make it work with smaller ones that are more energy efficient. and uh where we control the hosting

Speaker 1: so this provider here chose them because the data center is based in um uh region with low carbon footprint.

Speaker 3: Is it in France?

Speaker 1: It is in France. It is in Paris.

Speaker 3: Oh Paris.

Speaker 1: They're in Paris and they're in um Krakow in Poland, I think.

Speaker 3: The location of the next EuroPython.

Speaker 1: Yes. Are you going?

Speaker 3: Maybe.

Speaker 1: Ooh. Let's do a EuroPython plug. Uh oh, there is a twenty point six website, right? There is crack off

Speaker 3: Oh there's there's a familiar face.

Speaker 1: There's a familiar face

Speaker 3: Scroll up, scroll up, yeah, instead.

Speaker 2: First.

Speaker 1: Okay, moving on. So we have our full file. We have our instructions. And now we have which AI models we want. And so yeah, my point was we have our open source low footprint providers. And at the bottom we have Cloud Opus 4. 5, which is the heavyweight of the heavyweights. the ones that's been trained on all of the pirated Django and Wagtail books out there. So this one has no excuse not to be able to do that.

Speaker 3: Allegedly.

Speaker 1: No, not allegedly. They've been fined for that.

Speaker 3: No, I think they were fined for having trained on books. I think the there's still an ongoing suit for whether the books were pirated or not

Speaker 1: As far as I know, they have been fined for training on pirated books for sorry for pirating books. They have not been fined for training on books. But as far as I know, the books that have been pirated, people got uh $1,500 payout per per author. As far as I know, it's been settled.

Speaker 3: I thought there was an ongoing suit that was still about whether those books were pirated. So the source material was pirated regardless of whether they had the legal standing to actually use books for training. Anyhow, let's burn those tokens. Uh

Speaker 1: how do you want to test that the answer is correct by Atist?

Speaker 3: I was gonna ask you, uh probably you've already explained this, but uh I was curious how you had some green and some red in your previous spreadsheet.

Speaker 1: Yep.

Speaker 3: And was that manual? Was that another LM uh evaluating the correctness of the answer?

Speaker 1: Yeah, let's look at the actual docs together. Then I can explain a bit more. So we're using this software called PromptFool , which uh provide evaluations, which is the feature we're using right now. uh this is very broken inside uh cursor maybe i'm asking too much of my IDE displaying pages so I'll switch the screen sharing quickly to um Separate browser So what we're looking for is um what they call assertions and metrics.

Speaker 1: I think so far I've only used assertions, not metrics. which is pass fail scenarios and within this you will see two types deterministic matrix which is your usual regex, contains, Lewenstein distance, and so on. And model graded, which is the kind where you ask another AI model to compare kind of your expectation of an answer with the actual answer for another AI. So TLDR both I think there's value in both types of um validation. I do prefer to create deterministic ones if I can formulate a test that you can test.

Speaker 1: deterministically deterministically basically um just to be clear like i'm a beginner at this so you know this is my understanding based on having using this for a few weeks here and there not like uh a lot of work So yeah, for now we're using this top one LM rubric uh where I think prompt foo uses its own like kind of system prompt to tell another LLM how to compare things plus our own little like extra instructions from our expectation of uh the result Does that uh make some sense?

Speaker 3: Your explanation makes sense, yeah. I I'm always kind of I don't fully understand why uh feeding an LLM output to another LLM. Like you know it's it's kind of this uh Uruboros of uh the snake the snake eating its own tail of like how does that work you know like if the LLM could judge its own output in the first place uh like why couldn't it just output it at that point instead of doing an another round but I'm not a data scientist so

Speaker 1: yeah it's It's all about context management and training of the models. The models we pick here. They are pretty general purpose models that can answer anything and everything and are trained to do that. And their instructions are also gonna be like specific. to answering the question we're after. And then I guess you want a separate model that's been instructed or trained to compare the correctness of questions the answers so for example if you wanted to make your coding ai work with a very small model you want an up model that's very well optimized to do that

Speaker 1: But when you want to assess the correctness of the results, it makes sense to have a model that's been fine-tuned for that scenario. So that makes more sense That's how I understand it anyway. But you know definitely a bit of auroboros going on right there. So how do I add a custom column in a shoe in the chooser? That was your question?

Speaker 3: Yes, I can work with that.

Speaker 1: Okay. And um Do you want to say what the column should be doing?

Speaker 3: So well mine wasn't a page chooser, but I'm I'm still curious because that's the most common chooser anyway.

Speaker 1: Okay. Sage, what's the correct answer?

Speaker 2: Are we talking about page choosers or like model custom model choosers?

Speaker 3: I was talking about custom model choosers, but I don't believe that this is documented in any way

Speaker 2: I think we do have customizations for or we we do have documentation for custom Making custom choosers. I'm not sure if it goes in depth into like how to configure the columns, but yeah, uh you might have to look at the source code to see how the views are built. But we do have some mentions of building custom choosers.

Speaker 1: Ooh. Errors. We are most likely going over our token limits. Prompt is too long. Yeah. We're gonna have to um Limits the size of our full lms. txt folks. Um maybe we can remove the release notes again. So there's one catch here

Speaker 1: which is that this full file that we have currently isn't that well ordered So I'm going to have to hunt around and make sure that we only remove the right section. So we can probably preserve our index. Um let's check our token count first, shall we? Yeah, that's uh too many tokens. It's never gonna be able to fit that into anything. Um

Speaker 1: so

Speaker 3: can you delete every uh second order

Speaker 1: Every what, sorry?

Speaker 3: Every second word.

Speaker 1: How would you do that with your IDE? Is there like a secret trick?

Speaker 3: Yeah, sure. Just uh ask a cursor to delete every speaker. Hey

Speaker 1: cursor, please um I wonder how it would would do it. Okay.

Speaker 3: There's probably a pearl one liner that does it.

Speaker 1: Yeah, there's a one-liner for everything, isn't there? Okay, I think that's gonna Save us from about half the file, not quite. Which other parts of the docs can we delete?

Speaker 2: Contributing goods

Speaker 1: Yeah. That's um one of my big concerns with um AI. is um it only answers based on your current problem at a given point in time. I feel like there is a risk that um People will lose any incentives to contribute to our code to fix their issues if they're just looking at the AI. But for now it does feel um more important to make our docs more accessible, I suppose. Um sorry, I'm just scrolling through this to understand exactly where our contributing guides um

Speaker 1: And

Speaker 3: can't you um tell your system prompt that if the user is encountering a a bug they should be nudged to report it?

Speaker 1: Yeah, yeah, so that's what we were discussing with Sage. um earlier if we look at our user guide in the copy page feature here at the top I have copy prompt which is attempting to encourage um um encourage our users to use a ready-made prompt for the docs and in here we could definitely write uh If information is missing, encourage users to report this. But You know, it's not a given that people will think of using this ready-made prompt. And it's also not a given how AI follows instructions like this.

Speaker 1: But worth a try for sure. Um where are we at on token counts? It's gonna fail I assume with most models still, but um yeah There's just no way boys. Let's try with um the index file only. Um so the index file is much smaller. Because it only contains the headings of the pages and then the idea is that the AI will fetch whichever pages it's after.

Speaker 1: We can see what happens there. Do you think um it's something we should maintain maintain in uh in the long run, those kinds of automated checks of our documentation for common questions?

Speaker 2: I think it's gonna Take a lot of resources to keep it up to date and just to maintain it. I I'm not sure like having to maintain yet another thing when we can use the time to s to invest on the docs directly But maybe if if we could find a way to make it so that we don't have to spend so much time maintaining the tests, maybe it's worth it

Speaker 1: Yeah. I think my goal here was that hopefully almost all of the types of questions we'd have in files you could use the same assertions for them so that's really all you'd have to define is uh the questions like one question per line and then that's your test case you don't have to say like oh for this you have to look for this word for that this word um Whether that's realistic though, I I don't know for sure. But that's how I arrived at those default assertions here. One should provide a step-by-step guide. to should reference the docs free should avoid referencing things that are known to be

Speaker 1: incorrect Um okay. So we have our prompts with our smaller file and um We can see what happens with that. I think for a real world scenario, the way that you're meant to have this set up is um the ai will fetch your lms. txt file from your site and decide which page is to fetch after or you'd have an MCP server that provides a tool that's called like fetch docs

Speaker 1: and based on that um The AI would know which documentation pages to load into its context to answer your question. We are not there just yet. um okay so we have the recap can take a look at this quickly for a change uh for a change so we spent 13 000 tokens 10,000 for the actual tasks we gave the AI and then 4,000 for the um LLM grading So the orboros of the LLM checking is the LLM saying the truth or no. Let's see the results.

Speaker 1: And da dam Triple fail. Um I think I might switch to uh bigger browser window for us to look at this Okay. So uh Mistral Small

Speaker 1: Model Chooser Panel See at a glance this all looks decent to me, but I definitely don't know enough. R code to check how much how much bullshit there is in here. Sage, what's your bullshit meter telling you?

Speaker 2: I don't think we have model chooser panel.

Speaker 3: No, that does not exist.

Speaker 1: But is documentation reference? See, it's uh it's there somewhere

Speaker 2: So confidently adding the anchor to the link.

Speaker 1: Yeah, yeah. I love that. See that I'm I'm hoping here that something like this um we could probably output our whole list of API signatures in a format that's quite small and it would help us if we wanted to make smaller models support this. It will help us know for sure whether they are like using imports of real APIs or no. There is a project like this from um is it the fast api people? Um the Zensical people MK Doc strings does that ring a bell?

Speaker 3: I know that you do use Sphinx for the YTL documentation?

Speaker 1: Yep. Because

Speaker 3: Sphinx can generate this thing called the inventory file, which is what powers the inter-sphinx feature where you can cross-link documentation. And that's a giant blob of all the references that your documentation has.

Speaker 1: Do you know like the URL pattern for the inventory file? Or is it a binary blob?

Speaker 3: It is somewhat binary. It's a very strange format.

Speaker 1: That's exactly what we should be be trying to do, I think. if it well felt worth the time.

Speaker 2: There are tools that convert it to YAML or JSON. I have used it

Speaker 3: I think Sphinx might generate it in multiple formats as well maybe. Don't fully remember. I know I I broke it once for Django. That

Speaker 1: will definitely be more helpful in a scenario like this. How did you break it?

Speaker 3: I misconfigured the server to serve the file and create the infinite redirection loop.

Speaker 1: Classic.

Speaker 3: Yep.

Speaker 1: This is what I was thinking of. It's called Grief. And I think it's meant so that you can diff your Python codes API signature over time. When I say API like I don't mean uh REST API, I mean like the modules and methods and so on. Yeah anyway maybe we'll look at this at some point Um chat GPT OSS model choose a view set, that's something that's real Okay, for some reason it's showing its thinking but not its actual answer at the end.

Speaker 1: So we'll ignore that and assume that it's a fail on my part. to not know how to configure my prompt 2 correctly. Cloud opus. Maybe there's some prompting in my um setup still encouraging it to use the user guide. Is it as simple as that?

Speaker 2: I don't think choosers support that yet, but it is linking to the best page we have for it I guess. It might it might have mixed it up with uh because we we put the model view set documentation on the same page So this is what you're seeing here is for the listing view. So maybe it assumes that the chooser viewset uses the same

Speaker 1: Why do we put them on the same page?

Speaker 2: They're both um for setting up generic views. I guess we could we should probably separate them maybe. I don't know.

Speaker 1: What's your ideal documentation page length?

Speaker 2: Oh I think it really depends on what type of page it is. If it's a reference documentation, I don't mind it being really long as long as the uh table of contents thing on the right side um lets you scroll through through the headings

Speaker 1: Matis, any take?

Speaker 3: I think yeah, Sage is correct. There's very different or very different scenarios when you have reference or um non-reference I believe there's like what was it four types of documentation there's something with a quadrant I don't I don't not a documentation writer yeah that's the one For reference, like I never read it from top to bottom, right? Like you just usually arrive there by a search, uh, you already know what you're looking for, and then you you just look at what you're what you came to find. So in that sense the yeah the length doesn't really matter.

Speaker 3: I think the length is probably way more critical in a tutorial scenario or how to guide where you you tend to read it sequentially from top to bottom.

Speaker 1: Yeah.

Speaker 3: And there's probably science in like ideal length for page and stuff like that.

Speaker 1: I do find though even for reference there is um It's heavily dependent on your search functionality being able to link to the correct subsection. I feel like search indexing of docs generally finds it a bit easier if um It can link to top level pages. But like if we search for list display, presumably it's yeah, good enough to link us to the correct method Yeah, that's a good example of it working well. Um yeah, so definitely um something that the AI

Speaker 1: still misunderstands. that's there can be different things on the same page of docs that support different APIs. Though I assume Sage if we were to support this, that's kind of a good guess at what it would look like.

Speaker 2: I think so, yeah.

Speaker 1: Yeah. So I think to me, I I definitely hope we'll make use of those new tools to help us refine the one way or the other. I don't know if it's like better to focus on um ways to get more accurate answers always if it's better to focus on reducing hallucinations one way or the other but I do feel like um we have to invest in our docs and it's um pretty solid way to you know automate the process of seeing how people find information. Obviously it would be great to people to do this without relying on AI as well but I would rely on lots of analytics

Speaker 1: and I'm not the biggest fan either of tracking every single click of people going through our documentation pages And um yeah user testing or documentation like we just don't have capacity for that. Maybe we should maybe we should spend time with like a couple Wagtail devs and see how How they navigate through the docs. So yeah, I think that's the TXT and that's Evals and that's the current state of where we're at. And um I guess for us as far as our own files, there's definitely plenty of refinements

Speaker 1: to be had to make them more helpful. There's definitely clear trade-offs with our audience looking at our sites much less and at AI answers much more. But I'm feeling hopeful that again those files would encourage the use of smaller models that you have more control over, rather than just trusting the big models to crawl through your sites. Um I'm gonna c plug my call for feedback. I could really use more feedback on this. If uh Sage Baptist, you spend more time with those files, please tell me what you find.

Speaker 1: And um yeah I'm hoping we'll create some of those like opinionated abridged versions of the files. not try to strike a balance between having too much and too little content. Do you see yourself using this or not just yet?

Speaker 2: I mean I don't really have a need to look through Wagtail's dogs myself, um, to to ask it questions about Wagtail, but I can see other people using it. So yeah. Maybe if we have more uh documentation of our internals That would be useful for me as someone who works on Vital itself.

Speaker 1: What about you, Baptiste?

Speaker 3: I don't really have a a well-integrated process for using LLM in my my day-to-day. So I don't think I'd use and I also reach I'm way more at ease with just like gripping through a code base. That's kind of my first go-to when if I don't see the answer in the docs, I'll just open the codebase and I think I'm quicker with the uh Git grab than uh an LLM but I think that's not where The industry is going. The trend is definitely for people to use LLM for answering questions about documentation or code base So having tools for maintainers to

Speaker 3: as you said test or evaluate how a LAMPS performs on your code-based documentation is So interesting for sure.

Speaker 1: Do you think you know obviously that it takes some time to reload a project locally and get the you know your grep in the right folder do you think you would like download those files for projects so you have like Our flying records of the docs that are very grappable. Because you could you could grab this, right?

Speaker 3: Yeah, I mean for me grapping is more like I I know what I'm looking for like where is a class used or a function or

Speaker 1: Yeah, and you want to filter like is it a Python file, is it HTML I'm looking for?

Speaker 3: Yeah. I mean I I think I'm old and I have I'm set in my ways, uh which is not always the most positive of attributes. And I think documentation is the the way I imagine things. Like it's obviously a big piece of text and LLMs are And they're born out of this, you know, it's a large language model. Documentation is a big piece of language. It could make sense that it would be good at at uh finding uh things there. So I find it hard to uh I think that's more of a function of my my own brain of like

Speaker 3: formulating a question. I think my So experience or I don't know if it's innate but I think I'm much better at formulating a grep command than a a human language question of like how do I find you know it's probably uh also uh something you can train yourself but I uh I get the ick uh interacting with a large language model that is made to I think most of them, at least the ones I've used, are deliberately programmed to act like humans. On my less generous days, I think it's a design this way to trick you in a nefarious way.

Speaker 3: Might not be the case, but then when they behave when they're made to behave like humans, and then they give you a completely psychopathic answer of like giving you a link to an unknown existing page I I find this like for me this clash is like I guess almost physical pain out of it. So grep, as limited as it is, it will never lie to me. It will it is uh reproducible. I can give someone else the grab command I used and they would find exactly the same things. And the non-d the non-deterministic way and the kind of the The sloppy hallucinatory component of LLMs for me are really hard to deal with, kind of emotionally. So I don't reach for those tools very often

Speaker 2: Yeah, the last thing we need is being castled by computers.

Speaker 1: You're holding it wrong. There surely is a list display attributes on the custom chooser panel class. Oh, it will never tell you that

Speaker 3: it will never tell you you're wrong because if you then go and say this doesn't exist, you say, oh you're completely right And this if if a person was doing this, I think this is medically like uh it's uh I mean yeah. It's not a behavior that a real person would would have or I think they would be shunned uh in a society if they were to behave this way. And so the the aspect of like the They made them look like humans. They made you know it's it pretends to be a chat between you and something else and then they give them human names and They they tweak them so that they they they look like the human is on the other end writing you, but at the same time they give you this completely like

Speaker 3: scary answers I don't know. It's uh it's a very strange tool for me to use. I wish it was uh more deterministic and I feel like it's the eternal debate. I'm really curious about the How it's gonna look like for people building up skills. I um I mean as I mentioned, I'm old, uh I'm set in my ways and I have I feel I've I've learned through you know walking uphill in the rain. uh for 20 years and I don't know how it's gonna look like 10 years from now. Uh how with the generation that is grown grown on asking LLMs for how to solve things.

Speaker 3: Will they have learned better or worse? What task will they be better at? Like is there some Yeah, what can I pick up that is gonna be useful for me? Interesting times

Speaker 1: Interesting times indeed. I think as far as Wagtail, you know, we're not pretending that anyone and everyone should go ahead and adapt those things right now. Just uh trying to find I guess spend enough time in this space to find the good from the bad and hopefully get the good to emerge more. At least I'm really happy, you know, to have spent time with those eval suits, not just created those files and moved on, but actually testing. how much is the quality improved with or without the files and um it suddenly doesn't beat careful grapping I don't think

Speaker 1: just yet but for For people who expect more immediate access to information than grapping. It does feel like a step in the right direction.

Speaker 3: Yeah, I think they might be good complements because for grapping you need to know what you're looking for, right? Like this is the eternal thing of like you don't know what you don't know. And so I could see the the randomness aspect of the or the non-deterministic aspect of an LLM. Maybe it could somehow, you know, if you're prompting for something slightly wrong. Because if you're gonna grab for something that doesn't exist, you will find nothing. But an LLM might just be like, oh, the word that you look for is similar to this, and then it might actually re -put you on the on the right track

Speaker 1: Yeah. See what the next few years bring um it's definitely a different environment for people learning at this point in time Oh, do you want to see um do you want to see the um um proprietary version of this setup. I don't think I've shared yet, but like this was the open source version looking at Pound. to selling our own prompts our way. I also set up the Gemini gems version. Do you want to see that?

Speaker 2: Yeah, we can see it quickly maybe.

Speaker 1: Must be quick. Do you know what Gemini Gems are?

Speaker 3: No.

Speaker 2: I've heard of it, but I haven't used it again. So

Speaker 3: I can't tell if it's a great name or a horrible name.

Speaker 1: Um I really like your point about uh how cringe it is when LMs are named with human names. So At least I don't know anyone named Gemini nor Gem. Well

Speaker 3: Gem is a very short like common shortening of Gemma.

Speaker 1: Oh yes, yes. But it's Gem with a G here. Like a like a Ruby or a Sapphire or a Gemma. Yeah.

Speaker 3: You can write Gemma with the G as well.

Speaker 1: Really? No, but do you shorten Okay, maybe I just don't know. Uh I starting to dislike it if that's why it's named gem. Um yeah, I uh I set up two and again it's the same concept. We have our AI model, Gemini. We give it access to our docs and we give it a prompt saying how to we want you to answer this or that question. So I tell you the guide that's a custom gem that I set up and this is what it says. Oh the site has been configured by your developers. Yeah. You cannot manually add or hide columns. Filters, search, density, settings.

Speaker 1: Yeah. So Sage your question has been answered. It seems reasonable and hallucination-free. Baptiste, what about your question? Developer docs custom gem. How do I add a custom column? You should use the model viewset or choose your viewset and customize the list display attribute. Down down. So yeah, I think that's um interesting.

Speaker 3: I think it's interesting that it's probably what Sage identified the fact that the the um The chooser view set and the model view set appear on the same page seems to make the confusion that they kind of work the same So could be uh an argument for splitting them into separate separate pages Oh

Speaker 1: for supporting this.

Speaker 3: Yeah, because that's what I tried, because you like if you have worked with view sets You know that you could put the list display and it's gonna work. And so that's what I tried uh for the the chooser view sets and it didn't work.

Speaker 1: Do you open an issue?

Speaker 3: For this I don't think so.

Speaker 1: I love how you say for this I'm gonna have a peek after this call at Baptiste's list of future requests.

Speaker 3: I haven't really opened so many issues. But I did I do think I I did merge or I did uh contribute something that came out of this project about the it was about snippets and uh Signals? No, what are they called? Hooks. There it is.

Speaker 1: The settings hooks?

Speaker 3: Uh yeah, I think so. It came out of that project.

Speaker 1: Great.

Speaker 3: But I I did talk to to Sage about this. Like there's obviously a historical as for as with many things there's like The fact that choosers use viewsets is a somewhat recent development. And so I think if it was made today, obviously it would work this way. And the fact that the LM is hallucinating it isn't completely, you know, uh unrealistic, but

Speaker 1: Yeah, you It's um always interesting to compare of what a human to what a human would assume without them having done careful reading of the dogs. Um anyway. I think we're gonna leave it there. Do you wanna give us an outro as well, Sage, or are you more of an intro kazoo player?

Speaker 2: I think it's just intro for today

Speaker 1: Just as well. But he's still wanna set expectations too high, but Sage did say he would bring a different instrument to every one of those streams or the intro.

Speaker 2: Well I can say for the next one. I'm not sure if I'm gonna do that for every single stream.

Speaker 1: We have it recorded.

Speaker 2: Oh no.

Speaker 3: Record the record the recorder is an instrument.

Speaker 2: That's a good point.

Speaker 1: Have a good rest of the day and uh yeah, give me the MZ feedback.