The State of Wagtail

It's a really annoying thing about getting old that uh I need my glasses to see you and then I have to take them off to look at what I'm my notes. So I think most of the time I'm gonna take them off and you're all just a nice blur of smiles, hopefully. Okay, I'm Tom Dyson. I'm one of the founders of Torchbox, who the company that uh originally created Wagtail 10 years ago. I've given this talk a few times. The last time I gave it was in this same city, almost exactly two years ago. in a lovely another lovely venue, which was like a bit of a squeeze for the 60 people that we had last time and definitely wouldn't have fitted the hundred

people that I think are coming today. Thank you, Four Digits, for hosting this wonderful event again and for all your continued support to this community. So the the format of this talk is is is pretty simple. A couple of years and then I talk about where we might go next. And a lot has happened in the last two years. Our our GitHub stars have continued to increase from uh about 12,000 to 17 ,500 today It's not the explosive growth that you get from a new project, but it's trending upwards, so that's like a bit more than 30% of all the stars in just in a bit less than 20% of the time.

And if you dis divide that that difference by the days that have passed since uh I was last in Arnhem, that's a daily 7. 3 new humans from all around the world coming across Wagtail for the first time and thinking, that looks interesting. Click. We've also shipped a lot of stuff. The release numbers have gone up more quickly than before because we switched to semantic versioning. But we've maintained our quarterly release cycle and the release notes get longer each time. There are hundreds of tiny fixes and quality of life improvements, but also big ticket items like universal listings. a complete rewrite of snippets, the biggest investment in the user interface since Wagtail's launch.

I've here on I've used uh my colleague Thibaut who made this helpful video. I can move there. Okay, I should also which shows very quickly the changes in the editor interface from version two point one six to four The sidebar coming in, 4. 1 with preview. That was a really dramatic change, I think. 4. 2, you can see some styling improvements. Stops there and we're now on this there's the the mini map.

The Wagtail ecosystem also continues to grow and in particular among that ecosystem I want to mention Wagtail AI. I think the unexpected power of large language models has brought about maybe the biggest shift in our industry in the last 20 years. And I'm not going to go into detail about Wagtail AI now because I think it might be covered a little bit later. And also maybe many of you have watched the webinars we've put out about it. But just want to say I'm really pleased about the direction of this package. Which doesn't promise miracles, but provides a thoughtful and sensible platform to augment the editor experience with LNM features.

We've also attracted a lot of new talent in the last two years. In the room today, we have some new core team members, Sage , who's made a huge impact on Wagtail. Siva, Jake, who's just joined the core team. Jake's also just had his proposal for background workers in Django accepted. I think that's a really important step for Django. And interestingly, it's something that originated in Wagtail, which is where Jake originally wrote the proposal. Our core team members, Thibault and Dawn. are now on the boards of the Django Software and Python Software Foundations respectively. And it's really great for us to have these deep two-way connections to the language and to the platform that we rely on

We have sub-teams now in in that that help the core team, teams with uh focused on performance, security, accessibility, sustainability, developer relations, and there's more. If any of you are interested in becoming more involved, I think starting with the subteams is a really good idea. We've also maintained our involvement with Outreachy and Google Summer of Code. Both of which have been a real success for our project. Previous participants have turned into mentors, and the outputs of last year's work in partnership with Green Web Foundation have had real demonstrable effect. Uh Thibaut wrote a brilliant blog post about this last month which I I recommend checking out. I think Chris Adams is here.

Chris are you here? Or is Lisa is gonna be here and uh and is hopefully gonna hear more about that work later today. This year's Google Summer of Code participants are shown on the screen as Mezu from Nigeria and Nandini from India. They are continuing our focus on sustainability and accessibility. We've also restarted our presence at events, which slowed down over lockdown I've had a really good presence at uh PyCon in the US with talks and workshops, I think a month ago or two months ago, and just now at DjangoCon EU with more talks and workshops. Here we all are at Wagtail Space in the Netherlands and next week I'm going to be giving uh a similar version of this talk in Philadelphia.

I don't know if anybody else is gonna make it to Philadelphia as well. I hope so. And um I'm pretty sure that last night uh when I was talking to Saiva and Tumi they they promised to hold a wagtail space in Iceland. So uh anyway, maybe we have that look to look forward to. Another big thing that happened this year is that Wagtail had its tenth birthday. That's a pretty big milestone for an open source project, and it's something that I think we should all feel proud of. But big mature software projects have their own challenges, and that's what I'm going to talk about for the next 10 minutes. Some of you might have seen this blog post I wrote with a rather clickbaity subtitle to commemorate Wagtail's birthday.

I'm not going to read it out all now, but I'm just going to quickly read this short section. Managing software is hard, and content management has many more tricky bits than we could have foreseen. You start by thinking that a framework like Django gives you 80% of what you need. Templates, routing, user management, data migrations. On top of that you need a nice UI. Then some way of handling draft content and previews, those are not the same. Then versions, an API, search, non-page content, authentication, localization, image handling, comments, reporting, scheduling Each of these areas splits out to its own set of challenges. You add support for Elasticsearch, then you realize they make backwards incompatible API changes every year and you're committed to supporting the thousands of Wagtail sites who use it. Look at the epic list of updates in the latest release. and so on.

I went to school in the city of Canterbury in the southeast of England. I was in the choir and I used to sing in the cathedral, in this cathedral that's shown here, three times a week. My singing teacher told me about these tiny details that were carved at the in the interior of the towers at the top of the building. invisible to the congregation, created there by stonemasons on wobbly wooden scaffolds thousand years ago for the glory of their God. I'm not comparing Wagtail to an actual cathedral, but some of the work in our codebase does make me think about these hidden details. They're not made to help marketing teams with a cool feature list, but to protect future users from problems they'll never know they might have faced.

For example, consider pull request 5149. It's just started at Wagtail Space in 2019, I think, and with the title Make Rich Text Soft Line Breaks Configurable. Since then that that pull request has has prompted discussion between Thibaut and Kuhn and Matthew. Matthew yesterday wrote, as I see it, this is this is something we could introduce after developers have been given sufficient notification that they need to add BR as a feature if they want to continue using LIDER. breaks. This way if developers miss the memo there will be a a a period of time when the damage will be limited to them going, hmm, why can't I add line breaks here? Oh I see I need to add BR to the features now, rather than Wagtail corrupting their existing content on save This extreme thoughtfulness is, I think, an unusual quality of Wagtail.

And like the visitors to Canterbury Cathedral, Most Wagtail users are never going to realize the care and the attention that has been lavished on invisible components of the structure that holds it all together. This is a castle, also in the south-east of England, surrounded by a moat, which is a water-filled ditch intended to defend the building from attackers. Venture capitalists talk about the importance of having a moat, a special skill or technology or a feature, which means that your competitors can't catch up with you quickly. I don't think it's really a natural or a comfortable terminology to use in open source in a world where we want all our good open source projects to succeed.

But I think maybe mots are a useful concept for what I want to talk about. All these features which we've added in the last two years and all the thoughtful invisible improvements Are they a moat? Something that a new Challenger content management system has to provide before it can compete It's tempting for incumbents to feel this way. I imagine this is how the big old banks felt a few years ago. These newcomers, they don't have offices or branches, they don't sell mortgages, they don't support checks, they don't have Fortran code bases with a million lines of code handling every edge case we've seen in the last 40 years But now, in the UK at least, almost everyone I know has switched to new challenger banks like Monzo or Revolute.

Because they they've really focused on the experience for new users, like signing up just with your phone, and not on supporting features that people needed 30 years ago. It's the same with mature software projects. We might see a shiny new headless JavaScript content management system pop up. And we might think, well, they don't support WebP, and they don't have a well-documented way of handling non-versioned, non-paged content. But if their new CMS is fun to use and it lets you build 90% of what you need quickly, it probably won't matter. The people using it will figure out WebP and version content later. So all this amazing work that we've done over the last two years is important and valuable, but it's not a moat.

Two years ago, when I was giving this talk, I talked about the value of simplicity. I think it's a bit more nuanced than that. We have this thing that we've all built together. It gets faster and smoother, but also bigger. The documentation grows. The number of concepts that you need to get your head around to be a Wagtail expert increases. The challenge is How do we continue this valuable work? Reducing queries, supporting new, more efficient image types, providing helpful UIs to nudge editors towards efficient image more accessible sites and so on. without increasing the cost of entry for new developers.

I think there's a risk that we are accidentally building a moat, but not against future developers, not against future competitors, but against new developers I think Matthew's going to talk about this a bit later, and I'm looking forward to hearing what he has to say about it. This is a friendly slogan which some of you might have seen before. I think it comes from Drupal, but I've seen it used in Django. I've never felt very comfortable with this idea. I think we're lucky to have a friendly, kind community in open source, in Python, Django, and Wagtail. But also I feel like you should choose software on its own merits and that if something better comes along, we should use it

and just take our kindness with us. Here's a less snappy version. But the point I'm trying to make is that new features and endless attention to detail and a wonderful community aren't enough. And that it's precisely because we now have a mature, high-quality software project that we need to focus relentlessly on reducing friction for new users. to bring in fresh ideas and energy and to make sure that we stay relevant for what people are building now. With that in mind, I want to spend the last few minutes talking about some practical things that we might do to make Wagtail more accessible to newcomers while continuing to add features.

Firstly, documentation. Our docs are excellent. They're intricately linked to the code base. You can't submit new features without updating the docs. We have a great new editor guide. We've benefited from the advice and support of Daniela Proshida, who's helped us to reshape the docs along the lines of his diataxis framework. But the docs could be even better. They could be available in many languages. They could use more screenshots. And those screenshots could update automatically with the latest designs. We could stop all feature development for a year and just work on docs. And I reckon if we did that, the number of new Wagto users would increase. I don't know if any of you have used Langchain.

Hands up have you come across Langchain? Not many, three or four in the audience. It's a Python library for building large language model applications. It's grown incredibly fast since it was launched last year It'll probably hit 100,000 stars on GitHub in the next month or so. That's that's it's already more than Django, and it's only a year old. I don't I'm not sure that it's great software. It has a sprawling API, feels more like Java than Python. It's organized by features but also by back-end LLM providers. But it addresses a set of problems that a lot of people suddenly wanted to solve last year. And most importantly, the documentation is excellent.

If you last year were excited by what you might be able to build using LLMs, then the chances are that you and you wanted to use Python, then there's a good chance that you would end up using Langchain. because the tutorials just make it easy to you for you to get from zero to a working application just by copying and pasting the right bits. And I think that's really a significant reason for success. The second one on my list is starter kits. We have always proudly avoided being opinionated about the front ends that people build with Wagtail. But the downside of that is that it leads to an underwhelming experience when you go through the steps of building your first site We have Wagtail Bakery, which was developed in the the first ever Wagtail sprint

in Reykjavik. But that's intended as a demonstration of various features, not as a useful site. Nevertheless, we we see a lot of sites which are lightly reskinned versions of Wagtail Bakery. A couple of releases ago, we added support for providing a template to the Wagtail start command. And in the last few months, Torchbox has been working on a simple news site temp size temp a news site template. On the screen now is my dream for how easy it would be to build a Wagtail site from a template. And then once you've built it, you could deploy it in one step. Deployment is still the hardest thing in Django. Reddit is full of threads saying, Django's amazing, I built my new site, but how do I make it live?

So far we haven't quite achieved my dream, but we are getting there. On the screen now are the instructions for getting a nice looking production-ready news site running on your local machine. And and we are we're we're chipping away at this and uh and getting it closer and closer to the to the kind of four-step dream. I've never used a QR code in a presentation before, but this is the QR code that takes you to that gist. And if you would like, then you can copy and paste those lines. uh into your into your laptop and uh let me know if it works for you. So we don't have the managed. py deploy yet.

deployment instructions which uh which are pretty simple for fly. io which we started with uh digital ocean because uh we did a survey I don't know any of you saw that and digital ocean was was the the most requested But it's just um there are too many manual steps. You have to create uh you know, the digital ocean equivalent of of S3 for storage and uh you have to do that manually then refer back to it and uh that's exactly the kind of friction that I think. Will mean that someone who's playing with this for half an hour and their lunch break will give up and use WordPress instead. So So uh so we we we're getting there and I'm really I'm really optimistic about the difference that having a good starter kit and easy deployment will make.

The last one on my list is a bit more niche, but I have a feeling that it will become increasingly important over the next few years. As developers increasingly use AI-powered tools like GitHub Copilot to help them write code, I think there will be a shift towards the languages and frameworks that can be most easily accelerated by those tools. In the same way that open source text editors which support typing have led to a renaissance in strongly typed languages. What can we do to influence this trend? One way perhaps is making sure that the big LLMs index our documentation. Or maybe we can even train our own large language model on Wagtail examples. Here's a little proof of concept I built to show one way that

we could use large language models to help developers build their page models. If I have time. I'll show you it. I can do a live demo first. I'm gonna have to Now I can't even for see where I've put my glasses. So uh if I put in a uh textual description of the page I want. Uh A blog page with an image, maybe some documents

First thing I can do is I can ask it to create a model for me, or or or I might ask it to uh refine my description, come up with a better version So that sends it to the actually the Anthropic API. It gives me this refined version. Looks about right. I could tweak that, but uh I'm gonna go with the refined version. This one takes a bit longer and now I'm submitting um along with this with this description, quite a large prompt giving some examples about the sort of output that I might might get. And now you can see that The models have been generated for me

and again I could tweak this, work out what might what might not be working so well about it, and apply it. And then um more recently, with the arrival of uh of models with with vision You can do things like this where so here's uh here's an event page for the uh venue that we're in at the moment. I'm gonna take a screenshot of it Quite awkward taking a screenshot behind your back. Anyway. I hope I've captured that. And then Now I'm gonna

drag my screenshot in and send that off and uh see if that will give me a a useful model. Of course, large language models hallucinate and get things wrong. But that doesn't look like a bad job. It's recognized that this is uh this is an event and it's given what looks on first glance like a like a fairly sensible starting point, something that I could use in my in my code So I think tools like this are going to become increasingly useful. And um I'm I'm like probably many of you, I'm really wary about

about generating this kind of uh nonsense content that's gonna uh f fill the internet with slop that we don't understand. But I do think that LLMs can be really useful as uh f in in tooling and uh and I really hope that we can find ways that will make Wagtail more accessible to new developers or to speed up the work that we're doing at the moment. Finally, I just want to say that open source content management systems are important And I worry that smart creative people who work on content management systems might have their attention taken by artificial intelligence or crypto or computer vision or virtual reality.

But I believe and and maybe it's easy to feel like content management is a is a boring and a solved problem. But in an age of massive social media platforms owned by megacorps or by states who probably don't have our best interests at heart. Or capricious billionaires with frightening political opinions. I think decentralized information delivered on open, trusted software, is more important than ever. Thank you for the role that you're all playing in that. Thank you.