State of Wagtail

That's good. Hi everyone. I'm Tom Dyson. I'm the uh co-founder, founder and CTO at Torchbox, which is the UK-based agency that created Wagtail and open sourced it 10 years ago I've given a version of this talk a few times over the years. It's a simple format. I'll give a summary of what's been happening in the Wagtail open source project and then talk about where we might go next The last time I gave it in person in the US was in this same city, I think nearly five years ago, in a love in another lovely venue just up the road. And I just want to again thank Wharton for hosting this wonderful event again and for all your continued support to this community. So for the history part of this talk, I'm going to talk about

the last two years and a lot's happened. White tail stars have continued to increase from 12. 2,000 to about 17 ,500 today. And that's not the explosive growth you get from a new project that's just been announced on Hacker News, but it is trending upwards. So That's about 30% of our total stars and about 20% of the time that Ragtail's been around. And if you just divide that number in the last two years by the by the days that have passed since we've a lost since this last snapshot. That's about seven point three new humans every day from all around the world coming across Wagtail for the first time and thinking, that looks interesting.

It's worth a start. We've also shipped a lot of stuff in the last two years. The release numbers have gone up more quickly than before because we switched to semantic versioning about two years ago. But we've maintained our quarterly release schedule and the release notes get longer each time. Hundreds of tiny fixes and quality of life improvements, but also big ticket items, like universal listings. A complete rewrite of snippets. The biggest investment in the user experience since Wagtail's launch. My colleague Thibaut made this. little video of uh showing some of the changes that happened just in some of the versions in this period.

So from 2. 16 uh to then four the the the the new color the end of the teal the sidebar uh in four point two then we get a to Live preview, so I think one of the most dramatic uh changes for quality of life for editors. Minimizing the bar, uh the mini-map on the right hand side And then that was until about a year ago, and there's been like significant continued investment since then

Wagtail's ecosystem continues to grow as well. And in particular, I want to mention Wagtail AI. The unexpected power of large language models has brought about what I think is the biggest shift in our industry in the last 20 years. I know maybe there are some people who are a bit more skeptical than me in this room. And I'm not going to go into detail about Wagtail AI now, because I know that Jacob, my colleague, is going to give a detailed talk about it tomorrow. And many of you may have also watched some of the webinars that we've put out about Wagtel AR. But I do want to say that I'm really pleased about the direction of this package Which doesn't promise miracles and but but provides a thoughtful and sensible platform to augment the editor experience with large language model features

We've also attracted a lot of new talent in the last two years. Core team has grown. Three of those are Sage, Sage Abdullah. who works nearly full time on Wagtail and joined the core team about 18 months ago, who's made a huge made a huge impact on on the the uh the the the number of features that we've been building and the quality of the code. Cyber from Iceland and more recently Jake Howard, another of our my colleagues from Torchbox He just had his proposal for background workers in Django accepted. You might have seen that in the in the Django world. I think that's a really important step for Django And interestingly, it's a feature that originated, or at least a proposal that originated in Wagtail.

That's where Jake originally wrote the proposal. Our core team members Dawn and Tebow are on the boards of the Python and the Django Software Foundations respectively. And I think it's really great for us to have these deep two-way connections to the language and the platform that we run on. We also have sub-teams from the core team now, so sub-teams that uh are Focused on subjects like performance and security and accessibility, sustainability, developer relations. These subgroups are really thriving and uh and are helping to to move Wagtail forward at a new pace. And I'd really encourage any of you who are not yet closely involved with with the open source project itself

to join one of these core teams. It's a very kind of low low stress way of getting involved and uh hugely welcome. We've also maintained our relationship with Outreachy and Google Summer of Code, both of which have been a big success for Wagtail. As an example, previous participants from Outreach E and from Google Summer Code have turned into mentors for new projects The outputs of last year's work, which was in partnership with the Green Web Foundation, had demonstrable real-world impact. I really recommend that if you haven't already you read my colleague Thibault's brilliant blog post from last month about this. Just to summarize, these are changes that we've made in Wagtail as a result of these projects

that just sort of nudge site developers towards making websites that are more sustainable, that require less bandwidth, that also have the advantages of loading more quickly And we T Bow's taken a snapshot of I think a thousand websites and looked at the difference and it's just it's pretty dramatic to see that uh these small changes we make here have It's a lever that we can pull that can have a dramatic impact. And then this year's Google Summer of Code participants, they're shown on the screen here. That's Mezu from Nigeria and Nandimi from India. They're continuing this focus on sustainability and accessibility. We've always always tried hard to meet up in real life in Wagtail

And of course that took a hit in the years following the pandemic. So I'm really happy that many of us are seeing each other again, of course, as well as welcoming our remote participants today And we've had a great presence at various events this year. So at PyCon we had workshops and talks. And then at DjangoCon Europe , more talks, more workshops. Wagtail Space was hosted by Four Digits in Arnhem in the Netherlands last week, where I gave a version of this talk And um and of course and it's wonderful to be here now. And uh at at Wagtail Space in the Netherlands last year I was talking to Syver and Toomey from Overcast in Iceland

It's quite late on in the evening, so I don't think it was a real commitment, but uh there was some some promise of a wagtail space in Iceland next year. So maybe maybe some of us will get to meet again there on an amazing island And the other big thing that happened this year, as Megan said, is that Wagtail had its 10th birthday That's a pretty big milestone from an open source project, and I think it's something that we should all feel proud of. But Big mature software projects have their own challenges. And that's what I want to talk about for the next 10 minutes. Some of you might have seen this blog post I wrote the screenshot on the screen now with a rather clickbaity

subtitle to commemorate Wagtail's birthday. I'm not going to read it all aloud now, but I am going to read just a s a short section of it, which is relevant to what I want to say. Managing software is hard. Content management has many more tricky bits than we could have foreseen You start by thinking that a framework like Django gives you 80% of what you need. Templates, routing, user management, data migrations. On top of that, you need a nice UI, then some way of handling draft content and previews. Those are not the same. Then versions, an API, search, non-page content, authentication, localization, image handling, comments, reporting, scheduling. Each of these areas split out to its own set of challenges. You add a support for Elasticsearch, and then you realize they make backwards incompatible API changes every year, and you're committed to supporting the thousands of Wagtail

sites who support it. Look at the epic list of updates in the latest release. In the simple days of 2014, we could never have imagined that ten years later a kind stranger would ensure that table block cells are accessible when using keyboard control only. Which I thank El-Hussein Al-Nazari. I went to school in the city of Canterbury in the southeast of England I was in the choir and I used to sing in this cathedral three times a week. The choir teacher once told us about the tiny details that are carved at the top of the building. invisible to the congregation, created there by stonemasons who would have been on wobbly wooden scaffolds about a thousand years ago, for the glory of their God.

I'm not comparing Wagtail to an actual cathedral, but some of the work in our codebase reminds me of these hidden details. They're not there to help marketing with a cool feature hit list. but to protect future users from problems they'll never know they might have faced. For example, consider pull request 5149, started at a Wagtail space in 2019, I think. to make rich text soft line breaks configurable. Since then there's been a discussion back between core team members Thibaut, Cohen, and Matthew Matthew wrote last week, as I see it, this is something we would introduce after developers have been given sufficient notification that they need to add BR as a feature if they want to continue using line breaks. This way, if developers miss the memo, there'll be a period of time when the damage will be limited to them going, hmm, why can't I add line breaks here?

Oh, I see, I need to add BR to the features now. rather than Wagtail corrupting their existing content on say. I'm not expecting you to to to follow the the detail of that, but I I think this extreme thoughtfulness is an unusual quality of Wagtail. And like the visitors to Canterbury Cathedral, most Wagtail users are never going to realise the care and attention that has been lavished on invisible components of the structure that holds it all together. This is a castle also in the south-east of England surrounded by a moat which you probably know is a water-filled ditch intended to defend the building from attackers. Venture capitalists talk about the importance of having a moat.

I don't know if that's a familiar usage of the word for you. So a moat in the kind of venture capitalist world means like a special skill or technology or a feature that means that your competitors can't easily catch up with you. I don't think MOAT is a really natural technology to use terminology to use in open source, which is a world where we want all good open source projects to succeed. But I still maybe it's a useful concept for what I want to talk about now. All these features that we've added in the last two years and all the thoughtful invisible components Are they a moat? Something that a new challenger CMS has to provide before it can compete? It's tempting, I think, for incumbents to feel this way.

I imagine this is how the big old banks felt a few years ago. These newcomers don't have branches, they don't sell mortgages, they don't support checks, they don't have Fortran code bases with a million lines of code handling every edge case we've seen in the last 40 years. But now, at least in the UK, everyone I know has switched to banks like Monzo or Revolute because these new banks really have focused on the experience for new users. So you can sign up with your phone. And and they don't focus on supporting features that people needed 30 years ago. It's the same with mature software projects, I think. We as a as a as a group of wagtail enthusiasts, we might see a new shiny headless JavaScript CMS and we might think Well, they don't support WebP, or they don't have a well-documented way to handle versioned non-page

content. But if the new CMS is fun to use and it lets you build 90% of what you need quickly, those things probably won't matter. The people using it will figure out WebP and version content later when they need to. So all this amazing work that's been done over the last two years, it's important and it's valuable, but it's not emote. Two years ago in Wagtail Space in the Netherlands, I talked about the value of simplicity, but now I think it's a bit more nuanced than that. We have this thing that we've built together. It gets faster and smoother every release, but also bigger. The documentation grows. The number of concepts that you need to get your head around to be a Wagtail expert

increases. The challenge for us is How can we continue this valuable work? Reducing database queries, supporting new and more efficient image types, providing helpful UIs to nudge editors towards building more accessible sites, and so on without increasing the cost of entry for new developers. I think there's a risk that we are accidentally building a moat, but not against future competitors, but against new developers Matthew gave a great talk about this last week, the title The Impossible Art of Keeping Everyone Happy, which I think is relevant to this, and I recommend what watching it when it gets published. This is a very friendly, appealing slogan, which I think comes from Drupal, but I've also seen it used in Django

To be honest, I've never felt very comfortable with this idea. We are really lucky to have a friendly, kind community in open source, in Python, in Django, and in Wagtail. But I think you should choose software on its own merits. If something better comes along, we should use it and we should take our kindness with us. is a less snappy version, which I don't think is going to take off. But the point I'm trying to make is that new features, endless attention to detail, and a wonderful community aren't enough And that it's precisely because we now have a mature, high-quality software project that we need to focus relentlessly on reducing friction for new users, to bring in new ideas, fresh ideas

and energy. And to make sure that we stay relevant for what people are building now. I want to spend the last few minutes of this talk. thinking about some of the practical things that we can do to make Wagtail more accessible to newcomers while continuing to add features Here's a breakdown of things I want to talk about. Firstly on docs. Our documentation is excellent. It's intricately linked to the code base, so you can't submit a new feature without updating the docs. We have a great new editor guide. I hope some of you have seen that We've also benefited from the advice and support of Daniela Proshida, who's helped us reshape the docs along the side of the lines of his Diotaxis framework.

But our docs of course could be even better. They could be available in many languages. They could use more screenshots, and those screenshots could update automatically when there are new designs We could stop all feature development for a year and just work on docs. And I reckon if we did that, the number of you new Wagto users would increase as a result. Have any of you used Langchain? Has anyone come across Langchain? Toolkit for building large language model applications. Python library. It's grown incredibly fast since it launched last year. I think it's a it's it's just about to hit 100,000 stars on GitHub. To be honest, I'm not sure that it's great software.

It has a sprawling API that feels to me more like Java than Python. It's organized by features but also by back-end LLM provider. But it addresses a set of problems that a lot of people suddenly started wanting to solve last year. And most importantly, the documentation is excellent. And I think For all its faults, of which of course I don't blame it because it's mainly one one developer who's created all this incredible stuff in a very short period, I think the fact that the documentation good is good trumps almost everything else. If you wanted to start building using Python, uh maybe even if you weren't a Python developer, even if you wanted to start building an application that that uses large language model it's a pretty good chance you'd use you'll

use Langchain because there's so many examples out there and you can cut and paste in the documentation you get something working and then you refine it from there. So first up is documentation. That's not an exciting or new point, but I do think it's really, really important. The second is starter kits. We have always proudly avoided being opinionated about the front ends that people build with Wagtail And uh I think you know the I still believe in the in the rationale for that. But it does lead to an underwhelming experience when you go through the steps of building your first site We do have Wipetail Bakery, but that's more intended as a demonstration of various features rather than being a useful standalone site Despite that, we actually see a lot of sites which are lightly reskinned versions of Wagtail Bakery.

A couple of releases ago, we added support for adding a template to the Wagtail start command. And in the last few months, Torchbox has been working on a simple new site template. The lines on this on the screen now don't work But uh that's my that sort of represents my dream for how easy it would be to build a Wagtail site from a template. And then once you'd once you'd built that, you could deploy it in a single step. Deployment still feels to me like the hardest thing in Django. Do you ever look at the the the subreddit on on Django About every third query is someone saying, Django's amazing, I've built this site, it happened so quickly, the docs

were great, but how do I deploy it? And the skills that you need for getting your site up on in public are quite different to the skills that you might use for creating models and views. So I th I still feel I feel like and I know that a lot attention has been put on this, but I still feel like this is the kind of the big unsolved problem for for Django and therefore for for Wagtail. And so something like this is is is my dream for how easy we make it. And we are we're we're getting there, we're chipping away at this. So the the screenshot a couple of screens ago don't work, but this does work. And so with these instructions, you can get a nice looking production ready

new site running Wagtail on your local machine You don't have time to uh copy and paste that right now. Here's a QR code. First time I've ever used a QR code in tool, but it does seem to work, which uh takes you to uh Takes you to that snippet. And if you have 10 minutes during a break today or even during the rest of my talk, then I encourage you to try that out and give me some feedback on why why it doesn't work. And while manage. py deploy doesn't quite work yet, my colleague Jake has written some really good deployment instructions starting on fly. au because that's the easiest for us to get there quickly. Which are in the README for that news template. So once you follow the previous steps and then you look at the README

, if you have a fly. io account or you set one up quickly using GitHub for example, then you should be live within few minutes. So we're not quite there yet, but uh we are chipping away at this and I think I think succeeding in this will really help new developers. And I The next step of course is not just for people to point people at this news template that Torchbox has built, but to encourage people to build new templates and to make that as easy as possible. This this last one is a bit more niche, but I have a feeling that it will become increasingly important over the next few years As developers increasingly use AI-powered tools like GitHub Copilot to help them write code, I think there'll be a shift towards the languages and frameworks that can most easily be accelerated by tools like this.

In the same way, perhaps that open source text editors which support typing have led to a renaissance in strongly typed languages, or at least to typing in previously weakly typed languages like TypeScript and typed Python. Other ways that we can influence this trend. For example, one way might be making sure that the big large language models index our documentation. Or maybe we can cran train a custom LLM on Wagtail examples. We can also look for ways that LLMs could reduce friction for new web developers. I'm going to try swapping to a live demo now. As a little proof of concept that I built to show one way that we could use large language models to help developers build their page

models. So on this site, which is at actually can't quite remember the URL, but I'll share it on Slack afterwards. I could write something like uh blog page with a main image and maybe some related books And uh I could say tell me and at this point in time uh I've made a big prompts that include some examples and some instructions and the result of that is a blog play class that has a main image Relate to documents and then

specifies the content panels. That's a pretty simple one Something else that you can do because sometimes the way we prompt large language models can be improved. So this is a bit meta, but it's thinking about ways that we could give a more helpful prompt um it's gone quite a long way beyond my original ideas but I can clearly just add or or remove X sections here but let's try if I just run tell me on this This might take a little bit longer because there's a bit more detail. But at the moment these are taking about 10 seconds ago and I put the price. I think this for this one I'm using the the clawed, the Anthropix clawed API

So here's a much more detailed one. Worked out tags with a tag item base main information related documents. That's thought about search fields, which is good as well. For those of you who are already building Wagtail sites, you'll know all this already. This might not accelerate things much for you. But if you are coming into Wagtail fresh, then I think tools like this can can can accelerate a lot Here's something else we could try. So I was really pleased to see this article come out last week. If I zoom out a bit and then take a screenshot.

of this article, just get enough to get something relevant. And then Go back to my model maker. Try just putting in a screenshot instead of giving a description So again we're sending that screenshot to the Anthropic API, telling it this is this is an example of a page and asking for some advice on how to build a Wagtail page like this. So it's worked out that it's in a page about events. It's got a data, location, an intro, images, content panels look about right to me. And again, it's thought about search, which is a nice touch. So again, I I I don't think this is this is this isn't taking work away from developers

because you you're always going to want to modify this and and and uh get it just right. But I think Tools like this could be really great for reducing that initial friction that you might experience as a developer and thinking about what Wagtail models, page models look like. Okay, finally, I want to say that open source content management systems are important. And I worry that smart creative people who work on content management systems might have their attention taken away by things like artificial intelligence or crypto, computer vision, virtual reality.

And maybe it's easy to feel that content management is a boring and a solved problem. But I think in an age of massive social media platforms owned by Megacorps or by states who probably don't have our best interests interests at heart, or capricious billionaires with frightening political opinions. I think decentralized information delivered on open, trusted software is more important than ever. And I thank you for the role that you're all playing in that. Thank you.