Wagtail - Yet Another Django CMS by Tom Dyson

Hi

Speaker 1: everyone. As Edward said, I'm the my name's Tom Dyson and I run Torchbox, which is a UK agency. We do digital work for people making the world a better place. People like Greenpeace, Oxfam, the Red Cross, the United Nations. We built a content management system called Wagtail, and that's what I'm going to talk about for the next 25 minutes. This isn't a particularly technical talk So if you're not so interested in the story behind Wagtail, then you can just pretend to listen and uh you can use the pr the time productively by building a little Wagtail site. So uh inspired by Microsoft I've got a promotional idea. And

Speaker 1: if you show me a Wagtail site that you've built by the end of today, I'll post your t-shirt. Here's the one-minute version to help you get started. This gets you as far as the admin interface. If you don't have a photographic memory, you could take a photo of the screen, although that would give the game away Here's a link for the extended 10-minute version. This shows you how to build your own page models and templates. If you get stuck with anything, I'd be very happy to help you afterwards And finally, for the super lazy, you could just try out the demo. This is running on a little Docker container on my colleague's $5 DigitalOcean box, so maybe don't all try this one at once.

Speaker 1: Back to the story. Torchbox has been working with Django since the early days. My uh my brief, my fleeting claim to fame in the Django community is that I made the first ever Django screencast in 2005. Soon after that I hired Simon Willison and Andrew Godwin. In fact, the first version of South was built by Andrew for a Torchbox project. We have a team of developers at Torchbox using Django to build things like SMS apps delivering personalized agricultural information to subsistence farmers in Kenya. Ecological footprint calculators, grant management systems, tools for booking beds and care homes. But for the last five years, for big public content-managed websites, we've mainly used

Speaker 1: Drupal. Has anyone here used Drupal? It's maybe about half of you. It's easy to be rude about Drupal. But it's it's very powerful. It's it has an enormous community. The the ecosystem of modules in Drupal is probably as big as the ecosystem of Django applications. And most importantly, you don't get fired for choosing Drupal. That's especially true in the non-profit space where most of Torchbox's clients come from. And we've made some Drupal sites that we're really proud of, particularly perhaps Oxford University and the Royal Institute of Foreign Affairs. Both of these are high-profile sites, big sites with masses of content.

Speaker 1: high traffic, they work well for their users, and lots of people say nice things about them. But Drupal has some big flaws The first of these is that configuration happens in the database. You can imagine why a developer would think this is a good idea. But it becomes very painful when you move from developer to staging and then production environments. And then your client calls you and there's uh there's a there's a problem that you can't replicate because they've configured things differently to your version. And on a similar point, Drupal is too flexible The power that Drupal gives administrators, and particularly around creating new content types through the user interface and generating columns and tables, becomes pain for developers.

Speaker 1: Performance is an issue. Most Drupal sites rely heavily on Varnish. And that's okay for a mainly static, mainly mainly a read-heavy site. But Varnish doesn't help the the logged in editors who have to put up with a sluggish authoring experience. And when we deliver a Drupal site, we have to train the editors. That's not a disaster But I think ideally, if you can sign up for a Netflix account or buy a book on Amazon, then you ought to be able to understand the fundamentals of creating and managing content on a website, on a CMS But Drupal's worst sin, in my opinion, is that it's opinionated about markup. Drupal generates HTML for you.

Speaker 1: This restricts our design and our UI choices, and it makes site maintenance slower for our developers. Most importantly, it makes us more expensive for our clients. So faced with these frustrations, we persuaded one of our clients, the Royal College of Art, to commission us to build them a new CMS in Django. We launched their site in late 2013 and we launched the open source project in February 2014 at Django Weekend, an excellent short conference in Cardiff. I guess this was this was half about being a good open source citizen, but half commercial We knew that we couldn't persuade new clients to let us use Wagtail

Speaker 1: to build their sites. If the only people maintaining it were a small company in the Oxfordshire countryside. To be honest, we weren't very well prepared to manage a reasonably sized open source project. Despite that, everything seemed to go perfectly for the first few weeks. On the day my colleague Matthew announced Wagtail at Django Weekend, I posted a link on Hacker News and I asked a couple of friends with decent karma to upvote it. And it hit the front page and it stayed there for a while. And for a few giddy days, we were in the top trending projects on GitHub And then the contributions came flooding in. Obscure European language translations. Feature suggestions.

Speaker 1: One evening someone in Sweden noticed an issue, an edge case with admin permissions. But by the morning someone in Mongolia had fixed it. It felt like we'd discovered perpetual motion. People on the internet doing our work for free. For three weeks, our core team, two developers, worked hard on what we've thought was the most pressing issue, our lack of tests. We went from 0 % to 80% coverage pretty quickly, and then we eked out a couple of percent every day until we plateaued it about in the low 90s where we are now. Still documentation additions and thoughtfully worded enhancement suggestions kept flooding in.

Speaker 1: In our weekly team meetings. We were referring to these generous strangers by their first names. Have you seen Jess's new work on refactoring the JavaScript? Did you see what? Phil's ideas around documentation. But after a couple of months, we realized that as we were investing more and more time fixing the bugs that other people were kindly finding for us. The volume of contributions started diminishing, and the tone of engagement shifted. It took us a while to work out what had happened, and I think we've addressed most of those reasons. So from my vast experience of launching one open source project, these are my top tips.

Speaker 1: The first is to use that energy of launch. People want to be part of the new thing When we when we announced it on Hacker News and it and it got onto the front page, that's what you know that's what engaged people and made them want to be part of something that looked interesting I guess I'd imagine that there was going to be a gradual increase in engagement. But there was a big rush in those first two weeks that we've never really recaptured. And I think if I were doing it again, then I would make sure that we had even more resources available to support people and respond and feedback in that early phase. The second point is about establishing a predictable rhythm for our work.

Speaker 1: So this means being uh giving people clear indications about how often we're gonna deliver new releases, uh how the beta testing process works, what the roadmap looks like. In this case I think Velocity isn't so important as just giving people the impression of constant motion. That's what people want to see, that it's it's steady and then it doesn't stop. If I were doing it again, I would start with better documentation, particularly around making it real easy for people to get started, and I would start with tests. Not because tests are critical for a project that's only two weeks old, but because it avoids you having to answer snarky comments on Reddit.

Speaker 1: We made some assumptions about how developers work. For example, at Torchbox we use vagrant heavily. It's uh it means that we get uh clean parity between our developer and production environments. And uh and it makes it much easier to get started on a new project and you can you forego the uh the pain of installing libjp and uh Postgres drivers on OSX. And uh and we imagine that everybody else felt the same way. But it turns out that quite a big proportion of the Django developer community don't want to spin up a virtual machine just to test your shiny new content management. system. Similarly, we uh we use Postgres and we uh you know we imagine that most of the world does too but it turns out that that's not the case.

Speaker 1: I think reducing barriers to installation is really important. An example here is that we use SAS to create the uh to manage the CSS for the admin interface. And uh my colleagues had a philosophical objection to shipping the combined compiled CSS files in the repository. So this meant that we required anyone using Wagtail to have Libsass, which is a again, it's fine on Vagrant, but it's a it's a pain to to build on OS X and other developer environments. And I imagine there are a lot of people who again might have been interested in uh in Wagtail and and and and seeing these posts and who would have tried it out and after 15 minutes got frustrated about having to to build lipsass and gave up and probably never came back.

Speaker 1: And finally, this is a this last point is about communications and uh I think it's easy to feel that if people aren't constantly replying to your tweets and blog posts and Google group updates, it doesn't mean that they're not there And this point came home to me with uh one of our first big Wagtail clients, um , so a big broadcaster in LA And uh I went out there for the for the meeting and um it was sort of surreal experience for me. I was sitting at this big mahogany table and leather armchairs, just like in the films. And through the window I could see the Hollywood sign. And uh and the guy across the table who I'd never met was talking in uh talking with amazingly precise detail about the the coding habits of my colleagues.

Speaker 1: And uh, you know, I realized that even though people aren't writing to to tell me that they're using and they're interested in this stuff, You are under scrutiny and people are interested and the fact that they're not communicating back with you doesn't mean that it's not important for you to keep posting and talking about your work. So after we fix these problems, and particularly the ones around documentation and tests and establishing a predictable rhythm for releases. We started seeing some really high quality contributions. In particular, there were two agencies on either side of the Tasman Sea. They both decided to switch to Wagtail Wholesale and to invest in its development. Springload is an amazing agency in New Zealand.

Speaker 1: They've been building Wagtail sites for the Red Cross and the New Zealand government. And they started prototyping a next generation user interface with React. Take Flights is another fantastic agency based in Hobart and Tasmania who politely suggested and then implemented some fundamental changes that we should have seen before we'd started. And then in the US we're starting to get interest, and I guess that's why I felt uh you know I should make this trip from Oxford to Austin, because I started seeing blog posts and work by really respected agencies like Cactus and Revsys. Lincoln Loot referenced Wagtail in the high performance Django book. Wharton School of Business

Speaker 1: who are here and Tivix. Is anyone here from Tivics? I've never met anyone from Tivix, but they're doing some really good stuff on Waitel. And then there's some big sites, NBC, the broadcaster I mentioned before, they're building a massive news site on Wagtail. There's a political campaign and a tech giant both running sites that you've heard of but we're not allowed to talk about. Incidentally both of those sites use Wagtail's static site generation feature, which isn't really prominent in the documentation, but I think makes sense for uh for for sites where scale and security is paramount. The UK and Finnish and US governments are using Wagtail. And I know this because I can see their projects on GitHub.

Speaker 1: NASA and Oxfam are using Wagtail. But so are pawn sites and car manufacturers. And this this has raised some a sort of unexpected moral dilemma for us. We wouldn't work for some of the organizations who are benefiting from using Wagtail because we don't like what they're doing. I can't think of an obvious way around this. As far as I know, there are no open source licenses which limit usage to organizations who conform to my fluffy liberal Western ethics. So that's that's where we're at now. And the next steps are really around defining our roadmap more clearly The killer feature I think of WibTel is something called stream field, which is a bit harder to explain than it is to demonstrate.

Speaker 1: Essentially it's a it's a s it's a content type. that lets people build this sort of long form narrative content-rich content like we're used to seeing on Medium or New York Times or The Guardian in the UK. But without storing everything in a big blob of unmaintainable HTML. For those of you who who heard Andrew Godwin's talk about database anti patterns. I was pleased that he he he thought that storing uh Storing some CMS type content in JSON fields is the right way to go because that's our approach with with JSON with Streamfield. But uh we've got ideas about making Streamfield even stronger and starting to remove the requirement to have

Speaker 1: WYSIWYG interfaces at all. Um to to have rich text editors at all. Otherwise in the roadmap we've got uh this there's support for international internationalization in Wagtail, but we need to we need to have a better built-in interface for it. Wagtail stores versions of all the content, but you can't currently uh revert to previous versions through the through the UI. We wanted to have a cleaner separation of the internal API and uh And the and the user interface so we can start doing cooler stuff like the like the React-based prototypes that Springload have been doing. And we want better support for really massive sites with millions of pages and terabytes of content on remote storage.

Speaker 1: So we're identifying all this in the roadmap, but but m more particularly we uh we're using WEPS. So uh you you all know about PEPS. Python enhancement proposals, which has been a very successful part of the Python community and has been adopted, as you may know, by the Django community with DEPS. And uh and you can imagine what what webs are. This is our our own version where uh we there's a GitHub repository of a structured structured feature request that explain the motivations behind a behind a feature and and the implications and how it might be dealt. And I think and the idea of with a with a web is that uh it's a package that another developer or agency could pick up and understand that the but that by contributing they'd be delivering

Speaker 1: along the along the lines that that we want to see Wagtail grow. Finally, I'm going to address the title of my talk. Why did we build yet another content management system? As someone who who used to code and now mainly manages coders, I recognize the constant fight against not invented here. And it's true that maybe we it maybe we'd have been better open source citizens if we had contributed that effort to an existing platform rather than building our own. There are, as you know, some very established projects. Mezzanine is one. We we used mezzanine for a project two years ago and we liked it.

Speaker 1: In fact, some of Wagtail is inspired by by mezzanine's key principles. And Django CMS of course is very powerful, very mature. We recently inherited Django as Django CMS project for a UK government aid project and it's going well. But there's some fundamental differences in our approach. One is around control versus flexibility. This is a bit similar to the Drupal point. In a way, Wagtail gives you less flexibility from an administrator's point of view and it hands control to the developer. So new fields are defined by developers. We use migrations to make changes. We use standard Django templates, the tools that you're

Speaker 1: used to working with. Django CMS focused recently on front-end editing and WYSIWYG editing, which is appealing and works in lots of situations. But for us, we uh We follow more the the so-called cope model. Create once and publish everywhere. And uh and and that means having a a more focused editorial interface that that enhances the authoring experience. Aside from all that, we had a basic commercial imperative to build the content management system that our clients wanted. And in basic terms, it was quicker for us to start from scratch than to build on an existing project. But I don't think this is a problem.

Speaker 1: The demand for new content managed sites is growing more quickly than the supply of new content management systems. More quality tools in the Django ecosystem should be good for everyone And when your newer, cooler, Django-based CMS hits Hacker News next year, I'll smile ruefully and I hope that you'll make fewer mistakes than we did. Thanks very much.

Speaker 2: Hi, I'm excited to get started with Wagtail. I didn't understand your description of stream fields or webs, and I wonder if you can demonstrate. At one point you said it would probably be easier to show than to tell.

Speaker 1: Okay. I think the the way the best way to show webs. So it might be trick tricky for me to find it. This isn't mirrored on my screen so I'm looking over here. The Django ones are a better example So an example let's see what I can see. So here's one which is a a kind of template for a

Speaker 1: a Django enhancement proposal. which shows the metadata and gives some of the and gives some guidelines for how you might want to structure your your feature request. Ours are a bit simpler than this because I think uh we just want to make it easy to fix for people to get started. But basically it's a it's a structured standard way of describing features and their motivations that then gets agreed, collaborated on and agreed and becomes part of an established roadmap. I think I won't have time to show Streamfield right now, but I'd be very happy to demonstrate it to people straight after this.

Speaker 3: Hi, uh I've been building a Wagtail site for the last month or so. Um I've been really enjoying it and I highly suggest everyone here gives a shot and plays around with Streamfield and wraps their head around the idea. And uh and see how it allows you to build your own widgets for different types of content. One thing that I was noticing reviewing my code over the last couple days is that I have a lot of hacky ways that I'm including resources for individual stream fields in, such as having a widget that may be displayed six different times, but needs an individual resource each time, a little bit jQuery code. And so it gets included eight times in the code. I was wondering if there's any better way or any future plans to integrate some More front-end

Speaker 3: functionality ways to include resources in stream fields as we build this up because I see that just continuing in what we're building and expanding.

Speaker 1: I think that's a really interesting point and uh and uh it's so important for us to To look at the ways other people are using our software. And I discovered this yesterday with with Margie who just asked the first question and I I went through the process for 10 minutes of her setting up the site and immediately just looking over her shoulder notice some things which are which are obvious to me but but but wouldn't be obvious to someone coming through this for the first time. And similarly the way that we build our stream stream field objects. are uh uh you know in the same way that we made the mistakes about assumptions about the way developers work, almost certainly it's not going to be the the case the same way that you're building stream filled objects I think one way of approaching this is maybe having a section of the documentation around recipes so we can have some

Speaker 1: examples and and guidelines of of of how to do this in a in a maintainable way. But also I'm sure that there are improvements to the way that we can we can offer stream fields and particularly provide hooks around the uh around the the the mm JavaScript injection, which means you don't you don't have to repeat it many times. It'd be good to talk about that. Does anyone manage to make a Wagtail site during the tour? Great. Five t-shirts. I think that's it.

Speaker 4: Do you have any tips if you have uh one of those other CMSs, so Django CMS let's say if you wanted to migrate one of those sites to your system Any any general tips that you might offer for something like that?

Speaker 1: Actually I think this is somewhere where we can learn from Drupal. Drupal has a very good module called the migrate I can't remember if it's called a framework or a module. That uh the way they approach it is they have a kind of an interim database. and uh and you write the rules that get it in the interim database and then that becomes a common format that's easy to import. And I think we could we could do something similar to that I guess the our standard answer to that kind of question is Wagtail is just Django. So anything that you would be able to do, you know, to to migrate any project into Django, you should be able to do with Wagtail. I mean it's it's that's a bit of a cop-out, I think, because there are um You know, there there are some conventions that uh that we use that that may not be immediately obvious. I think it's

Speaker 1: If our goal was massive adoption of Wagtail, which perhaps it is, then I think providing some sort of simpler migration from common platforms would be a good move for us.

Speaker 5: Is there a way to hook it into uh an organization's existing uh username and password infrastructure with uh LDAP or Kerberos or any of that? Uh

Speaker 1: yeah, that's a really common requirement and it's one that we've done many times. In fact for our first ever Wagtail project, the Royal College of Art, that's all based on on LDAP. LDAP is really, you know, Django is doing all the hard work for us, but LDAP is just a a one-line conf change in importing the LDAP library. We've also done it with um SAML and Shibeleth for us for some other university sites.

Speaker 6: So in Wagtail a page is structured content and a block is also structured content and you're currently using JSON for streaming. field and I'm interested to know because I've had situations where maybe I would want to pull out a particular block inside of a page. Would it be possible to use models so that it's represented in the database in that same way? Because when you think about how the page pages are represented in the explorer It has like a path, right? But then you could infinitely nest blocks, like blocks within blocks. So then a block like 10 levels deep would technically also have a path within its own pace.

Speaker 6: So I'm just curious to know your thoughts on representing blocks and pages as being one type of data.

Speaker 1: So one idea is that we do we we move away from the JSON representation to to modelling in a relational database. I'm not sure that's necessary in particularly As you're probably aware, the the JSON support in Postgres means that uh it is possible to index and filter on content within JSON. And uh performance for that is getting better and better and you know and I think Postgres makes a fantastic document database in uh in competition to something like Mongo. So that's possibly one step, although obviously that doesn't help MySQL users so much. Things do get harder when you have uh when you have deep deep nesting. But but at the moment JSON is handling it reasonably well.