Publish Statuses for a Headless CMS – Nick Ivons

Speaker 1: Hello, my name is Nick Ivons, and I'm a developer for the City of Austin, Texas. And this is my first Wagtail space, so definitely very excited to be here. I'm working with a team to rebuild the official City of Austin website using Wagtail as a CMS. And like a couple of other people here, we're using a headless deployment setup. Our front end is a statically generated React app built with a library called React Static. Which is very similar to Gatsby. And my talk today is going to be about a particular caveat for people who are interested in adopting a headless wagtail instance. with

Speaker 1: a static site generator. Today we're going to be doing a deep dive into page status strings. So out of the box, Wagtail expects a page to have one of five different statuses. A page can be live, it can be in a draft state. Or it can be live with an additional draft that is yet to be published. And Wagtail has two additional statuses of expired and scheduled. but they aren't particularly relevant to this talk, so I'm going to leave them out. So this model works very well for Wagtail instances that are not building pages with a static site generator. But when you're using a static site generator, live

Speaker 1: does not necessarily mean that the page is live. This is one of those gotchas that comes with implementing Wagtail headlessly with static site generation. The Wagtail backend may think that the page is live, but the page itself may not have finished building. If you're serving pages with server-side rendering and Django templates, then yes, a page will be instantly live when the status says that it's live. But if you're using a static site generator to build in a headless environment, your website will not be built instantaneously For example, the build process that we're using at the city of Austin, it can actually take between four to six minutes for a published page to actually be built and deployed on our site.

Speaker 1: So let's think about the user experience from the content author standpoint. What's going to happen when an author clicks the live button on a page that hasn't finished building yet? They get a 404 error. And this has caused a lot of confusion and frustration for our content authors who were creating pages in Wagtail. And for authors who are unfamiliar with our headless build process, they'd see this error and think that they got it because the page didn't build correctly or that maybe they did something wrong. But even for authors who knew about the latency in our build process, it's hugely frustrating to not know when a page is truly live and ready to share.

Speaker 1: So to fix this UX problem, we made some modifications to the page status string method. Here's the solution we came up with. We now inform authors of the true build status of their page. And to do this, we've added three new page statuses. A page is publishing if it is currently being built for the first time from a draft. A page is live and publishing if it was already live but now has a new revision that is being published. And a page can be live and unpublishing if it is currently being unpublished. And another very important change is that a page will only say it's live if it is truly live and successfully published. So there are no more broken links for our authors due to latency in our build process.

Speaker 1: So what I'm going to do in this talk today is show you step by step How you can adapt your page statuses to reflect the reality of your headless statically generated CMS environment. And even if you're not using a headless CMS or static site generation , some of the techniques that we're going to go over can be applied towards any workflow that uses third-party systems to Publish, cache, or update Wide Tail page data. But before we dive into the code, just to give us a little background context, I'd like to walk over our headless build process at a very high level. So when I say headless, that means that while we're using Wagtail to manage the authoring interface and the page data, we don't use it to render our front-end

Speaker 1: site The build setup that we're using at City of Austin uses build servers and AWS to run the actual process of publishing pages. Our build service is aptly named the publisher. Our headless prop publishing process has three steps to it. So one, we send a build request to our publisher service. When we publish a page, it gets sent as a published request to a build queue in our AWS ecosystem. And the second step, when the publisher is ready to build that build process, it will query Wagtail for all the page data it needs to construct the site.

Speaker 1: And we're using uh Django graphene, GraphQL, endpoints in order to uh send that data over to our build process. And then the third step, once it has the page data from our GraphQL endpoint, it can turn that data into static web pages and then put them into an AWS S3 bucket. For the sake of this talk, it isn't necessary to know exactly how the publisher works. It's okay for it to be a black box for now. All that you need to know is that it will take page data from the Wagtail GraphQL API and output it into web pages. Specific implementation details aren't too important right now. And again, lessons from this talk can be applied to any type of build service that you use outside of Wagtail.

Speaker 1: So we've got Wagtail as our source of truth for page data. The publisher, totally separate outside service that builds your web pages, and then finally those web pages are statically generated, stored in an AWS S3 file server. Once this happens, our front and site is totally decoupled from Wagtail. And this model has worked very well for us for a couple of years. It offers benefits of resiliency, scalability, and flexibility that Stephanie and Don talked about earlier in the conference. But the downside is that this process takes four to six minutes to complete. And our content authors previously had no visibility into the true build status of their pages.

Speaker 1: Until now. Alright, so we're going to dive into some code. We're going to walk through the steps that you'll need to take in order to create your own publishing status strings. So the first thing we'll do is add some extra fields to our base page model. Out of the box are page status set by two fields on the WagTorCore page model, Live and HasUnpublished Changes. So the first thing we'll need to do is add a couple of extra fields to our base page model. And we start by using a page model that inherits from Wagtails out-of-the-box page model.

Speaker 1: And we're going to call that base page. The first field we need to add is a Boolean called published. For our headless pages, published is going to be the real source of truth that says whether a page is actually live or not. I didn't want to change anything about the existing field called Live because it's so entwined into the internals of how Wagtail handles page publishing and that's working just fine. So we're adding this extra published field that will let us know if a live page has actually finished publishing. The second field I added is called Publish Request Incued That means that the page has just been sent to the publisher to be built.

Speaker 1: And it is either currently being built or it is waiting in the build queue to be built. And publish request in Qt is only going to be set to true for the couple of minutes that it takes for that page to be built. So again, just to clarify, these two fields mean different things. Published means that the page has finished publishing. Publish request encued means that the page is currently in the process of being built. And in addition to these two fields, there's one more piece of code that is very important. You must extend your withcontent JSON method. This method determines what data gets loaded when you build a page object from a revision. And there are many places where Wagtail does this on the back end where it expects the data pulled from your latest revision to be used interchangeably with the actual page object itself.

Speaker 1: So the logic we have here is saying don't use the values of published and published request enqueued from the revision, get them from the actual page instance itself. And this is a proven pattern that Wagtail already uses. It's how Wagtail's page model already handles fields like live or has unpublished changes. Since our custom publish fields are meaningful for the page as a whole, rather than a specific revision, we want those values to be set by the page itself. And this is something good to keep in mind for anybody who's using external APIs to update page state. This created some very confusing bugs for me before I realized I needed to add this fix

Speaker 1: in. Okay, so we've got our base page model set up, we've got our fields, now we need to figure out how do we first set them. That's going to be the next step. We need to set publish request enqueued equal to true when we publish a page At the City of Austin, we handle headless publishing by using a receiver. So when a page is published, we have a signal set up to trigger a site build process manually. And uh I'm just kind of simplifying the code for the sake of focus, but basically within this receiver We'll want to start our build process by sending a publish request to

Speaker 1: our publisher in AWS. And then we'll want to update our page to set its new build status. This page is currently being built, so publish request encued is now true. After a page is published, what the signal is doing is Starting our build process and updating the page to say it has a publish request in Q 'd. And this is pretty standard for other headless Wagtail setups and tutorials using the page published receiver to make a signal. The only extra piece that we're adding here is updating our page with the new published request encued field.

Speaker 1: The next thing I'd like us to think about is updating our status string method. And just as a reminder, the status string method is the method that actually sets the page status in the authoring interface UI. Let's just take a look at this again. Pretty self-explanatory. I've simplified it a little bit, cut out references to scheduled and expired, but it's pretty straightforward. If a page is not live, it's a draft. If it's live but it has unpublished changes, it's live and a draft. And if it is live with no unpublished changes, it's live.

Speaker 1: So we already have an idea of what our ideal outcome should look like. So let's take a break from coding for just one minute to look at how this workflow should play out. Let's say I'm a content author who's created a new page, how to adopt a pet. What properties will this new page have right out of the gate? Well, it's not going to be live since I just made it. It's just a draft revision. It's also not going to have an encued publish request because we haven't published it yet. And it's not published. So we want the page status to be draft. Page is not live, hasn't been published, hasn't even been submitted to the publisher. It's just a totally normal draft state.

Speaker 1: So now let's publish our page. When an author clicks to publish, what should happen to that page on the back end? Well, WagTor's WagtailCore's published logic is going to set this page to live for us. And my custom signal that we saw earlier set publish request encued to true. But it's still not published yet because the build process hasn't finished. And we want our status string to read publishing So let's jump ahead into the future. The publisher will need to send a message back to Wagtail to alert us that the page has finished building. So the page is still live, but at this step we'll need to update the page model

Speaker 1: to no longer be a publish request in queued. And we'll also have to update it to be published. And now that it is live and truly published, we can feel confident declaring that this page has a true status of live. So I'm gonna leave that there for a second actually. Take a breath. Let you make a screenshot. Alright, so next step after this workflow, let's talk about the other workflow of unpublishing a page. So an author clicks unpublish and Wagtail has set our page to have live equals false.

Speaker 1: We have a publish request in-cued because when you have statically generated sites, it also takes time to unpublish a page, or at least the way that we have have it set up. And the page is still published because our unpublished process hasn't finished yet, so published will be true. And our status string will read live and unpublishing. When the publisher finishes its unpublished process, it'll need to send a message back to Wagtail again. And that's when we'll know to reset publish request encued back to false and set published equal to false. And at this point, our page is back to being a draft.

Speaker 1: Alright, getting back into the code. So here's the status string method that we started with, Wagtail out of the box. And now we need to modify it in order to incorporate our two new fields that'll give us some extra logic possibilities. We need to add published and published request encued. This is the logic that the city of Austin uses to incorporate publishing status into the page status string. We now have three additional possibilities. A page can be live and publishing, publishing, or live and unpublishing. And we were able to set these values with our two additional page fields, published and published request

Speaker 1: in queued. And like the original status string, the logic's pretty self-documenting. So if we have a page that's live, published, but also has a publish request in-cued, then we know that it's starting to update a page with a newly published revision. So it will be live and publishing And if a page is live published but doesn't have a current publish request enqueued, then it's just a normal live page. And we can use the same logic that the original status string method had. And if it's live but not published, then it must currently be publishing for the first time, which gives it a status of publishing. And if a page is not live, then that means it's either being unpublished

Speaker 1: or it's in a draft state. So now that we have our status string method, the next thing we need to do is update our page instance after the build process finishes. The existing build process that we had was fire and forget. Wagtail would send data to the publisher, the publisher would build the website. But there wasn't any communication from the publisher back to Wagtail after the site build had completed successfully So that's what we need to add now. When the publisher finishes the site build, it needs to tell Wagtail that its pages have finished building.

Speaker 1: So next I'm going to show you how we decided to set up an endpoint wagtail to receive build confirmations from the publisher. And I'd be very interested to hear if people have better techniques or strategies for doing this But the option that we ended up going with was just implementing a view with the Django REST framework. And on top of the Django Rest framework, we use the Django REST Framework API key library in order to secure the endpoint. Inside of my published succeeded endpoint that I created, I need to parse the request body that the publisher sent me. So the publisher needs to send me back the IDs of the pages that it has built

Speaker 1: so that in this step we know which page instances we need to go ahead and update. And we're actually going to handle the page update logic in a separate thread. We want this endpoint to function like a fire and forget webhook. For the purposes of our build process, the publisher doesn't need to wait for all the pages to update before it continues on its own process. And uh we actually had to create this separate thread in response to an error we were experienced a couple weeks ago during a big data migration. So there was at one point we had to update over 200 pages at the same time and republish them at once. And this endpoint was timing out

Speaker 1: before it could finish and send its response back. So by handling that in a separate thread and not needing that HTTP connection to stay open for 30 seconds or longer than it took to process the pages. We were able to get things running smoothly. So let's now take a look at what our handle publish success thread is going to do. We're going to take every page that we published. We're going to get the specific page instance by querying on the ID. And we're going to update the published field depending on which action was taken during the build request. And this action

Speaker 1: metadata was originally sent to the publisher during our page published signal. So, in order for a publisher to have this information, you have to know to send it to them during the first step. So if the action taken for that page was published, then our published field is set to true. Otherwise, if we unpublish the page, then we set published equals false. And publish request in Qt is now false because our build process is over. And then we save the page. And On a high level, that's basically it. It's everything we needed to do to update our page instance after a successful build process.

Speaker 1: We set the correct published value, reset publish request enqueued back to false, and now our page status strings will know the true state of our page's build status. And final step, pretty simple, you just have to update your page status tag template. And this is pretty easy to do once you've updated your status string method. We'll need to overwrite the Wagtail default template in order to render the status strings to our liking. And we just use a series of if statements to render different links depending on what status string the page has. You're free to implement your Wagtail admin HTML however you see fit. And you can look at our GitHub repo to see the complete implementation for how we handle styling, JavaScript, and all that.

Speaker 1: And there you have it. It's everything you need to know about updating page statuses for a uh Statically generated uh Wagtail front end. Just gotta update your base page model to add new fields, new status string method, new with content JSON method. Update the page instance when your page is published. Update your page instance when the build process is over, and finally , update your WagTilAdmin page status tag template. So that's it. Thanks so much for listening. And if you'd like to see any of our complete code, project is totally open sourced on GitHub. Janus is the name of the decoupled React front end.

Speaker 1: Joplin is the name of our wide tail back end. And as a bonus for anyone who is curious, You can look at the publisher repo to see how we use DynamoDB and serverless JS to build and publish pages. If you want to talk more about this, you can reach me on the Wagtail Slack channel. It's just my name, Nick Ivonne's, or on LinkedIn. So thank you very much, and uh I'm open to any questions.

Speaker 2: Thank you very much, Nick. That was an amazingly detailed and interesting talk. I have a question actually, and I'm glad you posted the link to publish it because I know you said We should think of it as a black box, but I'm I'm interested to know more about how that works and in particular if it supports incremental builds.

Speaker 1: Um good question. So The pieces are in place for it to support incremental builds, though it does not already. And we're thinking about implementing that. Possibly with a migration to Gatsby. We've heard a lot of good things about that. So kind of at a high level, what the publisher does is It takes it creates um a Fargate serverless uh Docker container and it will pull in the back-end Sorry, it will pull in the front end uh Janus code, our front-end site, run that build process, and it

Speaker 1: There's a few layers. It's basically we we have a build queue set up with Dynamo DB that Pulls in page requests that come in, consolidates them into one thing, spins up a Fargate container to run the build process, and then stores it into AWS. And that's it at a high level. There's a few interesting levels where we try to do caching and speed up the process and optimize it. But Yeah, basically at a high level, it's just kind of a uh standalone serverless uh static site build process that can be used for any number of Wagtail backends or any number of front end branches.

Speaker 1: So it's really useful when we're doing uh new feature development because you can publish from any wagtail back end to any front end without needing to manually like set up build servers or connections.

Speaker 2: Great, thanks. Well I look forward to to checking out the publisher app, seeing a bit more. There's a couple more questions. There's one from Andy. Who asks, uh so both for this talk and more generally for those with headless setups, how do users view or share draft rendered pages without publishing to the live site? Is there a separate parallel staging environment? And does that then mean redoing changes in the production environment?

Speaker 1: Good question. So the way that we handle previewing pages is we actually have a route on our any of our front-end sites production staging that is a route called preview. So what we actually do is that's the one place in our site that is not decoupled from Wagtail. So we have a preview route, and if you pass in a parameter for a revision ID, It will actually go into Wagtail, query the GraphQL data for that, revision, and then post it on the preview site. So that part is not decoupled. And another important point is that that part

Speaker 1: is also, for our purposes, not secured. So anyone can query in theory our GraphQL backend to get revision data, but This is kind of fine since our site is public, resident-facing, and we need the revision data in order to generate previews for any potential revision. So that's how we do it. We kind of sort of work around the static site limitations.

Speaker 2: Got it. Thanks. And here's a question from Bill Higgins. Have you had any other issues with communication interruptions between Publisher and Wagtail?

Speaker 1: Communication interruptions with publisher and wagtail. Um so The errors that we are most concerned about are within Publisher itself. We don't we haven't really experienced issues with Wagtail communicating to publisher or publisher communicating to Wagtail. The real issue that we've seen is potentially something in the build process breaking. Maybe you know queries are timing out or just something goes wrong up there. So what we've done is we actually have set up our publisher to be Failure tolerant. So if a build request will fail, it will go back into the queue and it will get rerun and picked up with the next build process.

Speaker 1: And if we need to make a hotfix or push some code to fix our build process, it will then rerun all of the builds in the queue with the correct code. So yeah, to answer the question, no, we haven't we have not had issues sending to the publisher, getting data from the publisher back into the Wagtail as of yet.

Speaker 2: Great, thanks. I think the last question is another one from me, which is how did you make the slides and specifically the syntax highlighted code snippets with incremental reveals?

Speaker 1: Oh, okay. Um yeah, that's kind of a fun one. So the way I did that is So I'm using Atom as my code editor, and there's a plugin for copying code as a rich chext file. So basically I copied that and pasted it in order to get the formatting that I want so it looks like code. And there'll be a link to these slides so you can actually like copy and paste it. They're not images. And in order to get this incremental code going, these are all separate slides. So I just made a new slide and pasted another line of code.

Speaker 2: It works really well. I bet I was hoping you might have a an easier answer, but it looks the the result is beautiful anyway.

Speaker 1: Cool. Thank you.

Speaker 2: I think that is it and we're at we're at time. So thank you again, Nick, for a really fantastic talk and a great way to finish up the Wagtail Space Talks for this year.