Code that creates content

Speaker 1: All right everyone, welcome. Let's get started. This is Alex Morega and he's going to talk about code that creates contents. Welcome Alex, over to you.

Speaker 2: Thank you, Storm. Hello. Yeah, so uh this talk came about uh because uh Thibault messaged, I think, on the WagTech Slack that it would be good for somebody to talk about uh creating content for tests and for uh migrations and I was like well why not I did this recently and I uh Yeah, I might be able to say something interesting, hopefully. So here we go. There's a GitHub repository with a lot of what I'm going to show. It's linked there on the slides, and it's gonna be on each slide. It also has a link to the slides if you want to follow along and see my notes with the cheesy jokes I'm about to make before I make them and ruin them for yourself.

Speaker 2: But yeah, let's do this. So Yeah, I'm gonna share a lot of code. Apologies. But I shouldn't apologize because the talk literally starts with codes that creates content. So you get what you ask for if you come here. So yeah, briefly we're gonna look at uh Wagtail from the perspective of Django models because this is what we're going to create. We're going to create instances of models and connect them and Do that instead of clicking through a web interface in the admin. And then we're gonna uh look at writing basic simple unit tests And then browser tests, which are a bit more interesting because they have to commit data to the database and then they wipe the database, so you have to make sure you have everything in there to run them.

Speaker 2: And then a couple of tips about importing content. So not writing tests, but actually importing content into an actual site that people interact with. Okay, so yeah, Django models. Let's get the easy ones out of the way first. Snippets, images, documents. These are pretty straightforward. A model you can instantiate, it has fields, you you set the fields, and it pretty much works as you expect. Um easy to create, easy to modify, easy to reason about. And then we have pages. Pages are more interesting. They live in a tree. This is managed by tree beards. This is your white-tailed page structure. You have pages

Speaker 2: nested inside pages. They're implemented using Django multi-table inheritance. So you have the page model of Wagtail and then you have your own uh page class that inherits from page but page is not an abstract model it's uh an actual model uh within its its own table So that creates some interesting uh things to look out for, which we actually, they're not gonna bite us, but it's it's it's good to have that in the in the back of our minds. They also have revisions. So this is when somebody saves a draft or publishes a new revision. There's a history of all the revisions. And we need to be mindful of that when we import content. And interestingly, when you run Wagtail Start, the template gives you some initial migrations

Speaker 2: which create a default site and a homepage for that site. And that's going to be important for us in our tests. Alright, so let's look at writing some unit tests. First of all , we start with WagTail start the like the default template for creating new projects. This gives us a homepage, which is exactly like we have online five and six. It's just a class homepage and nothing in there. It has a title because it inherits the title from the WebTel page. Module. And then I've added a new model content page, which we're going to play with a bit. Okay, so with that, let's write our first test.

Speaker 2: I'm not using the test from the starter template because I actually think it's a bit broken and it's maybe not indicative of how you would actually write tests in a project. And also I'm using PyTest. I think it's easier to work with and it's it's easier on slides as examples because there's less boilerplate around and you can just read the actual test Okay, so very briefly line six, we need to tell PyTest that we're gonna use the Django database, otherwise he's gonna complain. And then we create a fixture. which normally you use to set up content for tests, but we're just using it to grab the sites that was created by the migrations and the root page and return the root page.

Speaker 2: So then on line 14 the parameter home page is Going to be the root page of the site. And then line 15, 16, we are simply creating a page and adding it to the page tree. And honestly, that that's all there is. So We have a page at this point. It's not that difficult. And most of the code that you're going to see following is equally simple because Wagtail is so nice. the API is is really straightforward. It's it's it's you just when you read the code it's it's pretty obvious what it what it does. They're subtleties, but they are sort of nicely packaged and when you need them you you can read for them. And then in our test finally we use the Django test

Speaker 2: client to fetch the page. Page has a URL attribute, very nice. We can fetch that page and we can check that our title is is in there. Okay Now typically we're gonna have string fields on pages because data allows us to manage the content. So Here we have added a body field to our content page, which is a stream field. We're using the declarative stream block syntax to define a stream field with five types of blocks A rich text block. I should have called that rich text. It's paragraph. Sorry, that's gonna be a bit weird. An image block, a document chooser block, a page chooser block, and a snippet chooser block. And the snippet is right there at the top call to action.

Speaker 2: And then to write the test, uh not much has changed from the first test that we wrote. We have a body over there with a JSON-ish value. We have one stream block, which is of type paragraph, that's the rift text block, and we give it the string for the content. And then we yeah, we save the page, we fetch it, and we make sure that uh that the paragraph text is in there. And yeah, this works. Now I hear you ask, how do we know the structure of that JSON? How do we know what to write? And what we do is we create a page in a site, like by hand. We get the URL uh we get the page ID from the URL and then we can run manage.

Speaker 2: py shell, import our model, fetch the instance of the page Get the body, get the raw data from the body. We coerce it to a list because it's uh it's a special object type, it's not quite a list, and then pretty print would just print a long string. But this way we can have a bit of an indentation And you can see here some data block key and some ID values which we can simply ignore. Wagtail generates these when it needs to and it uses them for its internals. But for tests we don't need this. And yeah, you can see what each block looks like internally. These numbers are just primary keys. It's actually very straightforward. Okay, so then moving on, let's test. We

Speaker 2: test the paragraph. Let's test the image block now. The JSON is a bit more evolved because this is not the image chooser block, this is the newer image block Which has a couple more fields. So we add them here. And then the assertion is actually checking that the rendition is part of the content because That's what oh I should have mentioned at the start the the the stream field blocks that we are using they're using the vanilla templates in Django because Django uh in Whitefield because Whiteville does have vanilla templates It's very nice. It was easy to put this example together because you have so much out of the box. And the example template for image block uses the 600 by 338 rendition, so that's what we also check for.

Speaker 2: And then yeah, you can see all I7 how to create an image. And we're using a Wagtail helper get test image file, which gives us a nice white image, which is an actual image for tests. For documents it's the same concept. The value is even simpler, it's just the primary key of the document. We also have a helper to create the document. And then the page chooser block, we create the page, just like the page we test with, except it's a different page. And then yeah, we assert that it's there. We could probably use BeautifulSoup for the assertions to locate a bit of content in the markup, but that's a different talk. This is just about creating content. Um okay and then the snippet chooser block, same story.

Speaker 2: Yeah, create uh uh an entry in the body, which is just a list of blocks, and uh check that it shows up. So yeah, this works. This is simple. You can use this and be happy already. But can we do better? Yes, we can. There is this thing called Wagtail Factories. It's a library created by Michael van Tellingen. I hope they didn't mess that up. It's currently maintained by Josh Mann, my uh my colleague at Torchbox And uh what it does is it's based on Factory Boy, um which you can use with Django already. Um and it has some extra factories for Wagtail uh models.

Speaker 2: And here We are using it to create our content page, and you can see we set a parent, we set a title, and then we have this weird body under under syntax, which I mean if you think about it, it's obvious. It's adding content to the body, and those numbers 0, 1, 2, are indexes in into an array. And then Let's say on line 11, body under under one equals image. Well you would expect that to create an image for you, right? If if if things are set up. And that's exactly what it does. I've just mixed a match here a couple of ways of doing things. You can nest these deeper. You can maybe set two parameters for the same

Speaker 2: field. It's really powerful. And then for the assertions , you can see like on line number nine, I'm just getting the image object from the database because the factory has created it for me. And it's there. So if I want to assert, if I want to check that it shows up in the page, I get the one existing image object because there is only one at this point. And check with it. There is some complexity with this. We need to set up our factories, which is a bit verbose. So here we have just part of that. We have a factory for content page. uh which references the factory for

Speaker 2: content block which is our stream field definition and some of this is using like or lines five and six and seven. It's using factories that are built into Wagtail factories because we're using the vanilla blocks. But for our custom blocks we have to create our own factory. So this ends up being a bit verbose and a bit crazy. And I won't pretend that I understand how it's like in super detail how it works. What I do is find examples that work, adapt them, and once it passes, once the test passes, then It means we're good. So yeah, hopefully this repository can can help with that. There is documentation coming.

Speaker 2: Right now the documentation says basically look at the tests and see what see how you can use this. But the better documentation is coming. Josh has a pull request to with that. So uh yeah maybe I would encourage you to help with this by reviewing that pull request and uh and uh yeah improving the documentation. But anyway, yeah, you you need to set this set this up once and then you can write tests, write night nice tests as as much as you want. Okay, another topic I promised I'd mention is caching. So in tests, I out of the box this is not going to be a problem. But if you have caching set up on your site, Wagtail

Speaker 2: does use caching internally. For example, I think when it figures out the site for a page that it needs to serve, it caches the site somewhere. So this like if you're running tests with production like configuration for caching, for example Redis, then you might find that cached values are cached from test to test and this might cause issues. So yeah just either use a dummy cache or uh make sure to flush the cache either before or after each test, like maybe with a PyTest fixture. Alright, let's look at some more interesting tests. This slide says browser tests, but maybe actually it should say transactional tests

Speaker 2: because In Django, like normally you're gonna write tests that live in a transaction and the transaction is rolled back when the test is complete to restore the database to a no state. Now with browser tests That won't work because the browser needs to connect to a server that has access to some data in the database and your tests with its transaction is in its own bubble So you cannot do that. You need to use transactional tests, which I mean the name is weird, but basically it means that your transaction is committed to the database And because of that, Django needs to do cleanup after each test.

Speaker 2: It will just truncate all the tables. Which leads to interesting things happening. So let's look at a basic example first. This is Playwright , using PyTest Playwright actually, which is very nice And it's using live server, so it's this is a bit of Django help uh testing utility that will run uh on actual server during the tests so that Laywright can connect to it with a browser and this test works. So if if this test is all you have, you run it, it works, it's fine. But if you have multiple tests and we're simulating this with um parameterizing our test to run twice.

Speaker 2: Then the second run is going to fail. This is because after the first test, Django is flushing the tables. So then remember when I said that migrations create a site and a page out of the box, a home page out of the box, well those get flushed. They are they're not there anymore for the second test. So line 8 is going to fail because there is no site. So this is what we need to do to make it work. It's actually very straightforward. If we read from line 13 onwards. We get the default language code and then make sure there's a locale for that language code. I could have just hard-coded English there, but this is like the correct way to do it, I guess.

Speaker 2: Because that's also gets flushed and we need a locale. And then line 15, we create a home page, line 16, we create a site. And that's all we need. With these free objects, we can create pages and load pages, load them in the browser and run tests. But this if this is the first test that's running after migrations have run, then actually we end up with two home pages, two sites. And Usually that's fine, but sometimes it causes problems. And yeah, it's best to just get rid of them. So line 11 and 12, we just delete any page, delete any site, so we have a clean slate. And this works. This is uh this is kind of bulletproof

Speaker 2: already. And if our transactional tests run in the same test suite as regular Django tests with uh rollback in the transaction. Actually we need to do the same thing because we cannot guarantee that those tests will have access to an existing page and site. So we need to do this setup for all the tests. Alright, so that was about automated tests. Minimal content, just enough to run tests. But okay, let's see what it means to import content into a site that people will interact with. It's a bit more complex, not much more complex, but yeah, a few interesting things to to

Speaker 2: To think about. So first of all, revisions. This is a management command that creates a page. It looks a lot like the code that we used in tests. But it does something additional at the end, line 15 and 16. It saves a revision and it publishes the revision. So that's not strictly necessary. If we didn't do that, it would still work. But uh it's good to have that initial revision because then editors can see it in the history of revisions and they can go back to it if they want. it's more it's it's it's what you would get if they created the page manually so yeah it's like polite to do that And we can also use factories here. And uh I didn't mention this, but factories can create random content if we want.

Speaker 2: Um so maybe for fuzzing in tests or maybe for sample content with just random If you don't want your content to change all the time, you can set a random seed, a fixed random seed , if you're using this for test content. But yeah. Lots of things you can do. Right. And then okay, we have uh let's say we're correcting two pages, teal and indigo, and we want them to link to each other This actually happens in the real world because people will want to link all over the place, will want to link to link pages and make sense. What we do here is we just create the blank page So there's a so it's in the database and it has a primary key, then

Speaker 2: create the content and then save it. It's it it works, it's fine Rich text, so um rich text can contain references to other pages like links. Uh it can contain documents and images embedded There is a bit of a special syntax for that. It's not quite HTML. It gets transformed when the page is rendered. And we can just recreate this syntax when we import content. We just Interpolate primary keys into that markup, it works. Um and another gotcha, maybe if you are importing a lot of content, is that uh tree beard really is not meant to be used concurrently. So if you add two child pages to the same parent

Speaker 2: that updates some internal data structures, private data structures of tribute, don't do that concurrently. Grab a log somewhere for the parent page. um call refresh from db and uh and uh yeah add them that way. Okay, finally last slide and just um just just over time A lot of this was based from learnings from the Govai Migration, uh RISON, uh Mansuri had a talk at DjangoCon US 2025, which I cannot watch because I don't think they published their videos, but When and if they do, I think that's gonna be interesting. Um that was many tens of thousands of pages migrated. Um That's where the concurrency uh gotcha

Speaker 2: came in because we used uh for um workers to to do this concurrently And if you want to migrate content, I would encourage you to check out Isekai , which is a project which Mitch, my colleague, just open sourced recently. It's an ATL style framework, very nice And the co-snippets in this presentation were rendered with carbon. now. sh, very nice. If you're adding the snippets to slides, that's that's A good thing to use. And that's me, thank you.

Speaker 1: Thank you, Alex. Do we have questions in the chat or in the Q<unk>A section? I did see one question from or uh organizer uh megan it's a bit of a long one so i'll hope if i can uh paraphrase this right um Could this be used to create a static fixture file? So if the data is transient and only runs in test. Then it isn't test data to show editors and give them a visual idea of what the page looks like. It would be handy to have a way to generate some consistent test data for people to use. with a project that can be loaded in. So more like can it be used to create a fixture file?

Speaker 2: Yeah, like sample data. Like example data. Yes, yes it can. I was sort of hinting at this with uh Yeah, when I was mentioning factories here, you can have like a this n a nice big command, management command with factories uh yeah, creating uh richer content. Um you can utilize the fact that uh factory boy can generate random like titles and uh content like even lipsum test text many paragraphs and um yeah that that works that works And I wouldn't export it as fixtures, I would just have it in a management command and run it to generate the content. And if you do that, don't forget to either delete existing content.

Speaker 2: So you don't create duplicates or upsert somehow, but I don't think factory will help you there. Yeah All

Speaker 1: right, thank you. Any more questions

Speaker 2: I think Josh mentioned yeah. I think Josh mentioned basically the same thing and I think we were working on the same project where we were doing that, yeah. And also yeah, Pikes for the win, I agree.

Speaker 1: All right, well I have uh a question then. Um you mentioned something about how you did not use the test setup that came with the uh uh startup project in Wagtail Is there anything we can do there to improve it?

Speaker 2: Yeah, I think so. So let's yeah, I think I can quickly bring that up. This is what it looks like. And I think well first of all I don't think this works Reversing on because home is not a route name. We need to get a page URL here somewhere, somehow. Also, this is creating a page in void in in a void it's not connecting it's it to the page tree like it should be uh oh sorry it is it is it is um

Speaker 2: But it's not using the exist so there's it's not I don't think it's connecting it to s to a site. I think I was looking at this instance of the page and there is there's no site. So it's it can it can't actually be served by Wagtail, I think. Um so yeah, I I think we can fix this without too much trouble. And I'm not sure about the tests here whether it makes sense to create a root page and uh Just make sure you can create a home page. I mean you usually want gonna want to use a test client to to check that page renders. That's I think the what yeah what you usually want to test for. But anyway, we can we can discuss

Speaker 1: All right, well, it sounds like a good topic to bring up uh in an issue. Um Mesu's talk was about uh how small even small things can have a big impact, so this might be one of them

Speaker 2: I shall open an issue.

Speaker 1: All right. Well, I don't see any more questions in the uh chat so maybe we shall leave it at this

Speaker 2: Thank you very much.

Speaker 1: All right. Thanks everyone.