Wagtail in the News Room - Sævar Öfjörð Magnússon & Arnar Tumi Þorsteinsson

Speaker 1: My name is Saivar.

Speaker 2: And I'm Timmy.

Speaker 1: We are with uh Overcast in uh Iceland. Uh we've been building Wagtail sites for Since version nine uh 0. 6 or 8 or something. So uh it's been uh while. Uh we've had had with Wactev. We wanted to uh show you uh what we've been doing with uh news sites uh specifically in Iceland. Some of our clients are news organisations so and they are a bit special in some ways. So we just want to share our experience with working with them. Uh yeah this involves ads, so uh sorry about that. We're sorry about that.

Speaker 1: Uh so news organizations they are uh Like I said, bit different from our regular clients. They are really like fast moving, uh always reacting to like events and uh uh news in our society and what's going on in the discourse. They are usually working under a lot of pressure and they're also usually understaffed and Have no budget really. So yeah, the and also in Iceland the journalism landscape is uh still kind of uh behind, for example, Europe and uh the US because uh Print media is still quite big in Iceland, so they're still kind of trying to figure out how they uh how they should be moving to

Speaker 1: uh you know online news To us, Wagtail is of course the favorite tool to build websites, so that's what we propose to our prospective clients. even if they're news sites or not. Uh in the world of journalism specifically, Django is of course uh originated from uh Two developers had a newspaper called Lawrence Journal World. That was Adrian Holobati and Simon Willison. Uh yeah, the rest is of course history and the reason we're here today. Django specifically describes itself as the web framework for perfectionists with deadlines.

Speaker 1: Yeah, perfect for uh news organizations. Uh so uh Yeah, we run websites for uh three organizations in Iceland that have news sites, uh some of them more than one. And uh yeah, we just wanted to present with you with some challenges and some of the solutions that we've come up with to accommodate them. So one of the challenges is that uh The frequency of new content, it's just constant. You know how it is with news. It's just you know it's constantly uh a storm of news uh hitting you So we have content creators

Speaker 1: that are writing like multiple pieces of articles every day and multiple reporters. Here are two screenshots from uh two of our clients. So they have like 170 or 160,000 pages. One of them has 356 ,000 images, 11,000 documents, 12,000 almost. So it's a lot of content to work with. And most of it It's only read like for uh one day or one week and then it's just uh archived really never to be seen again. Uh And so, yeah, these journalists are really under pressure to deliver these news fast and reliably. So uh uh

Speaker 1: Some some approaches that we have had before with building sites in uh Blacktail is uh build a layout using stream fields. And then you know adding content as uh stream uh blocks or whatever, you know what I mean? Uh linking content into the blocks manually. This is not an option in this environment because uh They just don't have the time to do it and it's error prone and and they might forget something up there and two people are working on it at the same time and uh something gets left out and yeah, so it's error prone.

Speaker 2: Uh someone has a breaking news and then the next guy just goes over it.

Speaker 1: Yeah exactly. And also because we always have something new uh uh being uh reported, uh caching is uh kind of a problem uh because you know you you don't want to have an old version of your of your site online. But we still wanted to have the flexibility of building the layouts in the stream field so uh Yeah, so that presented uh kind of a problem for us, uh how we would approach this. Uh uh and we still wanted to allow the content to just fro flow three freely through the page and not have the editors manually linking content or selecting something from clicking buttons. The sections of the site, possibly sections within a front page or a category page.

Speaker 1: uh would have to be scoped to their respective categories. Uh subsections of the website should behave exactly the same. So And finally, uh articles should not be displayed twice in the same page, of course. So the solution that we came up with was uh We use a set of uh kind of custom template tags that uh query for news items. So each uh block knows how many articles it wants to render. Uh it can uh take into uh account uh if it should render a specific category or if it should highlight a specific item.

Speaker 1: And we do all this. We fetch the news, we do the bookkeeping, figure out what should be excluded because it has been used before, if it's the right category, etc. In the end, each block knows how many which items it should display and we know which items this block would display, so we will exclude them in later. part of the rendering of the page. So we still have the flexible layout building And we don't need to like manually add the content. It's just you just create the article and you forget it.

Speaker 2: We'll show you a little bit about this.

Speaker 1: Yeah, we'll have a short demo for this uh at the end. Another problem some of our news site clients have is uh that they stream a lot of videos. So uh Of course streaming videos is uh extremely expensive uh if if we uh use for example one of the available packages or or uh most of the available packages for Wagtail to store media, we would have to store it in like S3 or on the server or whatever. This becomes really expensive really fast. Of course you could uh put your CDN in front of it, but that's still expensive. So what we

Speaker 1: In our research we uh found out that uh Cloudflare has the best prices for video streaming in the industry. So I think it's uh

Speaker 2: Yeah, it's like ten times

Speaker 1: Yeah, it's like a factor of ten times cheaper. They also will include a ready-to-use video player out of the box. So what we developed was a solution in uh it's like a small wagtail package that we developed where you can uh upload a video directly to the back end, but it will bypass all uploads uh mechanism mechanisms in Wagtail and we'll just upload directly to Cloudflare, receive a token back. We will use that to uh of course render the player that Cloudflare provides. So that was uh Yeah, really good solution of that problem.

Speaker 2: Yeah. Then there's a little bit about the image image filters. Uh when we when we were starting out with these new sets uh we had you know not much we didn't know much about the image filters um how they worked um but we were forced to do some work on them because um the the news websites they wanted uh their images to be shared on social media um but when they were shared on social media they also got stolen from other websites. So uh what they wanted was um to bake in their logo into the share images. Um and And then you know when we had finished that

Speaker 2: they wanted to do more about more uh stuff with um with the images like uh color overlays and stuff like that so that it would be color overlaid when it like it's Facebook or or some other place. So uh we are now able to just add new uh filters to the existing image uh tag uh so we can get like for this one uh the watermark then we have a an image just as a general site setting. So every time an image has to be rendered with a watermark, it will just fetch that image, calculate the size you know respectable to the image, and then pay it in

Speaker 2: so that we have a rendition basically that has this baked in. Another example was um When they had like opinion pieces, they wanted them to be like colorized by the type of like a category of opinion pieces. So then they could, you know, have the authors colorized and then that image would be shared with the article. I will do a little lightning talk about those filters and there has also been quite a lot of work done on that during the sprint this week. Um this is another that's you know maybe not so relevant for English but but um

Speaker 2: this has This is uh important for us uh because we are a tiny nation and we have our own tiny language and because it's tiny, you know most companies are not focused on like developing things in Icelandic or or supporting Icelandic. And English is dominant in the technology that we use. So uh the Iceland government uh uh now has a like a language tech program where they're like you know getting the companies to include Icelandic and we are trying to present more technol technology to uh people. can support Icelandic.

Speaker 2: Um so we did a little bit of spell checking work. Um Baku doesn't have a specific plugin for spell checking, uh but it is possible to extend Draugh Tail to do this. And there was a research project uh in the University of Iceland and uh an Icelandic language technology company that's been working for a couple of years. And they wanted to enable spell checking on Wagtail sites. There was a a small uh project that uh Hinrik Hafstenson led uh which was a customization uh in trap jail Which um creates

Speaker 2: like a small spell checker. So you have like a button in the toolbar so that when you've uh when you finished writing your Text, you just click it, it will then take the text, send it to the spell checker engine, and it will get a lot of feedback back underlining all the suggestions and also like real solid reasons for why it's such like explaining this suggestion. So then the editor can either accept them or decline them. Yeah, so this is work that's like ongoing but um it's already been used being used by at least one or two of these new sites

Speaker 1: Yeah, it's not really ready, but uh it's yeah, somewhere along yeah, it works.

Speaker 2: And then ads. Sorry. I guess Um as we mentioned before um we we develop ad tech solutions as well um for the Icelandic market and AirServe is our ad serving platform. We're now serving out around 600 million impressions a month, which is quite a lot for a nation of 370,000 people. And most of the ad agencies are using our solutions. We have in the last three years been working on a version of this for the publishing part, so for the websites themselves, not just for the uh agencies. And uh we have developed a little package that we use um

Speaker 2: to interface with Wagtail so You can actually like just get an API key and then pull all the add zones from from our ad tech uh software into Wagtail and then you can just like select it into zones on your website. Um yeah. So you can see I'm gonna give you a little demo of uh how this works

Speaker 1: So here's the premier news outlet for Wagtail Space, of course.

Speaker 2: Wagtail News.

Speaker 3: So we're gonna preview the page. So this is a like a basic template built up uh using stream fields. Uh maybe you can show us uh how you lay out it Yeah. So you just uh create a newest block. It's nothing remarkable really. We have a few layouts defined, like how you want it to appear. You can preview it. So now we have uh one uh featured news item on the left And all the other news just get rearranged really through the rest of the blocks if you change something.

Speaker 3: So if you feature an article in uh in this block because you want to highlight it. So you notice that there's no content there really, it's just It's just referencing a template. We're gonna highlight uh Google Summer of Code So that's been moved up to the front and the rest just rearranges, which is nice. I think it's missing uh it's missing an ad. So we pull it out of uh the the we have an ad block, it's rendering the snippet which is pulled Well, create it from the API that is communicating with uh the ad

Speaker 3: server solution that you have. That's all, yeah.

Speaker 2: Yeah, so this is just a little and yeah and we'll do a little bit on the filters later on. So

Speaker 1: I think that's it for now. Thank you.

Speaker 2: Thank you.

Speaker 3: We do have some time for questions.

Speaker 1: Okay.

Speaker 3: Can I get

Speaker 1: All right.

Speaker 4: Yeah, the question you referred to editors earlier because of the well you have a newsroom people using this this CMS to Create the news. Was there a reason or was there one specific question that you had that they didn't have any time or the knowledge to use the interface to just create the pages all by themselves Or maybe the question is, what's the most annoying thing they mentioned that they didn't want to do which they had to do otherwise it wouldn't be possible?

Speaker 2: Um I would say that um Uh when we started this it was it was basically l like having to lay out the page every time because uh by default you kind of have to fill in your stream fields with data or or have some um Like some rules for it, but the problem is that the because the blocks are like they are, they don't really know what's happening around them. So if you say, you know, give me the latest three articles um you'll just get the related three articles in all of them. We had a lot of problem with like oh sorry. Uh so we had like uh uh quite a lot of of of duplicates. Even even when we were like fetching stuff from different categories and different stuff you always had one here and there that was duplicated throughout the whole thing.

Speaker 2: So as soon as we were able to kind of pipe the context through the stream field, then we could start doing like bookkeeping of what was already there Uh which makes it a lot easier for example if you have like a sports section very like very low on your side and then you have a huge piece of sport news at the top, then it just kind of falls off down there.

Speaker 1: And I also think that they had problems with the you know content becoming stale because it was a lot of work to maintain it. They always had to like uh regularly check, okay, are we outdated or not? They don't have to anymore, so Uh

Speaker 5: with 170,000 pages, did you have to do anything like specific with search or filtering to empower them to find the right pages? Because uh I know the page tree is not gonna be that useful at that scale

Speaker 1: Yeah, this is a constant problem. We use Elasticsearch, but it's uh it's not optimal. Uh it's really hard to sit keep in sync Uh it's uh resource uh intensive and uh yeah it's we're always trying to find some better ways to do it, but uh Elasticsearch works.

Speaker 2: Yes.

Speaker 1: Uh quite good, you know, for most of our cases.

Speaker 2: Uh one thing also we uh instead of having all the pages in like one flat tree, we have like these like uh super categories like news and people and and and events and stuff like that where people like uh write the the uh article enter And then we have like categories for each of those. So we can and then we can scope those. So you can say I want this block only to have that subcategory So you could go all the way down to just Icelandic football or or something and you would just get articles that are tagged with Icelandic football. Cool. There was more

Speaker 4: question over here. There you go.

Speaker 6: Yeah, I've got a question about the cybersecurity challenges for developing news sites. How were you able to prioritize that for the journalists, especially in this uh in this generation with uh media freedom

Speaker 1: Can't remember that we did, but uh Uh I mean uh uh well we are fans of Django uh to begin with. Uh Django is uh I in my opinion a very uh secure framework to to start with. So uh Um we have not really made any special uh arrangements for cybersecurity, but uh we do use uh Cloudflare for uh caching. We have sometimes had to use I mean we had uh like a denial of service attack at one time, so then we had to use a uh like a uh turn on some uh firewall features in uh in uh our Amazon cloud uh account but uh

Speaker 1: apart from that we have not had many problems with with this

Speaker 4: I'll be interested to talk with you after uh