Internationalization in Wagtail – Brady Moe

Speaker 1: So um my talk today is on internationalization in Whittail. Um my name is Brady Mo I am a senior Django developer at the Motley Fool. I have about four and a half, five-ish years of experience working with Django in the last three or four with Wagtail. I was scoping out a global website for a client at my previous job, and that's where I had to figure out all the internationalization pieces of um of Wagtail. And fortunately there was really good documentation in Wagtail 2. 7 docs, which have since disappeared for reasons. And uh moving forward

Speaker 1: the it looks like all of that is getting moved into a new library um but this was my scope of getting everything um multilingual and multi-regional so um So why should you internationalize your website? Well, every year there's more and more people getting online. The data that I had came from Wikipedia. Um, and so I don't actually even know how they were defining developed world in this case. Uh But uh the data only went up to 2017 and you can see on the brown uh the brown graph there uh that it's almost 50% of the global uh population that is online. Um so It's pretty safe to assume at this point that you probably have about half of the world online

Speaker 1: today, which means if you want to reach more and more people, you probably want to have their language, you want to have region-specific stuff. just so that the experience that they have is more akin to what you wanted them to see. So yeah, and the graph it it's this is a shoddy representation of it, but it does go up and to the right, so it kind of proves my point just by the graph. So yeah, some of the problems that you have to solve in terms of internationalizing a website is translations. So what do you just how do you decide what needs to get translated and what doesn't? Something that gets missed a lot is alt tags. Accessibility is a huge thing. It's a huge topic across the world right now.

Speaker 1: So making sure that your alt tags are translated is huge. Regionalizing the website, do you want to serve up the exact same content to people from different parts of the world? Or do you want to just serve up the same content just translated to that user? These are some things to consider, just knowing as well that sometimes you like a sentence that you might say in your piece of content in English. might mean something totally different just because the tone can change. So having regionalized content is important. And then the third bit of this is if a user comes to your site and They're trying to, you know, uh a friend shared the URL or something. Um do you want to be able to auto-detect where they are and then bring them to the right

Speaker 1: URL based on their language settings or you know where they're located. So these are some of the things that I was tasked with solving for my previous client and let's jump into it. So This is a quote from Mark Twain. The difference between the right word and the almost right word is the difference between lightning and lightning bug. I don't think Mark Twain was talking about translations at all when he said this. This is actually more so just a representation to me of why language is super important because it can be the difference between talking about weather and talking about an insect. So So translations. You know, I've mentioned what the right things are to be translated,

Speaker 1: but some things that can be easily missed, alt text for images, help text in the admin. error text also within the admin or even on your form fields. A lot of times that this stuff can get mixed or missed because it's either static or it's not in the admin for the user to be able to modify it. So having a list of things that you actually need to translate and making sure that you're checking off those items as you go along is super important. Because otherwise you're serving up mixed content, which doesn't look good. Um so this is how um to get stuff like uh ready for translations, at least static content. Um

Speaker 1: So I'm a big fan of Ginja2. So if we're looking at how to set up Ginja 2 for translations, you have to add the you get text lazy option within the environment globals. So that you can actually put, you know, the underscore before the text. And then that lets when you do uh compiling of messages in Django, That lets Django know, okay, this text needs to be translated, and then you compile that. It get it gets you PO files, which is a little out of scope from what I'm for what I'm talking about, but I still wanted to mention it. And then also with Django templates, I just wanted to quickly show how you would translate that. And then what does this look like in terms of like modifying an image block within Wagtail?

Speaker 1: So on the image block, One of the things I mentioned during Thibaut's talk was actually translating alt text because it's a it's a I think it's super important to have alt text translated for screen readers so that International users that might not speak English can still understand what's going on if they don't, you know, if they can't see the screen as well. So image alt text is something that I add. So anytime that we use an image block within a stream field, this is the image block that gets used The reason that we have the alt text right next to it is so that you're not getting a different image per language because then your image library is just going to blow up. So what this does is makes it so that you can still reuse images, but then have the alt text served up separately.

Speaker 1: And then this is just an example of what the uh under the help text would look like if you're going to put the underscore before it. And then the same with an error message. This is just overriding the clean method to make sure that the alt text always exists within the image. So that's the image block. As of this writing, the duplicate page tree approach is the most talked-about uh approach for translating things. In Wittale. There's another approach where you can actually override the tabbed interface to include different languages at the top, which is also a super fine approach. It wasn't the one that worked for our needs. Which I'll get into later.

Speaker 1: But what happens with the duplicate page tree approach is you essentially have this image here on the left where you start to you know, your what would have been your home page originally or your root page turns into just this slash, which does a language redirection page. So then the second that a user comes to your site, they're redirected to either English or French or whatever languages you support and then English French whatever those language codes are those are now your new home pages um And just to keep into the standard, you need to have those slugs for the home pages, B E N, FR, or language codes, which I'll also get into in a little bit. One thing I did just want to note too is that in part of the documentation from Wagtail, again, this is super old because it's from Wagtail 2.

Speaker 1: 7. Um they mentioned having the URL patterns include Wagtail URLs, but I did find that just getting rid of that actually was a better expensive experience mostly because Django will do the routing for you if you include the IL uh I18N patterns. So removing that was why I had to do that because we're already doing language redirection. And then this is also just directly from the WayTel27 docs about setting up the language redirection page, which does a lot of the work for you. So yeah, the next piece is regionalization. So regionalizing website is serving up content based on where the user is from. You know, for product content websites, this is this is really important.

Speaker 1: Um you know, for example, if you are in the US, you might want to just serve up hamburgers. But in Canada, hot dogs are really hot. So you serve up hot dogs in Canada. So that's the type of regionalization I'm talking about, like on your homepage. So your homepage might show hamburgers to Americans and hot dogs to Canadians. If there's anybody that's Canadian that doesn't like hot dogs, I'm really sorry. So um But for us to get all of this to work, we have to change things up a bit. So before we had that simplified root structure where we had root and language But what we need to do for the regionalization was actually extend it to have different regions.

Speaker 1: So USA, Canada, Canada, USA, French, Canadian. in Spanish in the US. So now what happens is we also have to introduce something called a region redirection page. So we still have our language redirection page which will redirect people, our users to to their language, but then on top of that, we also have the region redirection page, which will redirect them to their region based on some like specific signals that we set. So it's a little confusing because the language redirection page doesn't actually have a language associated with it, but the region redirection page has a language associated with it. Because the region redirection page actually now lives where EN, FR, or ES would live.

Speaker 1: So there's a lot more to the region redirection page because now we've added additional complexity and we've also increased what the duplicate page tree structure would look like. By a lot because now we're duplicating the page tree structure on a per region basis. And then we also changed the language redirection page to include a Just a simple little thing to grab the user's site language based on a cookie that we may have set. And that makes sense because if a user comes to our site and then selects their own language, like let's say in their browser they have a language set to French, for example, but they do like browsing in English

Speaker 1: for whatever reason. We still allow the user to go back and change their language, but then we want to set it so that the next time that they come to our site, They are served up with the language that they chose based on the preferences that they have. And then the next slide here is the region redirection page. So there's a lot here. Within the region redirection page, I've included a clean method that changes the slug to the language that has been chosen. Um that's in part because I didn't want to have our site admins thinking about what language codes were. I I didn't think that our site admins have to had to know what EN stood for or ES or JA or FR or any of those things.

Speaker 1: So They select the language by looking at French, English, Japanese, whatever, and then we auto-update the slug based on international codes there. And then This get context method was actually just used for a lot of debugging, making sure that I'm actually getting the IPs that I expected. And You can use it on, so the region redirection page actually ends up also serving up my language chooser, language slash region chooser. So you could actually show the user where we think that they're from and all these other things, but all of this git context could easily be removed. Also, if you're if this is like way too tiny or you're trying to write all of this down, I do have a link that I'll be sharing at the end

Speaker 1: that has all of um the code that I'm showing today um available and it'll be on my GitHub. So further, we want to modify the serve method on the region redirection page to also redirect the user to where they're supposed to go. So if we detect that they have um that they're from a different region. Like if somebody is hitting our site from the UK, we want to redirect them to the UK version of the site. Or let them just go to the link. If we can't detect their region or we're not supporting that region, we want to bring them back to the language chooser page so that they're able to actually choose their language slash region.

Speaker 1: And yeah, we do that by, you know Doing an IP lookup through um GOIP, which is a Django library, um, and I have links for that as well. And where you're getting where we're getting the IP database from, it's a Max Lite DB , which is free. Well it was free. You have to sign up for it now. And I think if you're using it for commercial purposes, you might have to pay for it. Anyways, that's that's kind of what the region redirection page looks like. So there we go. Um and you might be asking me, but Brady, that only handles people coming to our website from the base URL. What about links and such that might be shared? So what I mean by that is

Speaker 1: um Let's say you have a friend in Japan and you share an English um an English URL. They can probably understand it, but then they share a back with you um and it's a Japanese link so but you don't speak Japanese how do you get back to your site how do you how do you even know where to go um well We do auto detection. So auto detection is where we're actually setting up like a middleware to check where the user's from. So There's a couple different places that we're checking where the user's from, but this auto detection piece is really just to make sure if somebody gets a Japanese URL, they get a prompt or a modal at the top at the top that shows Hey, we see that you might be from a different region

Speaker 1: or you might speak a different language. So if you want to change it, let's show the prompt. And then that also makes it so that the prompt is in the language that they set they have set on their browser. So let's take a look at that. So this is the location prompt middleware that I wrote, which it's just getting the client IP based on the IPware library, which is a Django IPware library. And we check their IP, uh, we check it against the IP or the Geo IP database And then we set on the request object the detected middleware and the detected region. Just so that we can later on in a modal

Speaker 1: grab that and if those are different than what the site actually is at like set at in terms of like if it's English US. Um and they're like if we've detected that their language is English and their region is US, but they're going to um Japanese and Japan, then we'll bring down This lets us know that we've detected a different region entirely. So we can use that in our prompt. So there's some more code here. So these are some mix-ins that I used on each and every one of the translatable and regionalizable pages. These are mostly just for getting around and also used in the templates to uh make things easier for me to be able to get around.

Speaker 1: So um making sure that we always have the language homepage. And the region homepage accessible to any part of any template from any page is important so that we can go to the home page from any other page. And then I also have get alternate pages on the regionalizable page, which allows for people to get if They want to go to a different uh the same page but on a different language or region. This is how they might be able to get around to it. I didn't end up actually getting to use this a whole lot when I first created this this internationalized thing, um, which was unfortunate, but uh This was kind of my last project at the company I was at before the Motley

Speaker 1: Fool. So um so even more code. Um Just because I wanted to talk about the copy piece of this, the client that we had wanted to set up the English US version of the site completely, but they didn't want to have to do a bunch of content re uh re-entry for the other regions that are also speaking English. So we had to set up some some specialty functions around that. So first off, I didn't want my client to have to think of region codes. So whenever we copy a page, we can't have the same region code because that field is set as unique. So before we copy the page, we actually grab the region, we see any regions that are available, and then just randomly set one of the available regions to

Speaker 1: the new page, which has some downsides. It means that you still have to go back into the page and reset the region at that point, but it's a required field and needs to be there. This also makes it so that the user isn't able to create more than um more regions than are what available than are what Then are available in our settings. py. And that's important because a lot of that, a lot of the Django stuff is dependent on the languages and region settings that you have in your settings. py. So That's basically what the before copy page is doing. The after copy page signal, this was because we needed to get around the fact that when you copy pages.

Speaker 1: Any place that there's a page chooser field, it's going to copy the reference from that page to the next page as well, which we didn't want for our copying, because if we're copying something from English US, we don't want um Like if we copy English US to you know English German, we don't want to cut make it so that every time somebody clicks on one of the German links, they're brought back to the English US site. So This is going through the stream field and updating those references. Um, and we can assume that the page slug is always going to be there because the um Because we're copying it.

Speaker 1: So we know that that page is going to be there and we're copying that entire tree structure. So it works out that way. And then this can be extended quite a bit as well to look for any page chooser and any block type you have. This is the simplified version that I have. So um and with all of that, did we just make an internationalized Wagtail site? Um Short answer is no. Uh the long answer is sort of. Uh we've got all the building blocks in place, but there is so much more work to do for the site admins. Um I kind of touched on this a little bit with the um like the copying and everything, but uh this puts all of the work into the site admin's hands, which is great for devs because we don't want to deal with it.

Speaker 1: But the um I don't know. And increases the amount of time that it takes for content entry. So Um, I did this on my personal website. I actually got everything set up so that I can show for a demo here in a minute. But the Even just the two pages that my site is, it took a while for me to fill out all the content and get everything translated and all of that. So So yeah, and then the other thing I just wanted to mention is there's a lot of room and uh room for improvement in what I've done. Um this was a lot of just first attempts at getting everything working, um, regionalizing and internationalizing.

Speaker 1: Um, I think there's some optimizations that could be done. And um yeah, I I'd have to look at it a lot further, but there's also a lot of work being done in this space right now. I know Um Mozilla is funding a lot of things with Torchbox to get this working. And even just in the 210 stuff that I've seen so far, it seems like there's a lot more setup that's coming for internationalization, which is really cool. So here's some additional resources/slash learning. I talked about uh Waitail Trans a little bit, which is another library that's solving some of these problems. Language codes and region codes, they're very specific. They're set up for a reason. So those are some of the resources that I had for looking for those. I wanted to link the old docs.

Speaker 1: These don't exist anymore. They're not being updated, so your mileage may vary. Geolocation, that's how I figured it out, was based on the Django docs, the Geolite DBs, which is the IP detection piece of this. And then that's a link to the snippets for my presentation, which also contains a link to all of those. So thank you. Wagtail has been such an awesome thing to work with. And I can honestly say that most of my career uh is thanks to Wagtill currently. So uh I just feel really privileged to even be able to give a talk today. So thank you very much.

Speaker 2: Thank you, Brady. That was a really great talk. And uh there's some questions flooding in. Um Uh let me Are you okay to answer some questions?

Speaker 1: Oh yeah, yeah. I am super happy to answer questions. I also didn't go over the demo of my site, which I can quick. Um just show um actually so as you can see I'm at my hold on let me extend my screen a little bit here um So this is my my personal website and you can see if I go to Bradymo. com, it automatically redirects me to English US. And that's intentional. I'm in the US and I speak English. So if I go to an incognito window though, and I go to Bradymo. com slash S slash US. Which is Spanish US. This will show the prompt that I have. So this is the middleware prompt where it's actually detecting that I'm in the wrong region or language.

Speaker 1: So then I can actually just click change language or region, come here, it's all in my language preferences, and then I can go back to English US, which is where I'm supposed to be. And then we have selected equals true up here in the query strings, which does its own setting as well. So yeah, that's that's just like a quick demo of what that looks like on the front end. And then I can just click show, whoops. Um because when I showed this to a colleague, she wanted to see how easy it was to use in the admin or how hard it was. So This is the basic setup of what it looks like when we're in the duplicated duplicated page tree approach. So these are all the different languages, and then you jump in.

Speaker 1: to any one of these if I can click in there. Um there we go. So clicking in and then then now you see all the regions. And then so these are these serve as the actual home pages and then serving going into there. So all right. So yeah, that's everything. Uh I am happy to take questions. I'd love to answer as many as I can.

Speaker 2: Okay. First up from Kuhn. With duplicate tree, how do you handle untranslated pages?

Speaker 1: Handle untrunky untrunky.

Speaker 2: Untranslated pages.

Speaker 1: Oh, untranslated pages. So um Right now, there's no good way to do that. Um, I did not solve for that problem, um, which that's because the client that I was working with is going to, with the duplicated page tree approach, there Before they ever publish even one of those new languages, they want to go through and translate every single page. So a page doesn't get published until it's translated. So that's how we solve that.

Speaker 2: Great. And what about can you still use the redirect feature? Does that have any implications?

Speaker 1: The redirect feature.

Speaker 2: Uh Kuhn asking the question, but I think you know Magdal has this redirect feature which allows you to to handle redirects of specific URLs.

Speaker 1: Oh right, right.

Speaker 2: So I guess the question is is that How do you make that multilanguage aware?

Speaker 1: That's a good question. So I think you'd have to overwrite some pieces of that. Because it would get in the way a little bit. I'm trying to think of how we solve for it because I know that the client was going to be using that feature. But I don't I don't actually know that we even solved for that. So that's a really good question. That I don't have a good answer to.

Speaker 2: Andy asks, are there any tools or best practices for finding text that hasn't been internationalized?

Speaker 1: So I'm assuming that that means for like static text. And uh no, I the I don't actually have a best practice or a way to figure that out without like a big global search. And so my recommendation would be that, and I know this doesn't work for most people because a lot of people have sites that probably That have been around for a while, but if you're building a new site, it doesn't take much effort to add trans around any static piece of content. Wagtail has done that all over the place. Like basically every static piece of content Wagtail has a translated block around it. So if you're building a new site and there's any semblance of you maybe making it multilingual, I would encourage you to just put the trans stuff on it right away.

Speaker 1: That's I think that's the best approach.

Speaker 2: Got it. Um Matt asks, how do you handle things that aren't standard pages like menus?

Speaker 1: So from menus, um I put all of the menus into model snippets. So every piece of content is managed within the database, even menus. And then we're grabbing those um that menu stuff. So whenever you create that static, that new piece of content. in the snippet, you're actually picking the language that it's supposed to be for. And the client that we did this for, I want to say we only had a dozen pieces of static content. Otherwise everything else was in the database. And this is a this was like a thousand page website that we were building. So that's how we handled it was to put it all on the admin user.

Speaker 2: Got it. Uh and that there's a lot of praise and thanks which I won't pass on now because you'll read them later. But um there's also a question from Paul who says I had sounded that in Wagtail 211 there might be a translatable page mix in solving some of those issues. Is there someone who might know more about this? And I could, if it's all right with you, Brady, I could invite Carl. to uh to respond to that one who's the the the person behind these changes yeah

Speaker 3: hi uh yeah so there's a rc on the rc's repo number fifty That goes into all the details on what we're implementing in Wagtail Core right now. The other part, the Mozilla work, is implementing a sort of translation management as well. So that's going to be like a third-party module. that will plug into all these core changes. Um hopefully the core changes shouldn't interrupt anyone who's already um using internationalization, whether that's with Wagtail trans or or model trans or a custom uh implementation, it should it should It should uh all the implementation that I'm aware of, and I think this one as well, uh, it should it shouldn't conflict with anything. Uh but yeah, it should yeah. If you want to add it in

Speaker 3: in the future, then that's definitely probably going to be the best approach.

Speaker 2: Thanks, Carl. Carl also points out on the channel that there's a um is a hash multi-language, hash multi-hyphen language channel in the Wagto Slack where we're discussing some of these issues. But uh with that, I think we're out of time. So thank you very much, Brady, for an excellent talk. Um the slides are going to be available, I think, or certainly the recording of the video.

Speaker 1: Yep. And yeah, I'll uh I'll also share that GitHub link in the chat as well, just so that you can look at it. Um I'm not gonna change anything about it. So uh if it's bad, make a pull request, please. Thanks.

Speaker 2: Thanks, Brady