Better managing i18n and PO files

Cool, uh I'm Felix. Uh I'll be presenting better managing 910 uh PO files in Django. Uh okay, so let's start. Uh the agenda for today, I'm gonna tell you a little bit a little bit who am I, uh what is the ATN and localization, uh what is the current workflow in Django and the proposed workflow and the one that I work on. I'm gonna demo this and then we'll be summarizing what we have learned. So who am I? I'm Ecuadorian. I'm based in Quito, Ecuador. I come all the way from South America. I'm a web developer, the stack builders, mainly focused on back-end technologies. My core languages are Python and Haskell, and currently working on a Haskell project

I'm a community lover, so that's why I'm here and that's why in Quito, back in Quito I uh lead the Quito Lambda initiative that is mainly pro uh functional programming related topics. And a functional programming enthusiast and a sneakerhead. I've been PyConLatan a two times a speaker at PyConLetam. And it's my first time a Django con in Europe and yeah, so and my first event in person also. Uh that's a picture of my city that I took some years ago, like six or seven years ago. Uh still good, so I love it And maybe you know my country because of the Galapagos Islands. If you hear of that, that's Ecuador. And yeah, so let's go Uh what is INTN and localization and what are the differences?

Uh these are like mix of terms, so I wanted to clarify before starting. Uh these definitions are taken from the Django official documentation. So internationalization is preparing of software for localize localization and is usually done by us, by the developers. And localizing something is writing the translations needed and the local formats, and that's usually done by translator. So just to be 100% clear, we are in the internet signalization side of the things Okay, so how's done Django? Uh I 'm gonna focus this talk on static uh templates and translated static pages. So this is how it's done. Uh a developer must tag all the strings that need translation. Uh we'll be loading the uh internationalization tags in our HTML files.

I do can see then we can use uh a couple of uh uh uh keywords that we have like translate translate, broke translation and stuff. Uh then uh with a couple of uh CLI commands we will run and uh Django we go through these uh HTML files, we'll extract the target strings. I put them in a file that is called a PO file. So what is a PO file? And before continuing, uh how many of you have done internationalization in Django? Oof. Almost everyone. So yeah, well you know what I'm talking about then. So PO file is a match file uh that is plain text and it represents a single language in Django. And it contains all the avail all the strings that are ne that need translation

and that should be represented in the given language. And in my opinion, is the standard for translation. It's not like the official standard, but I've seen in many projects that does internationalization that this is the standard. So how is composed? Uh PO files are made of PO entries. That's what they call they have a white space, they have references, and they have a message ID that is the untranslated string. and uh message string that is the already translated string. So uh PO entry to be valid needs at least these three uh these three fields. Um so yeah that's how they are made So what how uh uh what is the current workflow workflow in Django? So if you know, uh we have uh

it looks like this. We have a make messages command and a compile messages. So when uh we have the make messages command, what it does it uh goes through the uh all the templates of the HTML and extract the target uh strings to the PO files. Then we send this PO file from translation or we translate uh ourselves or we use like a tool like D or something like that. Uh we translate them. Uh then we take this uh PO and the translated PO back uh into our base the code base and we compile messages we run compile messages that we create a binary uh the Django can understand and can interpret it and our site will be

uh will be uh translated. So yep, I forgot to pass that one. So uh before continuing, uh the issues that I mentioned here uh started to happen uh A little bit of history of the workflow. We started with this client that has many, many, many, plenty, thousand of pages Most of them were dynamic and they were managed through Wagtail and the CMS. So but many of them and thousands of them still static. So when we started translated uh the this workflow that we talked before this one uh it made sense like uh we didn't have any problem at all we were happy the client were happy Uh the

third party company that was translated or PO files, uh they were happy too. Uh but as time passed, uh around three or four months um that we started, we have like seven different language And like a five thousand pages being translated. So we got uh these PO files were a mess, like five thousand K lines of of PO entries and Uh they were not no nobody was happy. Uh so we we need to took a decision because the the last translation that we sent the whole PO file it took like a month and a half to be translated because of these difficulties. So yeah, uh the issues came uh when the uh the PO file uh size increased So we were sending the PO file over and over again.

Uh we needed uh to wait for the PO file to come back to run uh another round of translation. So that was an issue. We were basically stuck uh until they they sent us back the translations. Uh the PO bias were getting pretty big and the translators were not happy. They uh we uh the job for them was difficult to perform. uh we were not getting things done. Uh and no translation process as I said uh took a long time, especially when it's done manually and you if you have to work with third party uh translation services it could take a while So and we didn't uh we didn't have parallel translation projects uh because of the first project uh first problem I mentioned. We needed to wait uh to come the PO file to come back and then we can send another one

So uh this is the proposed workflow and the one that we implemented for for that. Um for the for uh solving this. So the main idea is to produce a smaller PO files that contains only untranslated or relevant strings and that can be sent in parallel. Uh so before going to the workflow itself, uh let's see a couple of more uh concepts around PO files. So as I mentioned before, we have three needed things that are necessary to have a valid PO entry, but we also have a couple of more options. For example, the strategic comments, these are defined like comments given

by the programmer directed at the translator. And these comments are uh are are called extracted sorry these comments are called extracted comments because the XGTX program extracts them from the program source code. Um so basically we have this option of adding something like a strategic uh comment. The thing is that uh if we did in the code in this uh we needed to go uh every time to go and add uh extracted comment in the code page in the HTML templates. So We didn't see that very feasible. We it was going to take a lot of time. As I said, we have like 5,000 pages being translated. So We decided to use the PO Liv

uh library that is uh using the Django core and is uh how make messages work under the hood. So we took advantage of this library and basically we started adding uh these distracted comments programmatically Another important concept is to the fuzzy flat. This flag shows that a message string or a translation could be not correct anymore. Uh this happens when a string changes uh and the make messages command is able to detect that change Uh but only a translator can just uh can judge if the translation uh is correct or not. So basically if you have the fuzzy flag, uh something is not translated or the translation is wrong

Uh so we took this fuzzy flag as uh and we consider it as uh uh not translated entry. Uh so yeah, then uh we needed to find the definition of an untranslated. Uh what condition should a PO entry fulfilled to be considered untranslated? Uh a message string should be empty, uh the entry is tagged as Fuzzy, uh, or in our case the entry does not contain a project comment. Because if it has the project tag, it will mean that it's being translated or in the process of translation. So yeah, this is the workflow. A couple of steps more from what we had before So we have the same message the same make messages at the beginning and the compiled messages I didn't fill the screen so I have to obvious that but uh the compiled messages is uh has also to be performed and those are the same

Uh well when talking about the make messages comment, uh we needed to monkey patch this one because uh as I mentioned extracted comments are uh lookup in the code but we didn't want to go to the 5000 files and and uh and add those uh those extracted comments. So we need basically to make micmesages preserve these commands uh sorry these comments Uh yeah, so we monkey patched this uh uh this this command. Uh and uh at the top you can see how it's run. In this case we are running the uh in the DE locale that is uh translation from German. So MINT messages uh will give us the PO files, uh same as before.

So here start the the workflow. Uh we implemented these tag messages. We can uh basically say, hey, just take the untranslated entries and tag with a destructive comment with a given name of the project. So we will get the main PO file tag with the untranslated strings. And this can be done by file if you want to translate just one page. or it can be done by subdirectory or even it can be done by a by an app so by a complete uh Django app. So yeah then we is uh We have these track messages that what we uh this comment uh will do is to take the previously previously tagged uh

entries and put them in an uh a new temporary file, a new PO file that just contains relevant strings, as I said. So with these new uh generated POs, uh we could we can have now parallel translations. Because we're not sending the same uh string twice, we are sending one by one and we are sure that they are not repeating uh because of the definition of untranslated that we saw before. So we are sending this for the translation process. The translator only has the relevant strings that will need to be translated. And when translated Uh we need a uh a way to merge back these translated messages to a main PO file. So we have this command called uh merge messages uh

that has a couple of more um parameters and as you can see uh it it has the uh relative path to the previous previously generated uh PO file. So yeah, with the mer merge message command what we are doing is taking all the translated strings and putting them back to the to the main PO file. And then we have finally we have the cling messages command. Uh that we will do this will uh uh delete the comments that uh we added at the beginning. And also we'll delete the PO file generated in the struck messages step. Okay, so it's demo time. Uh is a demo not demo as yesterday, because I didn't want any

and convenient so let's see uh what I'm doing here. I'll be adding um an untrose lead uh a new hopefully it looks Big enough. So I'm adding a new untranslated string to the index. html file and tagging in as and as that one needs translation And as you can see I'm pretty bad at typing. So we will run the the make messages command that I mentioned before. That is the one that has been mocked-patched, as you can see. We have a little more output than the normal MIC messages. We done some we performed some uh more steps. As you can see, D

was added to the Django PO file. uh this new untranslated entry. And as you can see on top of it uh we have two already translated uh strings. So what we need to do now is to uh Tag uh granted tag messages. And you can see distracted comment was added in the untranslated string only and not in the Two above above it. And as you can see also the the console the terminal yells something like one entry has been tagged and we with this project name Uh the next steps uh should be the struck messages, and this will create our temporary PO file. Um so as you can see it was written. Uh we have a new file called PO Project Django Con

Europe 2022. Uh it has the same heater as the original uh PO file, uh but it just contains the only untranslated stream. So imagine that we're the translator, uh we need to translate that stream So I'm gonna put something like hello Django pong. And yeah, my typing is that sorry. Uh in German, because we are in DD location. And sorry, I don't know German. Uh but yeah, so we can perform the mess messages command that has a a couple of uh uh more uh uh parameters but it yells that it improved was successful so yay we have our translation back uh but we still have the the

attracted comment at the top of the entry So we need to delete that. We need to delete the temporally PO file created before. So we're gonna delete that with the clean messages Pokemon, uh Local DE and Project DjangoCorn Europe 2022. And that should be it. Okay, so I have my uh mini Django app running. Um so we're gonna see I think Yeah we're done. Ah sorry. We have to compile the the messages right. So you can see that uh the other two PO files were already compiled and just one has changed so it's compiled again So now we can run the server.

We can go to our page that I had previously. As you can see, the two in the tab are translated. But the new one falls back to the English version since we don't have a translation. But when I reload the pitch, we have the already translated string, so it's working as before. Um so yeah, uh this is the idea behind the the the whole workflow. Uh you can take a look at my repo that contains all the code. uh that is necessary uh to create these CLI step commands and you can check it out and see how it's done and you can just copy paste into your project it has not been yet a library created library Uh but I I create a library or I want to contribute this to the Django

core. Um so summarizing The current Django fur workflow works well for a small to medium-sized project and you don't need to worry about uh taking these uh or over engineering your translation workflow if you are okay with it. So that's the first thing I have to say. Only if you start uh seeing problems like we did, like translation to a month and a half to be translated uh to be done. So then you have to think of something like this. Uh so the proposed workflow also works well and has been proven in real scenarios. As I mentioned before, we have this pretty big client that needed the uh the the translations to be done. So this workflow was improved like around six months and little it is here and little it

is there. So it has been proof and and it works By implementing this workflow, you also will be able to send parallel projects for translation. As I mentioned before, this is one of the major advantages that we have right now Uh we just send relevant information for translators so they can focus on the work they are doing. Uh we don't say just noise. And this can make your your your translation workflow easier and faster. You will not be uh wasting anyone's time. So I think I'm pretty much done with my talk. Uh baby thanks