Eita! Why Internationalization and Localization matter by Nicolle Cysneiros

Speaker 1: Hello everyone. My name is Nicole and I'm here today to talk about internationalization. and localization and uh why uh they are important process uh processes in your application in general Um does anyone here besides the Brazilian people uh knows what ETA means No one. Okay. So don't worry. It's not a bad language. I know that's the common thing to do when you learn a new language, that you learn the curse words first. I don't know what's up with that. But I swear it's not a bad language. It's just an expression that we use when something unexpected happens.

Speaker 1: And by the end of the presentation, you will know why this word is there. And um sorry And why do I have a Brazilian uh uh Portuguese uh word in the title of my presentation? Well, I am from Brazil. I am a full stack developer at Labicodes. I work mainly with Python in the back end, especially Django. And I have some works on JavaScript on the front end also. I am a huge fan of the Python community. I am part of the Python users group and a pilates group in my city, Recife.

Speaker 1: And I also organized a Jungle Girls back in 2017. And this year I had the pleasure of organizing a Python Nordeste, which is like a regional Python in our region. So Brazil is huge and so I flag in this map here my city over there um and it's a very sunny region it has a lot of great beaches for visits so if anyone wants to spend some vacation in Brazil I recommend you to visit Recife and give it a chance to amaze you as it amazes me every day Um and as I said, I work at Labicodes. Labicodes is a software studio based in Brazil. Um and it is

Speaker 1: Um specialized in creating custom web products fitted for specific needs So we believe that software uh should be built by fulfilling clients and user expectations. So our software is always aimed at solving problems and optimizing processes. It's an incredible honor for us to be here, to be part of this conference, to be giving talks and sponsoring Two of my co-workers will be giving talks here. We will have Luan that will talk about that technical debt debt tomorrow. And also Renato will be discussing about jungle authentication on Wednesday. So if you guys are interested in these topics, just stay tuned and look for

Speaker 1: their talks on the next days. Okay, so this is the agenda of this presentation. We will be discussing about the definition of internationalization and localization and why um they matter for your application, why you should know about this. Then I will go over some of the internationalization tools that we have available to build our Python or Jungle applications. And last but definitely not less important, I will be presenting to you how we had to adapt our development flow. To take internationalization into account. So I'm going to share a little bit of our experience

Speaker 1: working with it Okay, so let's start addressing uh the difference between localization and internationalization. Um funny story, I didn't know what I18N or LN uh L T N meant until very recently where I learned that they are numeronymous. Uh which means that we have um There are 18 letters between I and N in the internationalization world. And the same thing for localization, like we have 10 letters between L and N. Just a curious fact for you guys because I didn't know when I started working on this and I just discovered later on. So localization is the process of adapting an application

Speaker 1: a product or even a document to be more user-friendly to customers from different countries and cultures. On the other hand, internationalization is the process of enabling localization of the product on the application. So it's like implementing a software in a way that it will know how and when to show different content depending on the customer's locale. And jungle documentation perfectly summarizes this difference by saying that localization is done by translators and internationalization is done by developers. However, this simplified

Speaker 1: definition of internationalization and localization may give the wrong impression that this is just about translating. This is just about translating your app. But actually this process entails several other adaptations that uh you need to make sure uh that you review them to make your users from different cultures feel more comfortable using your product. So I will go over some of these examples that we need to take care of. Here we have a map illustrating the usage of different formats of date in the world. As you can see, countries colored with CN, blue, green, and grey, they accept the format day, month, year.

Speaker 1: The ones colored in yellow, um, green and gray, they accept the format year, month, and day. And as you can see, very few countries actually accept the and use the format month day year. So let's say that you have a company based in the US and your product is shipped worldwide. How are you going to show to the web uh on the website the expected delivery date to your customer in Mexico? You don't want to flip day and month around. Um it's going to be confusing for them. So how can you do this? This is another map, very similar, but this one actually illustrates the number format used in different parts of the globe.

Speaker 1: The countries colored in green use comma as decimal separators and dot as thousand separators The countries in blue they format the numbers the other way around. So dots are for decimal separator and comma is used for thousand separator So this can be very confusing. So how are you going to show the same number formatted for your American user or your let's say Brazilian user? So it is something you need to pay attention to. Talking about prices, I don't know. Another example is about QC conversion. So if I'm browsing through Amazon, I don't want to see the products in US dollars, I want to see them in my local currency, I want to see them in Brazilian real, that's our currency.

Speaker 1: So it's important to provide to provide these prices and other values correctly converted to your user. Another interesting one is bidirectional text and Unicode characters. It's a very common issue when bringing applications international. In this example here we can see that the Wikipedia page adapts according to the selected language. So here we have the setup in English With the navigation bar on the left, which is pretty common for most uh pages on the web. And here we have the same page but configured for Arabic. And as we can notice, the navigation bar migrated from

Speaker 1: the left to the right side. Because the language is written and read from right to left. So that's why you need to make an adapt an adaptation on your layout, your page layout, not only your translation. And there are several other aspects that uh we need to take into consideration when localizing our apps. Uh there are some countries. that still use in Imperial units instead of the nice and round metric system. So yeah, this is something that we still need to worry about. So you may need to convert some measures on your application. There are some legal requirements that may differ from country to country, for instance.

Speaker 1: Um betting, those betting sites for sports or something like this are actually forbidden in Brazil. So if you if this is your business you may need to consider another route uh instead of going to Brazil. When sorting and presenting a list, the way address and personal names are formatted, so uh should the family name go first or the given name. Also time zone, calendar, special holidays. And I know that we as developers we are so used to write code and documentation in English. that we may not realize that actually there are only 360 million native English speakers in the world.

Speaker 1: And I know that's very useful to have a common language for us to communicate, especially with teams across the world. But regarding your final user for the product that you are developing, English may not be that universal truth for them. So even if the user knows how to read and speak in English, uh they may have learned since the their babies, I don't know. But um it doesn't mean that they wouldn't be more comfortable uh interacting with your application using their native language. In fact, from the top ten countries with most people using internet , Only one is a native English-speaking country. So if you want to grab a share of the other

Speaker 1: market, you must be different. You must bring something new by catching new clients with an amazing user experience. So yeah, I hope that um I gave you enough facts uh to point that uh I could I have convinced you that about the importance of localizing your apps. So Now you must have been wondering how we can do this in our Python programs. So I'm going to review a couple of tools here that can help us with this task. The first one I'm going to start with is the GNU GetText package from the translation project. This package actually offers a runtime library that supports the retrieval of translated messages.

Speaker 1: A set of conventions about how programs should be written to support message catalogs, a library supporting the parsing and formatting of files containing translated messages. So how does it work? This is just a simple hello world type of app. I call it etaap. pywy. Where we are using the get text Python module to create a translation object for our app domain, specifying the locale directory that I called locale And the language that I want to translate my strings to, I'm starting with English from the US

Speaker 1: Then I'm assigning the getText function to uh underscore and flagging this string hello world. Here we have a string that I want to make it possible to translate. After flagging all the translatable strings in my code, uh we can collect them using the GNU X getText command lining tool. So this tool just generates a PO file containing all the strings that we have flagged with that function, the underscore function in our code. So What's the PO file? What this thing that it generates? The PO file, which stands for

Speaker 1: uh portable object file, uh contains a list, it's just a list of entries. Um And this is the basic structure for an A tree. So we may add some comments for translators, some references or flags for the string. And then we have the entry ID, which is going to be the untranslated string that we just flagged in our code, and the entry string representing the translated string. So when I run the command xget text um in the command line passing my eta app file as input, uh this is the PO file that it's generated. At the top we can see that we have some metadata about the file,

Speaker 1: especially handy for giving some information about the project, about the translation process. And down here we have a reference uh for where the code for where in the code that string came from, so line seven line seven of my um eta app. pipy uh file. Then I have my untranslated string as the entry ID in the next line, and an empty string in the entry string. If no translated string is provided for a certain entry, I don't know, you don't have still the translation for that, the entry ID will be returned, so the entry ID will be shown to the user. Okay, so

Speaker 1: I have my PO file and now I can start actually translating the terms that I have collected before In this example, I'm translating our sample app to Portuguese. So I copied the file, the PO file, and generated, generated by the XCAT text command and started to translate it in my terms. So I already put there. Olá Mundo, that's the translation for Hello World in Portuguese. Um one thing that's important to notice is that The GNUGetText library is going to look for your translated PO files in a specific folder path structure. Which is going to be your locale directory, then the language folder. And inside the language folder you must have a lc

Speaker 1: underscore messages folder And inside that folder you will have your PO file. And that's why I had to move around and change my folder structure to look like that. Um now I have two PO files for the two languages that I want to give support in my app, English and Portuguese. And I organized my directory in a specific way that's expected by the library. So, in order to use the translated strings in my code, I just need to compile the PO file into MO files using the message format command for each PO file. So I just need to go over each one of them and then compile each one of them.

Speaker 1: Now I'm able to change the language on my program to as many languages as I have PO PO files for. So I just need to change in my translation function over there. Um and if we run this code, it's going to the string will be translated to Olá Mundo in Portuguese. Another useful internationalization service is the Python Lokio module. This module actually has access to the POSIX Lokale database. And it's very handy for formatting dates, numbers, and currency. Let's take a look at an example of date, number, currency formatting. Use this library.

Speaker 1: Here I am importing the module. changing all the locale to US English and retrieving the locale conventions. Then I proceed to format a given date and price. Just an example. Using the locale. format method, I can format my number without worrying about decimal and thousand separators. I'm just passing the grouping as true, so I want to use the thousand separators in there. Using the %x directory directive to format date, it will display day, month, and year in the correct order of my locale. And from the locale conventions I'm able to get the currency

Speaker 1: symbol for that locale So here's the output for that Python code. We can see that the date is following the format month, day, year. The decimal separator is a dot , y the thousand separator is a comma, and we have the dollar sign representing the US currency. So if I go back there in the same code and just change the locale to Portuguese Brazil and run the same code again. We will get a different output based on Brazilian conventions for formatting dates that are day, month, year And we will also have coma as a decimal separator, dot s thousand separators, and the R plus dollar sign, which represents the Brazilian currency real.

Speaker 1: Okay, now that we know how to translate and format numbers and dates using just Python, I want to show you how Django can make our lives easier. Let's start with translations. So by default, internationalization is enabled when you create your jungle your Django project using Django Anime. The translation module uh encapsulates that GNU library and provides the getText function already set up with the translation object. And the determination determination object is already set up based on the accept language header that's passed by the browser in your request. So all that pi all that Python code that we did before, setting up

Speaker 1: all the translation and assigning the getText function, this is already encapsulated by Django. And we can just jump ahead and just using the getText function in our view. For translations, we can flag uh translatable strings in both Python and template code, um, besides the standard getText function that I just showed you. Jungle also have the ability of lazy translation, so the flagged string, which means that the flagged string will only be translated when the value is used in a string context. such as a template rendering. This is especially useful for translating like verbals

Speaker 1: name or the help text attributes in your Django model. On the other hand, we can use some special tags in our template code to once we load the internationalization tags. The trans template tag translates just a single string. While the block trans tag can uh mark a translatable uh huge block of strings if you want to. So it's up to you. Um regarding the those uh GNU commands command line interface, DjangoDME also provides an equivalent command for those for that one. Um Especially the two that I just showed you. To collect all strings MAC test translatable in the code,

Speaker 1: you just need to run jungla dmi make messages. Command for each locale that you want to give support to. So you just need to run this command for every language. And once you create a ULOKO folder in your project workspace, Django is already going to create the correct folder structure for you with the LC messages folder and everything. And to compile uh all PO files, you just need to run the Django Admin compile messages. Um if you don't specify a locale, it's just going to compile everything that it finds. So you're good to go. Um I'm going to show you a little bit how does this

Speaker 1: work on the browser so you can understand what I mean by not needing to do anything. So just Change here. Okay, so you guys can see that, right? So I have here a simple page, hello world.

Speaker 1: My view is just like sending this string back to the template and rendering the template. That's uh simple as So if I go here to my settings in, I'm using this in Chrome, but this should be able to change in any browser that you may use. Um if I change here my language to Portuguese as my preference and move that to the top, um and Reload my page, it's going to uh identify that and translate my message to Portuguese. So what just happened is that I changed the settings of my browser to instead of passing English in the accept language header it's going to be passing Portuguese. So I s I was able to generate a PO file for Portuguese, translated all the terms and compile

Speaker 1: that. Django already already captures the accept header language. and makes the uh correct configuration on the GNU get text uh module. So yeah. Um so now about format localization. Format localization is also enabled by default when you create your jungle project.

Speaker 1: It formats dates, times, and numbers displayed based on the current locale. This locale is also obtained from the accept language header, so it works the same way as the translation bit Just a special um reminder that to enable thousand separators uh by default in when you display any value on your uh jungle application you may want to use uh the use thousand separator settings in your project settings just if you're going to use in every every value that you have So here I have a sing uh a small example about um also date in price. Um and we have a form here that has a date, that's a date type uh

Speaker 1: date field and a price, this is a decimal field, and to make sure that it's going to um to take the user locale into consideration when receiving the input in the form. I just need to indicate the localized attribute for each one of these fields. So I indicate that as true. So they should be good to go and receive the correct value based on the user locale So let's see an example of that. If I change here, now I have a symbol that form that form um in a template. And here, reminding that I am in Portuguese ,

Speaker 1: so I'm in loqueio of Brazil. So Let's say that I'm here and I forgot about that and I just want to input the current date. So let's say September 23rd of 2019 And then I'm going to put any price here. And if I hit submit, it's going to give me an error because hey, this is an error in Portuguese that's saying please. input a valid date. So it's not considering my input as valid because for them they are expecting the format of Brazil locail, which is day, month, year. And we don't have a 23rd month in the year.

Speaker 1: So um if I change my locale back to English, it should be good to go. But you notice one thing that came out wrong? My price, I had comma as decimal separator, but I changed my locale to English, so now it captured my uh comma as a thousand separators. So now my price jumped to twelve thousand over there. So this is something that you need to pay attention to And here we have the format. So if I go back here and just change uh back again to my uh Portuguese loqueio. and reload the page, it's going to give me the date in Portuguese.

Speaker 1: And also the thousandth separator now is a dot and the decimal separator is a comma. So just to illustrate how that works. And now um I would like to give you uh some information about time zone, how time zones would work in jungle. I know this is a pretty difficult topic. It's very hard to work with time zones, at least I think so. It's very confusing. I mean I am jet lag right now because it's like four hours of difference from Recife to Los Angeles time zone. So it's quite messy. But how time zone would would work in jungle.

Speaker 1: So if you um if you enable time zone on your project. All the time information is going to be stored in the database in UTC time zone if that is your default time zone that you put in your settings Only aware data time objects will be used internally for any calculations, any comparisons that you want to do And these daytime objects will be converted to your user time zone when displaying information in templates or forms. One important thing that we need to understand before working with time zones in Django is the difference between naive and aware daytime objects. So naive daytime objects don't have the

Speaker 1: TZ info attribute uh configured in the object. And this attribute is just a value that indicates an offset and it's the offset of the time zone. So if you just want to do a timestamp using the data time default library and use the function now. uh by default it's going to give you a naive uh daytime object. However, when you are working with Uh time zones in Django, as I said before, you need to work with aware data objects that have that TZ infull attribute uh set. So Jungle Already gives you a time zone module that can facilitate that for you. So whenever you need to create a

Speaker 1: timestamp Taking into consideration the time zone, that's aware our aware the time object, you just need to use that and use the now function to to do that. Another interesting thing about time zones is how to determine the current time zone for your user. So when we saw before about translations and formatting, I said that the accept language header was being used and was informed by the browser. But we can't use that information to determine the local uh time zone. Um if you say that your Um your language is English from the United States. This doesn't mean

Speaker 1: it doesn't inform me uh which time zone you are in. You can be in the West Coast in the the audicals, so uh which time zone you're in. So the jungle documentation actually invites you to uh actually let the user select the time zone And by you can store these in the session, or you can store that in a user profile if that's something that your application have. Another option would be also to discover the user's time zone using some JavaScript libraries. Me personally, I have worked with Moment. js time zone and I was able to figure out the time zone for the user pretty easily, pretty straightforward.

Speaker 1: So you can try to do this with JavaScript too. So um I have just a example here for time zones. I have some alarms set up I think but let me try to set my time zone So here uh all this information about like all these lists of time zones you can grab from um the uh PyTZ module that is already stalled when you start jungle so you can have access of all these lists these have all time zones that they support

Speaker 1: So I'm here in uh receive is time zone, so let me change that to I think that here's Los Angeles, right? So if I change here to America I can't find it. Yeah, maybe. Oh yeah. Found it. Thank you. So now I'll set my time zone to Los Angeles and it's going to story my session. Um So now I can set an alarm, let's say for tomorrow.

Speaker 1: Um and I want to wake up at 7 So here I have mine in. So if I go here to my um Jungwa de Min, um it's going to show me that object. Let me a little bit here. It's going to show me that alarm object that we just did. But if I go back and change back to Recife. It's going to show me a different hour because in Recife we are five hours ahead. So it's going to give me another time.

Speaker 1: And here in my admin , it's going to give me a note down here that's going to indicate how many hours I am different from my original date that was set by the user. So this is this can be very helpful for you when you try to troubleshooting something. If something happens, your user puts some date in and you don't know what the time zone was. This can give you some more information about that Okay, so now we know how to do formatting of dates and numbers, we do know how to do translations, we know how to work with time zones, but

Speaker 1: Does this how does this change the development process if it changes at all? So we had uh the opportunity of working in an internationalization project uh back at lab codes and one of the main things that we had to change was our deploy process. So we need we needed to adapt our deploy process to include the translation process in between So we uh we had this stage uh deploy where we would put our changes in and then wait for some of our validations for the client in order to proceed to production. But now uh we had to uh not only put everything in stage we also had to send any new or any updated terms to translators so let's say that I just built a new

Speaker 1: a new page on my web app that has this I don't know, dozens of new words that I just created. So I need to pass everything to the translators so they can translate that to all the languages that we support in that application now We also needed to take care of integration tests for different locales. So now whenever we were doing all the testers were doing any tests. They would not only test the functionality per se, but also if that work in different locales in different countries. And our deploy, uh our production deploy would only be approved when all terms were being translated. So it was something that we need to take into consideration because

Speaker 1: The production deploy would not only depend on my team, on our team of developers that work closer. We were also dependent on translator, the translator, the translation team. So this is something that you need to take into consideration. And yeah, that's basically the message that I want to pass to you guys. I hope that you find that internationalization is actually a fundamental step. in design and development process. Don't do as we did because we had a project that was up and running several months, several users and We suddenly got the news that we needed to expand to support Canadian users. And we had to translate everything to French.

Speaker 1: We had to uh convert our prices in unit of measures for to accommodate Canadian users. So when we got the news We were very shocked and that's where the ATA comes from because we had a lot on your pla on our plate and we still had to do this. So don't do as we did. My recommendation would be if you are in the beginning of your project Pay attention to that and try to include internationalization in the design process of your application. If that's not the case, if you're already down the road, you're just building it or even just maintain it. My recommendation would be to follow the Boy Scout rule and

Speaker 1: try to flag all the strings whenever you are working with a piece of the code. Try to look for any strings that haven't been translated yet or mark any numbers that need to be specially formatted for different countries. So you can do this step by step, even though it's not a requirement right now, but trust me, it's going to be a requirement soon. So just follow this this I hope this is a nice information for you guys and yeah that's it I hope you enjoyed and you leave here with some more information

Speaker 2: I actually have a question on the Regarding the deployment process, do your translators use edit the PO file directly usually, or how does that handoff work?

Speaker 1: Actually, we integrated with a translation platform. We have several of them. If you type in on Google, we have several examples. Because typing in the PO file is very like it's a plain text, it's not very useful. So what we did, we uploaded this using their API to, I don't know, TransFax or SmartLink. We have several platforms that do this. And they just like parse it out and then show to the translator in a very much nicer way. So they could work the translations, it gives some like suggestions and It can give you some more context about the string. It's a pretty nice tool. We have several. So we actually use that to make it easier for the translators.

Speaker 3: How do you handle translation when you're dealing with a front end like say React or Vu?

Speaker 1: Actually, Django has a catalog for working with JavaScript. It can capture, like it exposes the getText function to your JavaScript code and you can do that there. In this project that we have to we had to internationalize, I was using Angular One actually, and we used their system of translating. It was basically just like some a a huge dict with the mapping between the key and the the words to be translated and you just need to put in the correct folder. It worked pretty similar to what Juggle does. One thing though that we noticed that we were using some strings in this CSS file like for arrows and stuff

Speaker 1: like yes or no thing. So we had to change that because I I don't know how to do internationalization in CSS and probably it's not a good idea because it's not good to have like a string hard-coded in your CSS. So This was something that they figured out in the process, so we need to change that and put the strings back in JavaScript where we could translate.

Speaker 4: Actually have uh thank you. I uh actually have a couple of questions. One of them is um any tips on how to handle uh the front end design for uh right to left text. For the Arabic text?

Speaker 1: When I saw that Wikipedia did that, I was also curious. But I didn't had the chance to research how they do this. It's it's pretty amazing. But hopefully not something like if and else in your code, but something a little bit smarter, but I don't know how.

Speaker 4: Okay, cool. And my s The second question is how do you handle translation for things that you saw in the database? Would you keep multiple fields for each language or how would you recommend doing that? that

Speaker 1: um well actually we we didn't have to to uh change uh anything our database for that project but currently we are in this stage of uh translating something that we have in a data store. We are actually using Elasksearch for a lot of things. So what we did, we have this child document. For each one of the documents with those strings. We actually extracted those strings, put the key, the ID, the string ID in it, and then we have child documents for each one of them for each language that we want to provide to. So this is the tricky part. Whenever we retrieve that document, we need to check what's the language that the user requested

Speaker 1: and then do the mapping correctly. I don't know how we would do this in a relational database because I have had the chance to work on it, but on on Alaska such this is how we are doing it It seems to work, but I'm not sure about the performance. I'm still afraid that there is something there that we will had we'll have trouble, but it's a work in progress Thanks.

Speaker 2: All right, that's all the time we have for questions. Thank you so much for all your hard work. Here is a towel.

Speaker 1: Thank you.