Becoming a Multilingual SuperHero in Django by Sanyam Khurana

Speaker 1: Thank you so much. Let me just introduce myself first. I'm one of you, a part of the community. I am a C Python contributor and Bugtrayaj access on bugs. python. org. I've been contributor to Mozilla 's ecosystem of projects ranging from add-ons, marinate, task cluster. Gecko Engine, etc. I've been a GSOC mentor for Debian and now GSOC Mentor. I worked as a back-end dev at Field , empowering iOS and Android and web apps. Uh this stock came directly from one of the projects uh I worked on, which was for a Chinese client So uh imagine you've got this awesome RESTful API built on Django and DRF and suddenly this requirement comes in.

Speaker 1: We want this app and the CMS or the admin panel in Chinese or we want it in German So, uh if you have this humongous code base not written with all those Yu Get text and Yu Get X-lazy stuff in mind, then boy, you're in danger. And you should better start praying to the Almighty to give you the strength and endurance on your path to become a multilingual superhero. The first and foremost thing to enable translation is to tell Django what is the list of languages that you support. Where does it find the translation for static data? And what is the default language it should fall back on in case there is no translation available for the requested language? So we'll have a look at the settings So we include a bunch of settings.

Speaker 1: First is the middleware classes. We have the special middleware, which is local middleware, which is uh placed just after the session middleware and before the common middleware. The order of The middleware is quite important and we'll see next how it plays a role. We define the list of languages. So in this particular example I am supporting English, simplified Chinese and traditional Chinese We also defined some more attributes. One of them is use I18N, which essentially makes some optimization to load all the internationalization machinery that helps you in localization. Uh we make use L10 N to true, uh which will help you format dates, numbers, calendars according to the current locale.

Speaker 1: We set the default language to English in this case and then we define the local paths. So This is a directory where all your static string translations go. So we'll see next how it plays a role. But first we need to decide how to track the language the client wants to communicate in. So there are different ways to get the preference in an HTTP request And in order to retrieve the language preference from clients request, local middleware tries to determine the user's language preference by the following algorithm. So first it would look for language prefix in the requested URL. So for example, if your URL is slash EN slash APS slash resource, Ian stands for

Speaker 1: the language code. And this is only performed when you're using I18N patterns function in your root URL conf Failing that, it looks for the language session key in the current user session. Failing that it looks for a cookie. The name of the cookie used is set by the language cookie name setting. The default name is Django language. Failing that it looks for the accept language HTTP header This header is sent by your browser and tells the server which languages you prefer in order of priority. Django will try each language in the header until it finds one with the available translations. For the REST APIs, I personally have found the except language header a much cleaner way to accomplish the task

Speaker 1: and for enabling multiple languages. for the admin panel routes since it's a get request we'll prefer having i18 n patterns function and modify the root url conf as shown here. So all we say is like We we describe the URL here and we just wrap it up in the IET and patents function. So once we do this, uh the IET and patterns will automatically prepend the current active language code to All the URL patterns defined within the I18 patterns function. So all your admin URLs with the current configuration having ZCN and EN activated will have the URLs as shown here So the star here indicates that anything can come over as a suffix.

Speaker 1: So for whatever language you want the end panel to be accessible, the use of corresponding language code in the URL can help you in accessing that. There is also a flag which is known as prefix default language. Once it is set to false , the default language code will not be preprinted. So and you can access the admin route at just slash admin. Although you can do this to all the URLs, but for the API endpoints, uh we prefer to supply this bit of information in except language HTTP header. So uh gotchas. Uh throughout this talk I'll go through some of the gotcha moments which I personally Felt a bit annoying like I was completely banging my head.

Speaker 1: So we'll see. The local middleware should always come before the common middleware and after the session middleware And why is that? Because uh as we see in the HTTP request, uh how how do you pass HTTP request Then the session middleware would first set the language code in the cookie if you're using that particular method, and then only the local middleware can actually uh the local middleware could actually process that information and take the language code from there. So what does the local middle help with? It's a secret source in the translation machinery. Let's have a look at the Django's request response cycle to understand it in a better way. Uh it does the following for the request part.

Speaker 1: It passes it and decides what translation object to install in the current thread context. For the response part, it does two things. So first of all, it will set the content language header in the response for the client to know what language is used in the response. So for example, if a client is requesting German language but we do not support German in this example, then the default fallback will be on English. And in the content language header we say that this particular response is in English. And the second thing it does is it formats the URL with activated language if the I18N patterns function is used. So it will say slash EN slash whatever. So the question is what to translate. Majorally there are two kinds of data.

Speaker 1: So one is the static data, which will include all the model names, field name of models, error messages, that will be starting in the application. When we come to dynamic data, it essentially includes the field value in models uh that will be input by the user. So for static strings we would need model names, field names, error messages, and we use them in many places in Django templates, uh all our code files, uh the models. py And let's see how we support those translation in all those places. So anything that should be translated in templates should be marked with either trans or trans block tag. To make the translation work, there is one more thing to take care of

Speaker 1: and that is loading the internationalization I-18N tag at the top of every file that uses trans or block trans. So let's see it in action. I have a dummy page which says sign up. It includes a username and password label and have a input for username and password. Pretty simple. Now how do I make internationalization and localization here? So uh I include uh To enable the translation and serve the static page, I'll first load the translation tag and mark all the stuff with translation. So I've included the IIT N tags Now I'll for the all the static strings I mark them with the transtag. So I say

Speaker 1: sign up, username and password are marked for translation. If there are translations available, then please go ahead and replace these with those static translations. Gotcha too. The I-18N tag should be loaded in every file even if it extends other file that already has it loaded. Now this particular thing can leave you bizarre. Like always remember to include IIT in tech at top of all the templates that use translation. If you don't do it, you'll keep like you you'll keep pulling out your hair some wondering what happened, but It won't work. So uh let's discuss about how to support translations and all. py files. So we'll talk about models. So I have this user model here

Speaker 1: which has the verbos name defined with the you get text lazy function. So if you see the import at the top of the file, it's From Django. tutils or translation import you get text lazy at underscore and all the verbos name property are marked with you get text lazy. So wherever they occur in your CMS or whatever, the translations would be picked up. So both UGText and UGText lazy are just Python objects that are evaluated to string at different times. So the string representation depends on whichever language is being activated. Let's see how let's see it in action. So I imported you get

Speaker 1: text uh and then I imported a bunch of utility functions, activate and get language here. So first of all I do activate EN. So I activated English language uh just to make sure that it worked. I used the get language function to know that okay English is activated. Then I called you getText for sign up. And since the translation was there, I get the English representation. Similarly, I did it for simplified Chinese. So I activated it, I tried to get language, and then I tried to get the translation. We'll see where do we actually put those stat extinct translations later. So the idea is like how do we decide which one to use? Should we use UGTX

Speaker 1: or should we use UGTS lazy? Well, uh if You need immediate evaluation of the translation then always use you get text uh for example in all your views. py file because once the request is coming in you need immediate evaluation so use you get text and Use UK text lazy in for lazy referencing the string object, for example, in your models. py. So uh now that we have all the static data marked with transtag, uget text, and uketxt lazy methods, it's time to generate the translation files and fill in the translations. So All the translations goes in Django. po file. PO stands for Portable Object Format.

Speaker 1: And for generating the Django. po file for simplified Chinese, we'll run the following management command. We'll say python manage. py make messages. Now we give a flag which is minus L indicating the language code. Once we do this We'll see something like this in our project directory there would be a folder named as local. So if you remember in the settings we already uh mentioned that what is the path of the locale folder. So This is how it comes into play. So we have local folder. Uh inside that we have this ZHCN. folder defined because we just executed the command. Inside it we get local messages directory and inside it we get the Django.

Speaker 1: po file Done. Now if we have a look at the Django. po file, considering the static template that was shown earlier, the signup page, we'll have something like this. So this is automatically generated by Django. It has message ID and message STR. So message ID is the text marked for translation and the message str is It's translated from. You have to do this for all the languages that you want to support and for each respective language there would be a different Django. po file. So let's fill in uh this for the Chinese language. So once I fill this in, I'll get something like this. And

Speaker 1: so far so good. Gotcha for. Uh while running the make messages command, if you have the virtual environment placed in your in your in your dire uh in your in the root Django directory then it it would kind of uh Try to generate Django. po files for all the packages that are in virtual environment. So don't make this mistake place your virtual environment out of your Django project. And this might not seem as a big problem right now, but we'll see how it can be potentially very big problem. So Now, this is the crux of the whole presentation, which took me like three days to figure it out.

Speaker 1: I was banging my head like I I I kept on reading the documentation. I was like, okay, let's see, let's see if it's not working. Why is it so? So notice that when you mention the language and settings, you do it by language name. So in this case we have ZH which stands for Chinese and CN for simplified Chinese and this is all in lowercase separated by hyphen But when you did this in make messages command, you used locale name, which was ZH underscore CN. So CN is the region and It's it's it should be in gaps separated by underscore. If you make a mistake here, you won't see any error and your translations won't work either.

Speaker 1: There is a small caveat, there's a small line in the documentation which says This is how you should do it. So take care and uh yeah. So uh We have defined all the stat extring translations and now we'll see how to compile these static string translations. We'll run the command python manage. py compile messages. And as soon as we do that This will generate Django. mo files corresponding to all the Django. po files that are there. Django uh the MO file stands for machine readable object. So As soon as this is generated, Django can quickly pick it up and show the static string translations. Gotcha 6. The compile messages command would generate the Django.

Speaker 1: mo files for every package in your virtual moment if it is in the root directory of your project. So uh I made this mistake. I had the virtual element inside my root directory and then uh once I ran the compile messages command, it was failing. And I was like, why is it failing? So I I through the trace pack I realized that okay this was particularly some other external package and it was trying to generate uh the Django. mo files for the packages that are in my virtual element. So gotcha 7. Always restart the whisky server after compile messages, otherwise the translations won't work So let's go on to the dynamic string translations.

Speaker 1: So most of the data in our Django application is dynamic and user generated. We can employ two approaches. uh in supporting translations. So first is like enable the end user to enter information in multiple languages. So we have to take care of how do we Make the changes in the database for this. And the second one is like translating the dynamic text using third-party services such as Transiffects. So TransiffX is nothing, it's a third-party service. You just dump all your data and there are actual translators in place which translate it and gives All the data back through webhook which you can store it. So we'll just see the first approach because the second approach is dependent on the first one.

Speaker 1: So let's do it for the model fields. So the tricky part begins here. Now you want to support multilingual data in your databases and let's assume that the user can input in just two languages, say Chinese and English for simplicity. And we can use the Django model translation package here uh which will create columns for each of the attributes that are marked for translations. If you see this code, this is very similar to what you write for Django admin for any data model in your app So you just say that I want to translate first name and last name in whatever languages I have defined in the settings. As soon as you do that, and you if you see the SQL behind it, It is something like this. So here the first name is the default field that was defined on the model and which tends to store and retrieve the value of the first name for the default language set in the

Speaker 1: Django app. which is English in this case. And for each subsequent language that your Django project supports, a new field with the same name appears, suffixed with the language code it is created. It is first name underscore ZH underscore CN for simplified Chinese version. So what if you don't want to burden your user with adding Uh information in multiple languages. You can use a third-party service such as Transifix uh for all the incoming data as we discussed. So uh since you all are still here full of energy, here is are some bonus tips and tricks for internationalization and debugging translation issues for you. Uh We we discussed that the request response cycle exactly knows which language is being requested by the client.

Speaker 1: But what if you are using async tasks such as the cell reque How does your async task know what language was there when it was called? So whenever the async task is called, uh the caller should apply supply the info for the language. Otherwise we'll have the default language already set. So I I I have this sent company registration email task here where I have defined the default language code will be in Chinese because this was a Chinese app. So in case like uh if a user is requesting English, this should be override uh once the caller is calling this particular task. So this is a service function which is being called by the task and it accepts the language code. So what it does is it first of all activates the language code so that whenever the template is rendered, it is rendered in that particular language.

Speaker 1: Secondly, it in the context itself it populates all the URLs that should be the that should be exactly matching the same language code that was in the request. So we use something known as translate URL here. We give give it the URL as well as the language code. So this language code right in the parameter goes here and then it knows like if it is the Chinese version of the template the URL also points to the Chinese version which is through the IED and patterns function. So what happens when you switch language in templates? So we have something known as get current language. We use that

Speaker 1: and we get the language code here. So Assume that this page in Chinese, so we'll get the welcome to our page message in Chinese. But for the second block, which is this one We have overridden it and said that okay we have used language uh En and said that okay this particular block should overwrite the English language. So even if this particular page is in Chinese, the first message will be in Chinese, but the second one, uh the welcome to our page, will be in English because we have used the language template tag to override that. Now how do you support multiple languages in templates? So assume that you have a website and you want to display like we support these many languages and

Speaker 1: You just click on the link and it will redirect you to the particular langu uh particular website that supports that language. So If your localized URLs get reversed in templates, they always use the current language. This is very important. And to link uh to a URL in other language, uh we use the language template tag. Uh it enables the given language in the enclosed template section as shown. Uh we use the get available languages. We get them in the languages variable we can then iterate on it and we get the language code and language name. As shown in the previous one, uh we just overwrite uh the language code with the help of the language template tag and as soon as we do that we just

Speaker 1: keep on rendering the links. That's it. Gotcha 8. Check in the Django shell if the translations are working with the activate and you get text. We already discussed it. You can just activate a language, get a Yu G text representation so that you know if it is working or not. Gotcha nine. So some strings are still not being translated. Uh if you go in your Django. they would have been marked as fuzzy. So these are particularly strings that Django marked as fuzzy because They want your translator to have a second look at it. Once you're sure that the translation is perfect, then you can remove that fuzzy line and compile the messages again and the the translation would work

Speaker 1: So conclusion. Django's translation support is indeed very powerful, but the initial setup becomes a lot of pain due to simple gacha moments which we discussed. It can potentially cause a lot of headaches and sometimes pulling out your hair. The more early you support your project in different languages and write the code correctly, the easier it would be. in the future to support multiple languages. Oh, do you know something? You just became a multilingual superhero with apps supporting multiple languages. Congratulations. So uh I'm also writing this book which is leveling up your Python skills. Uh it would be a free and open source book. The alpha version should be live till May 2019. Uh you can subscribe to updates if you want.

Speaker 1: and I'll be available on these platforms. You would find the articles on multiple lingual translations on my medium and here's my mail, here's my code mentor ID, GitHub and Twitter. Share love. Thank you.

Speaker 2: Um so when you y there was a step that you ran to um auto-generate the strings that needed to be translated And that was for the particular locale, correct? What what happens if you change some strings and you add some strings later? Does it overwrite the entire file? Does it like um keep the strings that are already around? So

Speaker 1: Django is particularly very smart around that concept. So say I ran the file at a particular instance and it includes say three strings for static translation. Uh then at later point of time I included more strings and then whenever I write uh whenever I r run that particular command again it won't override it but it would include all those things and At each of the steps in the comments it also mentioned the line number and the file at which the particular translation occurs. So it would it would check that and it would override that if it is needed

Speaker 3: Thank you.