Peaceful Django Migrations with Efe Öge

Speaker 1: Hello everyone, welcome for the last session of today. I hope that we're gonna have a good session. I'm gonna have a couple of questions. I would like to make it interactive So if you had your coffee, if you had your desserts, so let's start. I'm gonna talk about peaceful Django Migrations. My name is Efe, not Ife. It's pronounced as Efe, Efe. It's a Turkish name. I'm originally from Turkey. I'm living in the Netherlands for a year and a half. So in the past I worked as a founding engineer in early phase startups. Uh and I have experience with Django. I I'm working with Django around ten years. Now I'm working as a senior engineer at Scentcloud. SandCloud is a shipping company operates in Europe.

Speaker 1: I have a lovely blog which I uh rail write. It's at Fa. me. I also shared the slice of the presentation in this URL so you don't need to take any pictures, other stuff. You can just go there, check any links and references. Just please focus on the content. I also try to be active in the Django community. We try to have a community in the Netherlands for Django, which is Django NL. We've celebrated the 20th birthday of Django. I also had a chance to be a mentor at Django Girl Django Girls Amsterdam. And before that, when I was living in Turkey, I was also trying to be active in PyCon Turkey and Pi Istanbul community. I love talking about Django,

Speaker 1: Python, recent LLMs or backend all. So if you'd like to talk with me anything about these four items, I would be happy. And I'm also fan of boring technology. So if you would like to talk about recent Python packaging, I'm not very interested. But if you'd like to talk about uh PostgreSQL internals or How we can optimize these non -technologies, I would be happy. Uh so please share with me your best tricks. Okay, these are the pictures of me. This is me and my co-founder from two to twenty years ago And I like playing backgammon. I occasionally run. I would like to run also in Chicago Marathon one day.

Speaker 1: And let's come back to our topic, enough for my history. And in our agenda today, we have four four items. The first one is Django Migration 101. I wanted to have this talk as a beginner-friendly. So if you are a complete beginner to Django, you can also learn some stuff about Django migrations from the first chapter. Then we're gonna in the second chat second chapter we're gonna deep dive to the what is this downtime thing? Why we have downtimes when we are having a deployment. And the third one is more PostgreSQL specific problems, which is which are logs. What are the things which cause these logs and makes a downtime in our uh deployments? And in the fourth item, we're gonna talk about

Speaker 1: How can we predict speak before they happened? And if we have time, we're gonna have a QA at the end So what I like textbook definitions, so I would like to start with one of the one of the another uh textbook definition. What is Django migrations? Uh Django migration is a way of tracking and applying changes to your database schema so it stays in sync with your models. But also what's not about? It's not about migrating from PHP to Python Django. It's not about migrating from API V2 to API V3. It's not about migrating Django 4 to Django 5, it's not about migrating FaceTapi to Django. Also, it's not about migrating from another PostgreSQL

Speaker 1: version to another PostgreSQL version. And I am also an example-oriented person, so I'm gonna start with an example which is a library. It's not a framework or software library, it's a literal library which has books. And imagine that we have a library, we would like to keep the records of the books. What are we gonna keep? We're gonna have a book entity. And if we if you like to have a book entity, it's gonna be something like this in Django World. We're gonna have the title, author, genre And we would like to list the books for people who are uh a member of the library. And after we define our model, we're gonna run this lovely management command. In Django, management commands are things which

Speaker 1: followed by Django Python Managed Py. And the first one is make migrations What is make migrations which basically a management command turns the Django modus to Django migration files which are gonna be run in the database. And we when we run make migrations, this is the example output. It says that okay, I created a new migration file for you. And it also summarizes what is inside of the migration, which is create model book. And if you take a look at the migration file itself, you're gonna see this. And in here It inherits from the migration class. There is a special flag for the first migration in each app, which is initial true.

Speaker 1: Since it's the first uh migration, there is no dependency. So dependencies are empty. And in the in here you're gonna see operations. And As you can see these are also Python clusters and and models. Okay, you might ask me, oh F, you said that we're gonna Create a file to run on the my database. But we are using PostgreSQL. How PostgreSQL can understand any Python? Okay, let's go one step more. So there is another management command in the Django which is actually creates the equivalence of migration file which is SQL migrate. As you can see, I passed the books, the which is the example app name, and the migration number in here

Speaker 1: And it outputs the SQL code which actually corresponding to my uh migration file. And actually the when we uh Run our migrations. This code is gonna be run in the PostgreSQL or any database, not the Python code. Then we're gonna say Python managed by migrate, it is gonna Apply the migration in the database and you're gonna see that okay it's applied. Good. Okay, so as a nature of every project Requirements are changing and we had a simple uh prototype which only had title, genre, and Autors. Now we would like to also add ability to list by languages.

Speaker 1: How are we gonna do that? We don't have this information. So we need to add a new model, which is a language. It's gonna be a child field, max length. But how are we gonna do this in the database? I just updated my Python models. Then I'm gonna run my mac issues again and it's gonna understand the difference. What's the different state of my current database and what's the last item I have in the my make migration files? And it noticed that yes, I need to add a language field to the model. And this is the inside of the the content of the second migration file. As you can see, it says add field, model name book, language.

Speaker 1: And if you take a look at the SQL migrates, this is the output SV, uh SVGas. What it does, it alters the table of book, adds a column, this is the default value. Then we run again migrate because we need to apply this migration to a database. If we don't do that, it's going to stay in our Python and it's not going to change anything in the database. But how does Django know to apply only the second migration in this? Do you have any guess about it? It can be you can be loud. Yes, so good. Actually Django is super

Speaker 1: smart and they have uh Django has a default uh database tables which is called Django Migrations. And this is the structure of the Django table, which has ID, app, name, and app light. ID is basically auto increment ID, we don't use it that much. App is app name app name, which is books in this example. Name is the name of the migration file and apply it when we apply the migration. And if I do this in my application, I'm gonna see this response from the Postgres card. As you can see, 18 and 19 ones are the my my own migrations and these are the times, these are the names, so good. But of course running a railway scale is not that Pythonic or uh

Speaker 1: Django-esque. So there is a good management command which we can run, which is Python managed by show show migrations. It also shows you that if you apply the migration, you're gonna see X next to it, and you're gonna see the whole migration tree of your application of your application. I also like to compare Django with uh Rails. Uh Rails uh another web framework which written in uh Ruby on Ra Ruby Ruby language. And I was curious how do they do? Of course they also have a migration management system. And I think I like the Django Way more. Basically, Race also has a same database table. But they only keep the the ID of the m uh migration, which is a

Speaker 1: some kind of uh Unix time stamp timestamp. And they don't know anything about Epname, other stuff. I think Django's ways very well fit with the Python's philosophy, which is explicit is better than implicit. Also, it is possible to revert migration. Imagine that we said that okay, language, we don't like to add language anymore. We would like to revert it, so you can also uh give book uh app name and uh migration orders and it's gonna revert to migration as you can see So, as a wrap up, I would like to list the terminology of Django migrations. Django models describe the structure of the data we keep, like title, authors, for

Speaker 1: books. And Django Migration Files acts like a to-do list for the database, what needs to be created, updated, or remote. Like add language, review language, these type of examples. And I would like to also create a cheat sheet for Django migration management commands. The first one is make migrations after updating each module spy. Show migrations to see what we applied in our database, migrate to run uh the migrations on PostgreSQL and SQL migrate for uh previewing the equivalent SQL code of the migration file. So, I would like to make a small quiz because it should be interactive. So, as you can see, I run the same command in two different times, but the output is different

Speaker 1: In the first one, it it only says alter table, alter table and commit, and the other one creates table, insert info, drop table, and alter table. What can be the reason of this? Any guesses? I'm gonna check my time. Oh, okay, we are on time. Yes, please. Yes. But the opposite. Actually, the first one is PostgreSQL SQL and the second one is SQLite. You were right. So actually we have the same migration files. But the capabilities of the databases might differ. In uh SQLite, it's not possible to alter to alter the table. So therefore it drops the table, adds a new field, adds it

Speaker 1: creates a new table and exchanges the data. So this is the internals. But it's a I think it's a nice fun fact to know. So when you are running a migration, it's also nice to keep in our head. So we onboarded what are the Django migrations. Now we know we have idea what are the basics and principles of the migrations. So let's move on to the next topic. So I would also like to do some emotion test. I'm gonna list a couple of uh words to you and you're gonna say what what do you feel like? Do you feel happy? Do you feel sad Downtime incident uptime outage

Speaker 1: Avability 500. This is just a number, but I don't know. We have some kind of negativity. And yes. So now today we're gonna rewrite our emotions to towards this this verse and we're gonna understand the okay what's happening actually. So again, textbook definition: what is downtime? Any period where end users cannot uh successfully complete their intended behaviors. It can be they they they might click a button They might do some other actions, but somehow it doesn't work. Maybe it worked very slowly, it it doesn't work. Anything which affects the end user. And I thought about it. Like I would like to create I would like to summarize this topic very well. And I thought, okay, what are the real reasons of these things?

Speaker 1: And I come up with these two reasons, maybe there are more, but this is my summary. If you have more, please share with me. And the first one is Schema version mismatch in rolling deployment. It's a fancy term, but I'm gonna explain. The second one is a locking migration that takes a long time. Let's take time. So, this is an example chart for modern applications how we deploy apps. In the modern applications, we usually We have one load balancer or a few of them, and but we have thousands, not thousands, but lots of apps. And we would like to update the version of these apps without affecting any users. Then how do we do that? So the left one is the first and the right one is the last. So imagine that we have two

Speaker 1: v1, the first version, and we would like to deploy the second version of it. What we do? We add the second version We add another one and we create these old versions and therefore all users actually request to the new versions. This is a basic rolling deployment. You might also say it's canary deployment, boolean deployment. The principle is the same. We are adding more resources, removing old resources, and somehow it should work. Also, uh there is all most of time we have this. We only have limited databases, one database. And lots of app servers because our database is most of the time a single resource of tools, so therefore it's very hard to replicate it. But on the other side, we can run lots of applications, they're gonna handle different requests.

Speaker 1: So this is Osaniodorch diagram, we have one database, multiple web servers. Okay Imagine that we are doing a deployment, how it works. The first step is that Python managed by migrate. We we first we are adding we uh deploying deploying any ch we are before we are changing any code in any app We need to update our schemas. So, what we do what we do we call Python ManagePyMigrate. We already created our migration files, committed to the git, then In the continuous deployment, we pull these migration files and we are running them on the servers. The migration adds the language column to book. But while Ritbain is in process, not all app instances are updated yet. So we only added this field in the first application,

Speaker 1: but the rest of them they don't know yet. And all the instance, all old app instances with old chemical. They might still create new book instances without setting the language. But I already said that I need this information in my database. How am I gonna do that? If the column is already not null, those inserts will fail. In our example, they are not nullable. If it's nullable, they might insert rows with null values. It's so good. But the NIV app doesn't expect this. Maybe they would like to do something different. And the NIV app instances, they expect every book to have a language. Imagine that I also change my Python logic, and if I if I have a serializer, I don't expect them to be null in the front

Speaker 1: end, in the API. Uh but they might read rows created by old instances which have null values. And this might also cause us a problem. And this is basically a window, mismatch window. During this rollout, there is a period where some application instances speak the old Chemma contract and others speak with the new contract. This creates a temporary incompatibility between application and database. And this is the first reason of time times in our deployments. And this is the summary of what we have. What about the second one? A migration that takes a long time. So

Speaker 1: let's let's see the brief steps of our deployment. The first step is we are detaching from the load balancer, we are saying Python is managed by Migrate, but it takes 300 seconds. Why? Then we're gonna deploy the first step, but we are not in that step yet, and it takes so much time. And people are complaining they are it doesn't work right now for end users. What's the reason Because apps cannot read or write to the database table. Why? We're gonna see it. So these are the main two reasons. The first one is rolling deployment have different schemas in first step and last steps. And the second one, we have a migration which takes so much time, but it somehow locks the other processes. So let's learn about the shama

Speaker 1: changes that causes logs. Uh this is a super nice website. Don't Look at the design, but content is the king. And this is pglogs. org. Uh I love it. This is a website of Jusin Nassar. He is also a very nice uh developer who has quite nice content about Django, PostgreSQL. I really suggest it. And in this website, you can see the internals of PostcrewScale logs and how it they work. Basically, I'm gonna I would like to onboard you a bit. So in PostgreSQL there are two types of locks which are in the main categories table locks and row locks. And their names are not that intuitive in my opinion, like access share lock, row share lock, share lock. They are a bit weird, I

Speaker 1: I would I would admit that. Then there is comments. And you can see that if I do a select, what type of looks I'm gonna acquire from the database? And what what type of uh comment comments comment comment sql comments are gonna conflict with this log if you see this as a like list of items it's I I couldn't understand very well, so therefore I will I wanted to create a graph of this uh information. So when you are making a select request, when you would like to read something from the database, actually it creates an access share log And access share log conflicts with these SQL statements comments. But I don't know, there are 27 of them. I cannot memorize them from my mind like so I'm not a

Speaker 1: I'm not that smart. And I created the main graph of it. I'm sure you don't you cannot see it, but if you see it, you are not gonna also, it's not gonna be that easy. I would like to share this as a some kind of interactive thing for training purposes after the call And if you would like to dig dive into the tabla logs of VSQL. Gilchin Råter a super nice artikel. I really suggest this. I'm not gonna go super details of the PostgreSQL details. But please, so good. Okay, we know that SQL comments are can conflict with each other, but how can we predict that Of course, I love interactivity. I'm gonna ask you a couple of true or false questions. The first one: adding a nullable

Speaker 1: field creates a long log. Is it true or false? It's another bullet an E field. What do you say? False. False. Correct. You won. The second one adding an index creates a long log. Is it adding an index creates a long rock long rock? Long log or not False. I see some false, some true. It's actually like this. It's true if you create index And if false if you create index concurrently. And as default, Django creates index without concurrency. And if you have a super big table , And if you'd like to have some kind of data consistency which

Speaker 1: adding a index, post PostgreSQL needs to wait to finish it. So therefore, please. At your index is concurrently. I also don't know why we don't have this flag in Django settings. I would like to also have a setting like this in Django. If we have it through, it's gonna automatically use Django index concurrently. And the third one, another question: adding a non-nullable field with default value creates a long log or not? True? Who says true? Who says false? We are half divided, yes. It's also nice. I like this. So who are these two people? Any guesses? They are great people, I really admire them. And you are reading right now a PostgreSQL

Speaker 1: 11 release notes and it says that Uh in here, allow alter table to add a column with uh a non-null default without I'm so bad that's right, I know I'm sorry. These two people were Andrew and Serge. They added a new functionality in the PostgreSQL QL11. And this is the comment. I tried to read the code, I don't know if anything C, but so nice to see they're super smart. And if you are using bottomless story, if you are using PostgreSQL 10, it creates a log. And if you are using PostgreSQL 11 and above, it's not gonna create any uh any long lock. But there is a trick. If the default value is constant, so if you say that default values

Speaker 1: uh Turkish in the string It's not gonna, but if you have some kind of no now function which actually calculates the time, or if you have some kind of function which gets a random UID function, it's also gonna create a log. This is a huge mental lot for a developer. We cannot know all of them. So and this is so It makes me paranoid. Like, was it like this? Which PostChair version we do we have? What about is it a default constant? What about dynamic one? I don't know. I need to try, I need to see, and it is it it makes me mad. And this is my friend Heat Kular. He's my first uh mentor in

Speaker 1: Django World. He is a sailor. I really like working with people coming from different backgrounds rather than just full software people. And because they bring some knowledge from their past principle and Since he's a sailor, he has lots of principles about sailing, sea safety. And he is also a dictative of aircraft investigation. He loves watching this type of post-mortems of planes. And we had this philosophy in our team: sleeping in the storm. So if we have this type of mental load in our applications in our heads, we cannot sleep in the storm. So therefore we need to automize this type of checks. And they they cannot be a checklist because we cannot

Speaker 1: Create these items one by one. We are gonna miss a couple of them. And if you like to deploy something in our stressful time, are we gonna have focus to read this all of the checklist items? I think we should learn from the aircraft investigation. So according to statistics, because you're not gonna believe me, I'm gonna throw the statistics to you. Airplane burning systems have cut crashes by 85%. Since 1970s. And it it saves a huge amount of money because we actually prevent the mistakes before they happen. We don't need to deal with the after the after the accident. So we should sleep in the storm without having a mental load. So therefore, I'm going to introduce you a couple of automated

Speaker 1: automation tools to predict the logs without guessing, like, is it true? Is it false? The first one is Django Migration Linter. It's a great package. Actually, I was planning to write a similar package to this, and I saw that it's already exist, and I felt so happy. It's so nice. It's a native Django package which you can use from tomorrow. What you need to do, you just need to install. Uh I don't know. I'm boring guy, I'm using still pip. Pip installs Django migration linter. And It introduces a NIV uh management command, which is a Django way. Uh you can just run Python Python managed by lint migrations and it's gonna Make analysis over your migration files and it's gonna say that okay this is suspicious.

Speaker 1: Please take a deep look on this. Maybe you need to change a way to do it And the second alternative from uh go land, which is uh squawk, it's so hard to spell, yeah. And you can install it with the npm It's a bit similar also. It is more like linterbase. There are rules you can disable and enable. Basically, you can pipe uh SQL migrate To scope and it's gonna give you analysis like this. Of course, this is not the Django way. I created a small synput for it, and you can just run PythonMashPySquok migrations and it's gonna just give uh output. And it's gonna only give this output for unapplied migration. So it's a very nice thing to plug your CICD

Speaker 1: tests. Before deploying anything, you can just have this output and you can have a control on your deployments and you are you don't need to be paranoid what's gonna happen. And uh you can also find the management comment from this URL, tinyurl. com squash slash squash migrations. And the third one is a bit Controversial, but it's also nice to mention. It's a quite edge technology, which is called PGRL. It uses internals of the PostgreSQL, it creates the temporary shadow columns and views During the migrations, allowing both old and new migrations can work at the same time. I'm gonna briefly explain it, but I will say that I'm not sure that it's a production grade uh ready system It also doesn't have full integration with Django, but it's worth to look

Speaker 1: eye on it. Basically, before starting, you need to init this. It's gonna create some database models in the PostgreSQL. Which with this command PG PG roll in it. Then you need to have these settings. And as you can see, we need to set options in here and we need to add a search path. And this search bed actually come coming from the uh environment variable. We need this during the deployment. And I'm gonna briefly explain the deployment with PG roll. First, we need to. So PG role doesn't work with the Django migrations. We are actually using Django migrations as a some kind of semi-source of truth. But we are gonna create pg roll migrations from Django migrations

Speaker 1: and we are get we are gonna get SQL equivalent of migration and we're gonna bypass it to the PG role and it's gonna create a new file, JSON file for us Then we're gonna in the beginning of our migration, we're gonna say, okay, we are ready. PGROL start out start our deployment. And it's gonna create a new schema for us actually. And it's and in this never nif nif schema, we're gonna have our nif field. So then we're gonna set our um environment variable. And we're gonna say that okay, in this app we're gonna use this schema, so therefore get it from the environment variable. After the migration of first application, we're gonna do it in the second one. Then Each application is going to be deployed, then we're gonna do we're gonna fall back to the default one. Okay. Now everything is settled.

Speaker 1: We're gonna have the public shama back. For all applications, then we're gonna say that okay, this is completed. I think this is a nice technology. I haven't used it before. This is a disclaimer, but it sounds it's it looks nice And if you look to diagrams, you can see that this is a time. We have old version available until the schema changes are completed. And this is an example of how they uh map the fields under the foot. Okay, how we can predict the locks. The first one is Django Way, Django Migration Interest installed the package, have new and management commands.

Speaker 1: Run in this in your uh CICD pipeline. I'm gonna share the slides by the way you don't you're gonna have it. The second one is Scrub, it's a go package, it's more like a linterway, you can disable enable the rules It also works and the third one is PG Rule. It's an alternative to Django migrations. And this is the end for my talk. If you have any questions, I'm happy to answer them if we have time.

Speaker 2: Uh can you go back into the details of the last slide you showed? Uh sorry, the second to last. Uh uh third, last, fourth. This one, yeah. Yes. Um so uh w i is this like tenants with uh deployments? Like your migrations are in the tenant schemas?

Speaker 1: What is tenant schemas? I don't know very well, so therefore if you uh

Speaker 2: so when you have public, public is your uh

Speaker 1: Default, yeah.

Speaker 2: Right. Uh and then you create another schema. Right. So you're using that as the deployment.

Speaker 1: Exactly.

Speaker 2: Okay. So how do you tie the data?

Speaker 1: I don't know the internals of it to be honest, I'm not gonna lie, but it's it works under the foot. I don't know the real answers, but as far as I know, they are using virtual schemas and it copies the data on the background. So therefore you can make this uh it's it's like a view of uh PostgreSQL. When you have a view you don't need to have a create a new table It's it actually works virtually. I guess it also works like this. I don't know the internals, but they have also a great blog post, maybe they explain better how do they do it.

Speaker 2: Okay, thank you.

Speaker 1: Any other question?

Speaker 3: I have one. I've I've definitely hit the issue before where you uh add a not knowable field and in a hurry and forget to, you know, make the separate steps migration. But did one of the apps that you recommend um have a protection for that or some automation for you know creating those separate migrations that are necessary to add a not knowledge field?

Speaker 1: So there are also some other packages from one of them is from Yandex. It creates also self-it's called Django SafeMag Safe Migrations, but I would like to have multi-step migrations and manually operate the migration flow more Uh it's more most like mostly more personal preference, but there are also other ways to do it to be honest. What what do you choose to be honest? I'm also curious.

Speaker 3: I've I just tried to remember right now. So any other questions? All right, I think that's it. Uh thank you very much, Chef. Uh please give him another round of applause.