Migrations secret powers ⚡

Speaker 1: I'm super happy to be able to introduce Majosh as the release maintainer of Django currently and working at Django Fellowship. And you're going to share a talk with us about something that despite having used migrations for a long, long time, we might have missed something. Some superpowers.

Speaker 2: We will see.

Speaker 1: Yes, we can to my

Speaker 2: Hello everyone. Thanks for joining me, both in person, here in Copenhagen and remotely, all over the world. I just want to say that it's really amazing to to meet Jen Vogue after the last three years, which which have been quite difficult to to all of us. This talk we'll discuss one of the secret power downcoming. But it's definitely not widely used and not well known. I should start from introducing myself My name is Mariusz Felischak. I'm a member of the Django security and operations team. I'm also a Django contributor, a Django 3.

Speaker 2: 1 and 4. 0 release manager, and a Django Fellow since 2019. So if you have any questions about the Django Fellowship Program, about the Django Software Foundation, or about Django itself, don't hesitate to ask after this talk. You can find me on Twitter and on GitHub if on. This will be my first jungle talk uh ever, so please be understanding. The migrations framework has been a part of Django since version one point seven. It is a way of propagating changes in models. Into your database. And I think that we'll all agree that it's it's basically awesome. It was a big step in making web development in Django more accessible.

Speaker 2: Because the migrations framework and its ancestor, the South Package, cause web developers to no longer had to know, to no longer had to write, and to no longer had to maintain SQL statements with data definitions. And maintaining SQL statements can be can be really painful, but that's how it was in old days. Over the past 15 years. I've written a lot of SQL statements with data definitions. Fortunately, you don't need to do this anymore, at least in most of cases. The standard workflow for changing your data definition has two steps. First, make a change. In this case, we are adding a new model called Mountain

Speaker 2: to our test app. Step is to generate a new migration file by running the make migrations command. A new migration file which describes changes That we made in models and changes that are needed in the database structure. They are not exactly the same. So we have here two different layers: changes in models and changes in a database structure. And differences between Between them will be quite crucial for this talk. The last step is to propagate changes into your database by random migrate command. At the point we already have a new table in the database. how our migration file is structured. The generated migration file contains the list of dependencies

Speaker 2: For example, the previous migration names, the list of operations to perform. In our case, we have uh create model And whether it is a first initial migration or not, via the initial flag. This is the first initial migration, that's why the initial flag is set to true. So far, so good. But but the question is what to do with changes. Business requirements change every day. You want to add new features, want to change the existing ones, you want to remove features between versions, etc. Changes in an app logic are causing changes in models and at the end also the database structure. Luckily for us, almost all of them are handled really efficiently by migrations. And we are trying to improve this in every version of Django.

Speaker 2: In Django 4. 0, we improve the detection of uh noob operations. Operations such as changing field attributes that do not affect column definitions. So for such operations we generate for such changes we generate operations uh in in migrations but in the same time no square statements are issue. We generate operations because me the migration framework on its own it's It's able and must be able to recreate models at any point in their life cycle without checking our models definition. That's why it's important to reflect all changes that we made in models uh in the migration framework and describe them uh in subsequent operations.

Speaker 2: That's how Migrations are able to recreate models at any point in the like cycle. So we generate operations, but no SQL statements are issued for uh for such a um uh for such changes Unfortunately, there are some uh specific transitions that do not play nice with jungle flow. So there are changes that Seem painless when we look at ROSQL but in the same time are problematic from the Django point of view. Maybe Problematic is uh is a wrong word. Changes that cannot be handled automatically in the most efficient way. Let's take a look at a simplified example of such change.

Speaker 2: Suppose that you have a climbing site. A climbing site that provides information about climbers and their ascents. Nothing fancy. Three models. One for mountains, one for climbers, and one for ascents. We have a many-to-many relation between climbers and mountains via the ascent model. So we have a custom true intermediate table called Ascent. We have a custom true table because it has an extra flag code called confirm which source uh whether this client is confirmed or not We've maintained this site for many years, we've collected data about hundreds of thousands of ASCIIs, and at some point of time we realized that we don't have many unconfirmed ASCII. To be honest, we don't need them. They are not used anywhere on our site.

Speaker 2: So we would like to remove them. From the database, but also we would like to simplify our model definition by removing the uh custom true table uh by by removing the custom true model called ascent. Because without the confirm flag It will contain only two foraging keys, one for climber and one for mountain. So exactly the same as Django would use for auto-generated intermediate table. So without the confin flag, it is completely unnecessary. And of course we'd like to Simplified our model definition and and uh remove it In the first step, we're going to remove all unconfirmed assets from the database to make this confirmed flag unnecessary. We could do this by run

Speaker 2: the delete statement directly in the database. But of course we'd like to keep this in migrations, uh do this in Python and use the the the the powerful Django RM When we want to keep data migrations in the migrations framework, we need to create a new blank migration to which we will add uh data migrations. For that, to create a new blank migration, we can use the empty flag to make migrations command to generate a new blank migration, a new blank migration that will be way structured, that will be structured as an immigration file. In order to follow good practices and to avoid migration names based on based on uh on

Speaker 2: timestamps I recommend to always pass uh the name flag when we want to generate a new blank migration file. I passed remove not confirm essence, which makes it clear what this migration uh what this migration We'll do. That's how our blank migration looks like. As we can see, dependencies are already filled, and we have an empty list of operations. An empty list of operations to which we will add run Python. RadPython is a special operation which allows to run Python code in So with RunPython, we will be able to use the RM to filter out unconfirmed ASCII and remove them without writing any row SQL.

Speaker 2: And that's our main goal at this step. This is a ready-to-go migration. We added runPython to the list of operations. RunPython accepts two arguments: code and reverse code. Code is a function to be run when this operation is being applied, and reverse code. It is a function to be run when this operation is being rolled back. In this step, uh in this case uh we don't need uh unconfigured assense So we don't need to make this step reversible. We just want to get rid of unconfirmed ascens. In such cases, we can use a special hook that is defined in around Python. It's called noop And it's it's basically does nothing. It's it's a noop.

Speaker 2: But it's a it is uh uh a perfect shortcut when uh transition from one direction or or an opposite direction uh is a noop. Functions that are passed to code and to reverse code must accept two arguments, absent schema editor. It's uh Extremely important to always use models written from apps getModel method and not imported directly from models. It's important because we need to use model from a certain uh certain point in its lifecycle and not its current version. It's important because in a subsequent migration uh we want to move confin flag so this filter will not be valid anymore And at the end we want to get

Speaker 2: rid of the ASNT model so it will not even be importable from from uh from from models That's why it's it's it's so important to always use model uh from a certain from a certain point in its lifecycle and not current version. When we have model, we can filter out unconfirmed Assets and Delete them. Now we have only confirmed ASCII in the database, and confirmed flag in the ASCII model is completely unnecessary. We can use a standard flow to remove it. So first So first remove confirmed field from the ascent model, then run make migrations and migrate. At this point, we have the ascent model with exactly the same structure as Django would use for auto-generated intermediate table.

Speaker 2: So it only has two foraging keys, one for climate and one for mods, and nothing else. The main question is how to persuade Django, how to persuade Django that the appropriate structure is already there and there is not much to do. And the only thing that we need to do is to rename, is to rename table. that is uh currently that currently exists for the ascent model to a name that Django expects for auto-generated intermediate table. My answer for this question is a special and extremely powerful operation which I want to encourage you to use. It's called

Speaker 2: Separate database and state. It has quite long but uh self-explanatory name It allows to separate database and project state. So it allows to separate changes that Are recognized by Django as being made in models from changes that are needed in the database structure. So it allows us to persuade Django that That uh Django recognized as being made in models were applied as it expects, but in the same time use optimized database operations To make tricky or sometimes even impossible transitions more feasible.

Speaker 2: Before we use it, let's take a step back and see what Django. Uh what what Django We'll do with our change. This is the div. We want to get rid of the uh custom true table. So we remove the ascent model and we also remove the uh true parameter from our many to many field. These changes uh the these two changes Literally describe what we want to do, but that's uh That's not exactly the the the the the the uh entire the entire change because we don't want to remove data from the database.

Speaker 2: We don't want to get rid of or uh data about us and that we collected over it. So we want to keep them want to keep them in a perfect world, keep them in the same place without any data migrations, without intermediate steps. Uh we would like to do this in the most efficient way, but in the same time, these two two two changes describe literally what we what we need to do. Unfortunately, Django does not know our intentions, so it will interpret so it will interpret these changes literally. Make migrations, generate two uh operations, altered field ascent on the climber model because we removed the true parameter, and delete model

Speaker 2: ascent because that's exactly what we did. Of course that is not uh exactly what we want to do because we want to keep uh data in the same place. Let's Think about it. Is this even possible to handle changes in the true parameter automatically? All changes In the true parameter would require multiple SQL statements because true parameter describe table that links to other tables. So, even in the simplest case, we need to create new intermediate table, migrate data from the old structure to a new one, and remove the old structure.

Speaker 2: This transition would require at least three, five, seven. It depends on the circumstances, uh intermediate intermediate steps. At least few SQL statements. Transitions that require multiple SQL statements are complicated and they are error-prone. There is also an open question about atomic. In some databases, as PostgreSQL, you cannot mix DDL statements with uh data migrations in the same table in the same atomic transaction. So we would be forced to create intermediate steps In in intermediate steps

Speaker 2: to perform this change. This flow does not sound really reliable, and it's not. And Django called itself a web framework for perfectionists. So what Django could do? Write an exception So value error is right when you try to change true parameter. Uh when you try to change true parameter on many to many fields. So Django leaves user to deal with it on its own Because when you know a specific circumstances, when you know when you have an extra information about this relation, you can make this visible. You may know that. You don't have any data in this intermediate table.

Speaker 2: So it's easier to to to to uh to remove it and and change this to this relation. You may know that You can afford for a long downtime to make this five or three or seven steps one by one without creating any intermediate steps. You you can afford for a long downtime, it's fine for you. In such cases it's also it's also feasible. You can you can do this in multiple steps. For example, you can create a new uh relation with some temporary name, move data from the uh old relation to a new one Remove the old relation, remove the ascent table, and at the end rename the new relation from a temporary name to a name that we previously used.

Speaker 2: So it's it's feasible. It's feasible. uh to do this in multiple steps of course uh would like smarter than that and you can be smarter than that with using separate database and states to synchronize uh project states and database state in the in the most efficient in the most efficient way and we can do this with separate database we can use it to Persuade Django that the ascent model has been removed, that the uhSent Ration has been altered to use auto-generated intermediate room and in the same time use database operations to To rename the table that is currently used for the ascent

Speaker 2: model to an able to a table that Django expects for auto-generated uh intermediate uh table That's how uh the migration um migration file generated by Jungle looks like. So we have two operations: uh alter field uh ascends because we remove the two parameters and delete model ascends uh ascent. We know that they cannot be performed and we don't really want to do this, but in the same time, these two operations literally describe changes that we made in models. So what we can do, we can use them in separate database and state, we can wrap them in this special operation and move them to a state operations to persuade Django that these two changes was

Speaker 2: were made. Expect but in the meantime use uh database operations to rename the existing table That's how it looks uh that's how it looks like. Uh we moved these two operations to state operations and in the database operations we ran we added uh run sql. Run sql is Quite similar to run Python. It also accepts two arguments, SQL and reverse SQL. And in this case, we added a row SQL to rename, to rename the table for the ASCII model To a name that Django expects for auto-generated intermediate table. One missing part uh are names for for uh table, for the ascent model and for uh auto-generated intermediate table.

Speaker 2: We can inspect them. By checking uh meta. So timber uh is our model, Assence our many-to-many relation. True, when we do not define count true table, then Django uh implicitly creates Creates many to many intermediate table and many to many model for us. So true is a model. And as for any other model, we can inspect meta and check db table That's how we figure out that Django for this auto-generated intermediate table expect uh test up climber ascends. table. We could do this we could do this uh the same for the ascent

Speaker 2: model but we already remove it. So as a fallback we can uh use sql migrate command to Check the alt migration files to check the alt migration file in which we added the ascent model and as we can see There is an operation, there is a square statement described as create model ascent, and we have a table name, test app assent. So we have both names. Now we can use them in separate database and state, and everything everything is ready. We have a migration that describes potentially not physical transition in the most efficient way. So without any data migration

Speaker 2: And it's even reversible. It's even uh reversible. You can ask if That's maybe an edge case that this special operation covers. Of course not. There are plenty of other use cases. An opposite transition. So changing a many to many fields to use true model is even described in Django Docs. We have At least few changes in project states that do not require any database changes, like modernizing your indexes or constraints definition from the old way with using index together, unique together, db index or uni flux to a new meta indexes and meta constraints. So we don't Indexes, we don't want to recreate them, we don't want to rename

Speaker 2: the same place, and we can do this with using separate database. It's described in uh details uh in Adam Johnson blog post It's uh also a great solution for moving models between because it's also not obvious how to how to do this When you want to move model from a source app to a target app, then you can use separate database and state to simply first move it. then uh generate migrations on a source app, move removed uh model operation to uh uh state operations and in the database operations use rename to rename it from a name expected

Speaker 2: I'm expected for a model in a target app. And in a target app, do the same

Speaker 2: Thank you for having me.

Speaker 2: I would say that it's not necessary for most of users, and secondly, it would require to generate two migrations uh in a single step and then users uh would need to control that they deploy only the first one two instances and then uh leave the leave the second one for for a second step yeah for example so it it would be tricky

Speaker 1: And you are the release maintainer also.

Speaker 2: I cannot allow

Speaker 1: Any more questions? Yes, in the front.

Speaker 3: Yes. That you made a first migration to delete um some rules with uh true values on the inf and then you need the second one but Yeah, do you recommend to separate these or you can also make all the steps in one creation? So is pros and cons of this

Speaker 2: I I made it that way. Uh Because uh in in some databases it's not possible. Yeah on on PostgreSQL you cannot you cannot delete uh rows from a table and in the same atomic transaction uh delete a uh remove a column from from the staple. It's it's simply uh so PostgreSQL does not allow for for for that. That's why I split it into uh multiple steps.

Speaker 1: I think there was another question. Yes.

Speaker 4: When doing um data migrations with web python, it's uh really annoying sometimes that it It's impossible to use uh properties on models. Sometimes you have to repeat a whole bunch of code because you have a property you would calculate some value or something, but they're not available inside the web Python. Will that ever be uh changed so properties are usable in uh run python

Speaker 1: uh so if i understand the question correctly you cannot access special properties methods inside the run python uh Yes, and will that be possible someday?

Speaker 2: So uh you cannot access them for exactly this reason that that uh I I mentioned in this talk that the migration framework Does not use uh models definitions from models. Yeah, it recreates model each time uh by inspecting migration files So properties are not available when you are doing this that that way. So keep keeping uh so to to to make this work we need to keep properties in migrations because they may not be for example backward compatible yeah you may in properties use field that are not exist

Speaker 2: anymore Or you may change these properties in their life cycle.

Speaker 1: It sounds difficult.

Speaker 2: Really difficult.

Speaker 1: Any more questions? Yes.

Speaker 4: Yeah, just continuing that.

Speaker 5: Couldn't we have an operation to add properties to the migration state?

Speaker 1: So the question

Speaker 5: that this model has this function.

Speaker 1: Theoretically can migrations be aware of uh methods of models.

Speaker 2: So I I I would say that that's exactly the same issue as we have when we for example uh squash migrations. Well when you squash m when you squash m uh when you uh squash migration then you also uh cannot And so you need to manually backport functions that that that you use in in uh migrations that that were squashed. Yeah. Uh so Here we will have exactly exactly the same issue that keeping properties keeping properties in migrations would be As complicated as keeping any other functions in

Speaker 2: migrations. So we we need to have a way to uh to to uh serialize them somehow. Yeah.

Speaker 1: So you are also saying that you're kind of free to go yourself and put those in your migration files and then you can use them. Yeah, I uh saw that you used this uh run python no up uh noob function that I thought was really clever. Um but is there any best practice around these um reversible uh non reversible uh Um sometimes I feel bad that I'm actually causing data loss by removing uh uh a field and then pretending nothing happens if people go the other way. Uh s is there an exception you can also throw?

Speaker 2: So uh it depends. In in uh a case like uh I I I described in in this talk, it was not necessary. Uh in some cases uh It's possible even in a run Python to recreate remote things because for example you can uh rely on some other uh properties or or models. Like for example in built-in migrations we have uh a content type migration that is also reversible but it's quite tricky to to reverse it when we uh change uh when we change columns in in uh

Speaker 2: uh in in per I don't remember in permissions uh uh at at some point in Django. So it also we we change from two columns to one column, then it is reversible in Django. It's this step is reversible, so it recreates it again. But it's it's it's hard to achieve. I would say that if it's feasible, then we should make any step reversible.

Speaker 1: All right. Um yeah. Th isn't any more time for questions now, but thank you so so much. My wish

Speaker 2: thank you.

Speaker 1: And uh make a direct change without any break at all. And uh