A Pony On The Move: How Migrations Work In Django 🐎 - Markus Holtermann

Django has a built-in migration framework and system since version 1. 7, which was released in 2014. Most users of Django will only ever get in contact with it when using the Make Migrations or Migrate commands. Some may also look at a few other commands such as Show Migrations or SQL Migrate. More advanced usage may include writing migrations manually, for example, for data migrations, or to do things Django cannot do by itself. Today's talk is going yet another level deeper. It's about how the migration framework actually works. For those of you who don't know me, I'm Markus

Holtermann. I'm a senior engineer at Crate. io. We built a database CrateDB, which is perfectly suited for industrial IoT-scale time series data. We also build the infrastructure around hosting the database for you so you don't have to do that yourself. I'm also a member of Django's operations and security teams. I started using Django in 2010 And in 2014, just after the original author of Django's migration framework, Andrew Godwin, merged a pull request that added migrations to Django I started contributing by fixing bugs and adding smaller features to the migrations all over the place. Given my involvement in the migration system at the time, I'm still considered one of the few people who understands how it works

and and who knows its quirks. These 17 blocks are the main parts of what make Django's migration framework. And I want to try to bring some order to them. You might notice that I didn't include the management commands, nor did I include the individual database operations. The management commands use some of these components, but they are not particularly complex compared to the rest. The individual migration operations are grouped together as operations in the second row on the right. I think the best way to structure this pile of boxes is to not sort them alphabetically

as it's done here. Instead, I'm going to arrange them with some arrows, showing which component is using which other component. That means A uses B will be shown as an arrow pointing from A to B. That is what I've done here. As you might have noticed and also might have expected from the previous slide, there is a lot going on here. And I do mean a lot. I've also omitted some errors to keep it readable. Since this talk is kept at 25 minutes, I'll be ignoring some parts in the remainder of this talk.

But just because I don't talk about them, or just because I only talk about a component very briefly, do not think they are not important. Each of these boxes is fundamental to the overall working of the migration framework. So here's what we are going to look at in the remaining 20 minutes. We'll be talking about the core of the migration framework, the applying and unapplying of migrations, and how Django figures out what you changed in your models to then create new migration files. I will highlight the boxes as I'm talking about them, and gray out all the others that weren't covered yet.

Essentially, we are going to be the pony that's migrating through this graph of boxes I want to start with these three pieces: model states, project states, and state apps. If you want to define a hierarchy of importance among all the components making Django's migration framework, these three are by far the most important ones. Nothing else works without them. When we talk about migrations, we are almost always talking about the timeline of your project. And we look at the state of the database and the corresponding models at some point in the life of your project. A project state is essentially a snapshot at an arbitrary point in time on that timeline

A model state represents a single model at some point on that timeline. In many ways it looks like a Django model, but it isn't. It's a light white class without a lot of magic or logic, because this class is instantiated hundreds, often thousands, and sometimes even millions of times. It contains a model 's name and app label, its fields, indexes and constraints, and other stuff like that The state apps are pretty much an app registry as you may know it from Django Apps Registry Apps. The place where all models of your Django project are registered.

Its purpose inside the migration framework becomes more clear when I will be talking about the executor later. For now, let's say it's an app registry with some models at some point in time, pretty much like the project slid. I just said a project state is a snapshot of models at some point in time. What that means is whenever you or Django changes something in the models, such as deleting a field or adding a model That is a new state of your project. Because your models before and after these events are different. And in Django's migration system terminology, these events are called migration operations.

And that's what we are going to look at next There are about two dozen migration operations. They are for example Add Field and Remove Field, Create Model and Delete Model. but also less commonly used ones like Auto Order with Respect to, Remove Constraint, or some specific ones for PostgreSQL, for example create extension. Operations have two tasks. They look at an instance of a project state and then mutate the model states that are in the project state It's important to know that there's only ever a mutate forward. The method on the operation classes for that is called state

forwards. The second thing operations do is deciding on what to do with the database. For that, and unlike for the state. There are the database forwards and database backwards methods. They talk to the schema editor and ask it for example to create a new table or add a column to an existing table. A migration is a bundle of these operations. All operations within a migration are executed in a single database transaction. That is, if the underlying database supports that. A single migration is also always bound to a single Django app, which

means all operations within the migration work on the same Django app. But, and I can't stress that enough, each operation has access to the whole project state, and thus to all apps and models. That is important when migrations deal with forrowing keys. Because let's say you change a model's primary key from an integer to a string. Then the Alter Field operation will go through all models and check if there's a field that points to the model you just changed. And if there is, it will change that field's database columns data type as well.

Now that we know how the migration system keeps track of state And how to mutate the state and talk to the database, how does the migration framework know in which all of the changes are applied? The answer to that is the migration graph. And I'll only briefly cover this one. In a mathematical sense, it's a so-called directed acyclic graph. A migration has two, one might argue three attributes that the graph considers for ordering. First and most commonly used, dependencies. That's a list of other migrations that need to be applied before the migration in question can be applied.

Let's say you have a migration that creates a model and another migration that adds a foreign key on another model to that model. Well, the target model needs to be created first. Otherwise, the foreign key constraint can't be fulfilled. Secondly, barely known and less often used, is the attribute run before. It works exactly like dependencies, just the other way around And thirdly, there is a replaces attribute on a migration, which comes into play when you squash migrations. And it essentially replaces a set of migrations with a single other one. Now the question arises, how does Django

know about migrations in your project? And the answer to that is the migration loader. The potentially obvious task for the migration loader is loading all migration files from all apps in your Django project and adding them to the graph that we just talked about. As part of that, the loader is actually going to import the Python file that contains the migration and instantiates the migration class with the app label. This is where the migration gets the app label from, since it's not an attribute on the class defined in the file. But building the graph is easier said than done. Because remember how I just quickly skipped over the replaces

attribute on migrations? And how it's used with migrations squashing? Building the actual graph in the migration loader will need to account for applied andor unapplied migrations. The graph may go and replace some migrations with a replacement. But if some of the replaced migrations have already been applied, then the replacing one cannot be used Instead, the remaining replaced migrations need to be used. The loader does a few more things. For example, it checks that the migration history is sound. If you have applied migrations, but some of its dependencies haven't been applied, for whatever reason, it's going to yell at you

And lastly, the loader provides an interface to create a project state from the underlying graph. At this point, let's recap what we've talked about so far. We know how to load migrations. We know how to turn them into a graph. And we know what defines their order. And we know the underlying data types that represent a database state at some point. Which brings us to our next component. The executor. It's the brain of applying and unapplying migrations. From the outside, there's only one important method: migrate The method takes one required argument, targets.

Targets are the names of the migrations that you want to have applied or unapplied at the end of the method call Which means if you want to apply your whole project, it's a list of all the last migrations in each app. And these migrations are also called leaves. From those targets, the executor will attempt to build a migration plan. If you're curious, you can use the show migrations management command with the dash dash plan option to see what the plan would look like when you apply the whole project As part of building the plan, the executor will look at which migrations have already been applied and which ones are still outstanding.

Or vice versa, the migrations that are applied and need to be unapplied. Now when migrating forwards, that is applying migrations, the executor is going to start with a fresh project state and then iterates over all migrations in the plan and calls mutate state on each migration. Each migration will then in turn call state forwards on each migration operation. And that builds up the representation of your database operation by operation. And it does that up until the point when the first migration will need to be applied. Which means up until the point when the s database operation

need to talk to a schema editor Because at that point the most crucial part of the migration process occurs. Rendering models After the rendering, the executor is going to call the apply methods of each migration, which will mutate the state forwards operation by operation and also applies the database changes operation by operation. Before I go into the unapplying part, I want to demystify the rendering of models. Remember how in the beginning I said the model states are just like a model? And how a project state knows about all model states at a given time?

Modeled classes and their fields have references to each other and among each other. For example, you can use a models underscore meta API and get all fields on the model. And each field will in turn have a model attribute, pointing back to the model it is on. And if you, for example, add a foreign key on a model A, pointing to model B then Django will automatically add a reverse foreign key, which is a many to one rel from B to A, which is effectively a field on B And the fields themselves will have attributes related model and related field, which points to the corresponding part on the other side of the relation.

So adding a foreign key to A not only changes A, but also B, and the reverse foreign key on B, which changes the foreign key on A. And just with a few models and foreign keys, you will end up with millions of pointers that the migration framework would need to keep track of. And to avoid that, the migrations work on model states, which do not have these references. And because the schema editor only works with model classes, the model states need to be converted into them. And that's called model rendering. It is a far better approach than working with model classes to start with, because keeping track of these pointers is pretty much impossible.

And you can trust me on that because I've spent literal days debugging issues inside the migration framework where pointers were stale and pointers were cause of a problem. There's an infamous ticket 23745, which is about caching the rendered model classes. For those of you who have been around long enough, that is from Django 1. 7 onwards Migrations got a significant speed boost in 1. 8 with this ticket. There's also a lot more into rendering model states and project states that I'd love to talk about, but don't have the time for in this talk. Like for example, figuring out when to evict some cached model classes, which is pretty much enough content for a whole talk.

Now for the unapplying part. It starts off like the applying part by generating a plan to follow. But since model states can only be ever be mutated forwards, the executor will cache all intermediate project states for each migration that will be unapplied. If you've ever encountered a huge memory load during unapplying migrations, this is why. Once the project states have been cached The executor is going to call the migrations unapply method. And it's doing that in reverse order of the plan. As a last trip of our journey through Django's Migration Framework, I want to briefly look into the autodetector.

The autodetector looks at your current models and the current project state represented by all migrations in your project, and then tries to figure out what operations need to be added in order to get your current project state to the state that resembles the models in your project. At this point, I was originally going to say nothing is as simple as that. Because on the surface, it looks kind of simple. But the devil is in the details. There is this method DetectChanges, which synchronously calls dozens of different methods to generate the changes needed for new and removed models.

added and removed indexes, and everything else you can do to a model. And each of these generation methods more or less works similar. They are iterating of all models in the project state represented by migrations and the project state representing all models in your Django project. And for each model, they then do their thing. For example, they compare a list of fields on a model and then decide that two fields were added and are not in migrations yet Thus two add-fields operations will be added. And it's actually these generating functions that hide the complexity.

And some of them are a few hundred lines long. Now we have reached the end of our journey through Django's migration framework for today. As you've seen, there are numerous parts involved. And I can only encourage each and every one of you to take a look at the code and explore it. And if you've been hesitant or scared away by its complexity so far, I hope this talk made you curious to learn more And we've almost reached the end of the talk. I want to take the opportunity to answer some frequently asked questions.

A question that pops up repeatedly is around the field attributes, such as why does changing verbals name or choices cause a new migration? And I hope you may now know the answer to this. The answer is because the model state for the corresponding model changes. And the reason why we can't filter out some of these attributes is that nobody knows which attributes somebody may use in some migration. Like choices, for example, it doesn't have an effect on the database when you create a f or add a field. But it could very well be used for data validation inside a data migration, so we need to include it.

Another question is about alt-field classes and why they can't be removed. And again, the answer is because of model states. If there's only one migration in your Django project that refers to that field, well the field class needs to stay around because it's imported in that one migration You can use migration squashing to possibly get rid of the field. And if that doesn't work automatically, maybe try writing the squashed migration by hand. But as long as there's a single migration that still imports and uses that field, the field class needs to stay around.

When I explained what model rendering is, one might ask why does the schema editor not just work with model states instead of model classes? And the answer to that is twofold. Firstly, the schema editor is part of the database backend and doesn't know about the internals of the migration framework. Making it work with model states would mean opening up some of the internal API, such as the project states and model states. Which would not necessarily be an issue because the benefits almost certainly outweigh the cost in this case. Secondly, because that change needs to happen in a backwards compatible way. Which is where

it gets fairly tricky. It's not about the built-in database backends. The schema editor has a public documented API. We will therefore need to provide a proper migration path covering one LTS cycle, which adds a lot of maintenance burden But there is this ticket 29898. That is precisely about this. Adapting the schema editor to operate from model states instead of rendered models. And this was also a project suggested for this year's Google Summer of Code. And from my perspective If anybody wants to look into improving and speeding up migrations, this part

is probably the most valuable one. And with that, I want to thank all of you for joining in today and listening. I want to thank all the organizers for DjangoCon Europe 2020 for running the show. Thank you