Deploying Django migrations at Kraken scale with Tim Bell

Thanks everyone. Um my name's Tim Bell. Um I'm based in Melbourne, Australia. I'm a senior software engineer at Kraken Tech, which is part of the Octopus Energy Group. We build the Kraken system to empower businesses to lead renewable energy transition from generation to supply. I'll say a bit more about the Kraken system later on. For those who are close enough, this is Katie , one of our mascots, and you'll see her on the slides as well as on the corner of my podium during the talk. There'll be a QR code at the end of the slides in case you'd like to follow the links that I'll show at various points. What is a Django migration? If we quote from the Django documentation, migrations are Django's way of propagating changes you make to your models, adding a field, deleting a model, etc.

into your database schema. In this talk, I'll be covering issues that arise when deploying Django migrations, including ones that emerge using Django at the scale of Kraken. But to start, let's look at some deployment issues that are relevant to Django used at almost any scale. By the way, the Django docs don't say much about migrations on the deployment page. You can see I've searched for migration and nothing turns up. And they don't say much about deployment on the migrations page either. But there are plenty of articles about issues with deploying migrations, and we've developed our own documentation, some of which came out of incidents we caused when doing a bad thing.

So let's start with a simple scenario involving an application where downtime can actually be scheduled. Here, downtime is available for doing maintenance work. This diagram shows time passing was moved from the left to the right. And it starts with the application running, and then we stop the application to begin the downtime. During the downtime, the migration is applied. Then when the downtime is over, the application resumes As long as the application is down while the migration is being applied, most of the issues that this talk will cover aren't actually relevant. But usually these days, regular scheduled downtimes are a luxury or not possible at all. Or maybe you're allowed a small amount of downtime, but longer periods of downtime, like six hours or more, would be unacceptable

So let's look at a no downtime scenario. Your application is running while migration is being applied. Here's that diagram from before updated. The application is running the whole time and for a short period there is a migration being applied as well. That means that your application might be trying to do selects, inserts, and updates on the table that's having a schema change applied to it. Not surprisingly, that can cause issues. We'll look at some of those issues in just a second, but first, a note about terminology. Django has some words for things like field and model that SQL databases have different words for, that is column and table.

Django query sets use . ,. filter, . get, etc. and those turn into select queries in SQL. And other Django ARM operations turn into insert or update queries in SQL. I tend to use the Django and the database terms somewhat interchangeably in this talk. Also, this talk specifically relates to PostgresQL, aka Postgres. Although although the various concepts described may apply to other database systems, I won't be covering them explicitly. The first issue we're going to look at is a slow migration. Here we're starting with a model with an integer field. We're going to change that to a positive integer

field. So we've run make uh we've made the model change there When we run make migrations, we get an alter field operation as expected. Note that I've admitted the imports and the migration dependencies there When we run SQL Migrate on the migration, we see that the SQL that Django will execute to apply it. And a quick aside, using SQL Migrate to see what your migrations will do is a good habit to get into. Postgres doesn't have a dedicated type for positive integers, so Django implements the positive integer field by adding a check constraint that checks the value is greater than or equal to zero.

Note that the rows of the table don't need to be rewritten. We'll see later a migration which converts a data type that does require rows to be rewritten. So here's our add constraint SQL again. The Postgres docs warn us about two issues here. It has to scan the table to verify the constraint. And the operation requires a lock, which will be held for the duration of the operation. Now Postgres uses locks to control concurrent operations on the same object, such as a table. And that lock will prevent not only other updates to the table, like updates and selects, inserts, but will also prevent reading via selects.

Preventing the application from accessing a table for a period of time will cause essentially an application outage. How long it lasts depend on the size of the table. So how do we make this change without causing an outage? The very next section of the Postgres docs gives us the answer. Postgres allows a constraint to be added as not valid, which skips the table scan. But the constraint will be enforced for subsequent inserts and updates Then a second operation is done to validate the constraint. That validation operation does not need to lock out other rights, since the constraint is already enforced for any inserts and updates that happen.

Since it doesn't need an exclusive lock, the application can continue operating as normal while the table is scanned to validate the constraint. So let's look at how to implement this two-step process in Django. Django 4. 0 added two Postgres-specific migration operations to address just this issue Add constraint not valid and validate constraint. We also need to specify migration operations that affect the model state separately from the database. And they're going to be two migrations. Add the constraint as not valid and validate the constraint. So here's the first migration. The state change makes the field a positive integer field

That's the first bit under the separate database and state. The database operation adds the not valid constraint. using that uh Postgres operations add constraint not valid operation. Note that the name of the constraint, which is my app underscore, my model underscore, number underscore, some hex string underscore check. was taken from the output of SQL migrate from the original migration. That way Django will know that this constraint actually implements the constraint that it would have generated itself The second migration is we we create an empty migration file using make

migrations-empty and then edit it. And this is what we put in. We add the validate constraint operation. And once again we use the the name from the output previously from SQL Migrate to name the constraint we need to validate. Note that we don't need to use separate database and state here because the validate constraint operation doesn't define any state change. So here's the SQL output, SQL migrate output, and I've left off the begin and end. The first migration operation requires an access exclusive lock, which is the case for most alter table operations. And the second migration requires only a share update exclusive lock, which doesn't prevent read and write operations.

So going back to our diagram, one long disruptive migration has been replaced by two migrations. A very short disruptive one. And a long migration that does not block read or write access to the table. So the application can just keep running as normal. Success. We did this with a check constraint to ensure the integer was not negative. It works basically the same way to add a not null or other type of constraint. There are some key techniques here that turn up again and again. Splitting one migration into two or more. Reducing the period of time an exclusive lock needs to be held and special handling for an operation that needs to process each row of the database.

Another example of a slow migration is create an index. Here we're adding an index to the integer field from before Once again, the issue is a long-running operation that requires a lock that blocks some operations. Rights. It requires the share lock. It's not a very informative name for a lock. I'll talk about locks more shortly. Postgres provides a solution via the create index concurrently operation. So we'll take our existing migration and we're going to edit it to replace add index with addindex concurrently. The add index concurrently operation uses Postgres's Create Index Concurrently command to build the index. And that command requires only the share

update exclusive lock, which permits reads but also writes while the index is being constructed. Note that the migration file must include atomic equals false, which I've got in bold there. And have a look at the Postgres docs for more information on how create index concurrently works and why it can't run within a transaction. There'll be more to say about creating DICX Gun currently later in the talk. Here's that brief aside. How do you know what locks an operation requires? Here are three great sources of documentation: the Postgres docs , a page called the Postgres Locks Explorer, and a page called Postgres Lock Conflict. Now the latter two also include what operations

one particular lock will prevent from happening on the same locked resource. Alright, back to this statement. Your application is running while the migration is being applied, but your application is also running while the application code is being updated. There are various ways of deploying new code, for example, canary deployments or blue-green deployment. In almost all cases, old code will be running at the same time as new code. So, the old version of your application code will still be running for a while after the migration is applied. The change the migration makes must therefore be compatible with both old and new versions of the application code. And we'll illustrate that by looking at a drop

column change. So if we naively attempt to drop a column with a single migration, there will be issues. So in this code here, I've taken the number equals models positive integer field from before and I've I've just commented that out. And then I've run make migrate make migrations which creates a remove field. Once the migration is applied, the old code will still try to include the column in select queries and will still try to write to it, both of which will cause errors. That's because Django always enumerates the columns in select and update statements rather than doing select star, for example. So how do we make this change backwards compatible? As before, we need to split it up into multiple migrations.

In this case, we're going to need three migrations. So, first of all, we need to remove all uses of the field from the code, which presumably you're going to do because you're going to delete the field. Then we make the field nullable in the first migration. So you can see null equals true in bold. In the second migration, we're going to remove the field from the Django state. That means their new application's code won't know about the field. This is a no-op in the database. Finally, in the third migration, we drop the column in the database. Now this needs to be done in SQL since the Django ARM doesn't think the field exists anymore. So here's what the process looks like, with four versions of the application code and the three separate migrations.

And note that we need to ensure that each migration has been deployed successfully and the old application version has stopped running before deploying the next. So, next topic is locking hazards. Yes, you can cause a major outage without a table scan. Waiting for locks. So when you run an alter table , which is what you're often doing with a migration, it requires an exclusive access lock. Which conflicts with all locks, even the ones taken by select statements. So it waits for any existing locks to be released in order to acquire the accessive exclusive lock, and that might take a while

While waiting, all the other queries will wait as well in the lock queue. That's shown here in the red circles where we've got normal queries waiting and that will cause an outage. I talked about this at DjangoCon Europe last year, where we had a very significant outage caused by just this scenario. So how do we deal with this? Postgres provides a lock timeout setting. And we can use a small value for the lock timeout, a few seconds maybe, depending on your application and how responsive it needs to be. An operation that fails to acquire a lock it requires before the lock timeout expires will abort with an error. And when it aborts, any other operations that were waiting behind it in the lock queue are then able to try to acquire their respective locks.

So the application can resume operating as normal At Kraken, we've released a small package, open source package with some useful tools, Django PG Migration Tools. one of which is the migrate with timeouts management command that sets a lock timeout automatically when applying Django migrations. So you use migrate with timeouts instead of the usual uh migrate command. So as promised, we're back to create index concurrently. It turns out that using lock timeouts by default has added a new hazard that we need to handle Add index concurrently doesn't use an exclusive lock, but it does still acquire locks at several points during its operation.

If a lock timeout is set and gets triggered, then the operation fails. And when create index concurrently fails, it leaves behind an invalid index. If if we then retry that migration, which your deployment mechanism might do, um And we've had a ad index concurrently fail, it will fail again because the index already exists and it's trying to create an index that already exists. So to avoid these problems, use safer add index concurrently from Django PG Migration Tools. What this does, if a lock timeout is set, it resets it to zero, since Create Index Concurrently doesn't lock out reads or writes, so it's perfectly safe

for it to take a while to acquire a lock. If an invalid exists, sorry, invalid index already exists, it removes it first, and it adds if not exists to the create index concurrently command. So this makes it possible to manually create the index outside outside of applying migrations. Why would you want to do that? Well, it might be needed at Kraken Scale. Which brings us back to the full talk title and the question, what is Kraken Scale? Kraken has many components working together. I'm talking here about the main Kraken Core application, which is about nine years old at this point. 374 Django apps when I last counted, over 2,000 modules, over 10,000 migration files.

Some database tables have billions of rows. And we've got 23 production instances in eight different time zones around the world, which means that there's no such thing as outside business hours across all of the instances at once. We do about 150 to 200 deployments a day, and we've got about 500 developers working on the code base all around the world. So there are implications of crack and scale. Deployments happen at the same time, and it might be middle of the day in Australia and the middle of the night in the UK. So there's no simultaneous quiet period where we can do deployments for all the instances. Per row operations like table scan or backfilling can take a long time on some of the larger instances.

And we'd like to avoid manual per instance work. But we want to do long-running operations like index creation for larger instances outside normal deployment mechanism. So that's a contradiction. And we often want to fix forward problems by faking failing migrations to avoid blocking our deploys. One of the implications of having so many developers is that most of them deal with Django migrations very rarely. So we need to support them as much as possible to help them avoid issues. So we have documentation, we have some helpers in the make migrations command, and we've got some stuff that happens with pull requests to provide useful information about their migrations. Next, we'll look at two examples of the implications. First one is failed migrations, and the second one is backfilling last tables,

large tables. Here's a new scenario to consider when deploying a migration to multiple instances. So if a migration fails in only a few instances, which I've shown here with the the first migration, um, it might be faked to allow the deployment to continue. That basically marks it as applied even though we haven't actually changed anything in the database But if the migration fails and is faked, the new version of the application code will be running before the migration is actually applied. And the change the migration makes must be compatible with both old and new versions of the application code. So the new version of the application code must be compatible with both database schemas after and before the migration is applied.

That means that application code that depends on the changes introduced in the migration can't be in version two of the application code, as shown here. It needs to be in a subsequent version, version three or something. Another example is changing a column. Earlier we changed an integer field to a positive integer field And that change had compatible data types, so we didn't need to rewrite each database row. But if we change from integer field to a big integer field We're changing the data type in a way that requires each row to be rewritten, since a big int is twice the size on disk as an integer. So, for a big table with billions of rows, this change shouldn't be done

in a single step. Again, the solution is to break the change up into multiple migrations, and in this case, as well, a backfilling step. So we've encountered this problem previously with Kraken , and it's encountered commonly when converting integer primary keys and foreign keys to big int. It's a multiple stage process involving four migrations along with a backfill process. And backfilling is where we copy the value of the old integer column into the new begint column for each row of the table. Since it eventually has to rewrite the whole table, it can take a lot very long time. So once again, we follow the same general principles as before as splitting a single migration into multiple migrations

and avoiding long blocking operations. Step three here is the challenging one at Kraken Scale. So the current state of the art for backfilling is to run some SQL manually on each instance out of hours This is the SQL. It manually it backfills in batches and pauses between each batch so it doesn't dominate the write load on the database. It's pretty ugly, which is why I've put it sideways and small so you can't read it Because it's so bad, we're considering some alternatives to this manual SQL. Async migrations by post hoc. Don't particularly like the name, but it's a great idea. It provides for operations similar to Django migrations, but they're not linked into the Django migration system

or dependency chain, and they can be run at arbitrary times. Another alternative is a housekeeping framework that we're working on that's similar to Post Hog's async migrations, but it's much more general purpose. My own proposal is to automate backfilling tasks via a cron job, which backfills in batches, as the SQL did, but it waits for the database to vacuum up old dead rows between batches. So that it's always, it's not going to keep running while a vacuum is happening at the same time and adding extra load. And when I get back to work, I'll begin testing the third of those options. as part of a larger project to convert integer big keys, uh integer primary keys to big end.

So in conclusion Django migrations can be tricky to get right. You need to watch out for table scans, you need to break single changes into multiple steps, and you need to watch out for locking. And improving deployments for Kraken Scale is still a work in progress. So here are some packages that help implement some of the ideas presented in this talk. Including our Django PG Migrations Tools package that I previously mentioned. Here are some articles and other helpful info, as well as the talk I gave last year about a migration-related incident. And as promised, there's the QR code so you can get these slides and get all of those links. And thank you very much.