Django migrations, friend or foe? Optimize them for testing

Speaker 1: Will this talk be about? Welcome on stage. Benny?

Speaker 2: Thank you. Just a second. Let me stop this. Well, okay.

Speaker 1: We'll do it again. Welcome on stage, Jenny. Big hand.

Speaker 2: Thank you. Hi everyone. Like I said, I will talk about Django migrations. Are they a friend? Or maybe not always And how to optimize them from testing. Quick words about me. I am Danny, the one on the right, if you are unsure on which one. And I'm a Foodstack developer working with everything. JavaScript, sorry for that. Python, a little bit of Go and Rust recently, and I work remotely for on fingerprint from it. But first thing first, I'm the last thing separating you from the lunch break, so please don't think about

Speaker 2: food. Food. Forget about food for a moment. So let's talk about migrations. Migrations are a way to propagate changes to models in to your database schema and in Django we have uh useful commands for migrations. For example, make migrations, migrate, show migration and SQ and migrate. Let's inspect each one of them. The first one, make migrations, creates new migrations for your Django application and synth is this one so you can create optionally an empty migration customizing it with your Python code and you can even define a custom name

Speaker 2: for your migration otherwise Django will automatically suggest your migration name. Oh wait my computer will restart in 30 seconds. Not now. Not now, sorry. Okay. So make migrations. Let's try with a custom model called oh sorry, pizza Well, let's create some sort of Twitter for your favorite pizza. So you create your pizza model we created by uh linking to a user create a death date time and a text field. The text field will contain a validator to check that you one place in the text pineapple

Speaker 2: because it's against the law. I'm sorry, I'm from Italy and it's against the law. Over there, I don't know here After that you can create your migration and in your Django directory in your project directories you can find your initial pipeline file containing the migration. Inside your migration file you may see that you have this class migration with initial equals to true because it's the first migration of the project and then uh a list of dependencies just one in this case because we are relying on the out user model and apart from that a list of operations

Speaker 2: again just one in this case because we are creating our model model. Then you can use migrate to apply your change to the database schema, specifying optionally the application label and also the migration name so if you just want to apply all migrations to your database you just run migrate otherwise you can select migrate application name admin in this case and zero in order to roll back everything to the beginning without models applied to the database and you can even move forward and backward between migration specifying the migration name, the full name, or just the number.

Speaker 2: And then again apply all migrations to your admin. But how does this work under the hood? In your database you will have a Django migrations table containing These fields and in the table you may see a list of applications, migration names, and the daytime when they've been applied to the database. So you can see in every point of time which migrations have been applied to your database. If you're lazy like me, instead of opening your uh database and navigate through tables and data you can use show

Speaker 2: migrations and you will have a list in this case by application a list of migrations with a a tick if the migration has occurred already been applied to the database. Other than this, if you want to inspect the migration code on your database, you can use SQL Migrate in order to see the SQL statement of a migration Please note that every single migration will happen in a transaction. So before every migration SQL code, you can see begin and commit. and the end because every single migration happens in a um in a transaction Because if something

Speaker 2: goes wrong in here, then everything will be rolled back before the migration without leaving you in a State in a bronze state, so you can always retry the migration, correct your code and retry the migration. Now, what if you want to increase the length of a field? Well, you increase the field in your model, you create another migration, you can inspect the migration, you can see that it this is not the initial migration it relies on the first one uh one initial and it contains the alter field in instruction. The SQL migrate will show you begin, SQL command and commit

Speaker 2: and then you can apply the migration. Pretty easy, right? But what if you want to add further changes? For example, enable pizza like or repeats retweets for old school tweeters. And oh if you forgot things or you want to enable new models then you can create another model, make change. to your database and create another migration. And with show migrations you end up in having a list like this one that you can apply or unapply when you want. But now stop for real talking about food and talk about a more real application, for example a shop application.

Speaker 2: Because what if a customer needs your customers need a shop application handling customers? Then it asks you to add an ispremium field to customer just a Boolean field. Then you create everything, you create dedicated shipping addresses because each customer may have different shipping addresses then you need to migrate the data from the customer model with a single address in it to the new model and then you need to remove the customer customer shipping fields. So let's see the highlighted point. What if you want to create custom code? Well you can create an empty migration

Speaker 2: and inside it you can run Python code with this uh syntax here. If you don't want to run for example the backward function you can add migration run Python no hope so you can specify that you don't want to run custom code when rolling back the migration and the migration code will be something like that so a for one function with a schema editor that you can use to extract the database alias in order to use it to extract data from the database. And in this case, the customer model will use the model in that specific point of time. during the migration

Speaker 2: and not the current model in your Python code because when you deploy, you deploy your last code, last version of the model. But in this case, in the migration you need the current model not the last one and you can use it like this. Now the customer calls you or emails you and ask for other changes. For example, oh I need to increase the length of the shipping address because I want to ship in an exotic place with a long address or add models, add fields and stuff like that, and then you may need to um you may need to rename a field.

Speaker 2: In order to rename a field, you if the migration contains just that change you can just rename the field in your model, run main migrations, and Django automatically detects that the field name has been changed. and we'll propose you to rename it. Then if you want to provide initial data for your tables, you can, you don't want to, but you can create a migration creating data inside the migration or even better you can create fixture files containing your data Fixture files are like JSO files

Speaker 2: that you can dump directly from your database using dumped data and the application and the table name in a JSON file and it will contain code like that included a primary key for each record record. Then if you want to load data from those JSON, you can use load data, application name and the fixture name. But what if you fear to if you are afraid to forget to run this command when deploying? Well, you can call load data inside a migration, a custom migration, using the call command imported from Django

Speaker 2: core management and loading data in the same way inside the migration so you can forget about writing documentation. No please never forget about Writing documentation. And what about tests? Well, uh in tests, in your test case, you just need to add fixtures with a list of fixture file names and That's it, you will find your data inside your test database. But now we have a lot of migrations in our application. And of course during development and deploy they are great because you apply just the migrations you need.

Speaker 2: While in tests, it's a different thing. As a disclaimer, timings may change from laptop to laptop. So next slide. Time was taken from my previous laptop, but it's similar. similar to this one. So running tests, simple tests for these uh demo applications, uh with 20 applications like Shop It takes just one second. So that's it. You may ask why are you talking about optimizing one second of tests

Speaker 2: on 20 applications. Well, uh when we deployed this pro the working project on uh GitHub we were using GitHub actions and we pay for uh the time we use for actions. The problem was that this single second In reality was uh like 30 seconds roughly because 20 seconds for creating the test database and just one second for running tests Okay, single execution in a single day, not a problem, but what if you have a lot of these tests running during your

Speaker 2: month then you can exceed your pro period and you need to buy the enterprise one so how can you optimize them a first workaround is to set migrate equal to false in your Django settings file and when set to false migration won't run when create the test database because it creates the database from starting from your model. It should be Perfect, right? Well, not exactly because this one, okay, it's a single line change and it doesn't run migrations during test, but it's like make migrations. So in needs to analyze your models and create migration

Speaker 2: and then migrate before running tests. So in our case it adds five seconds in in your test execution time. Another workaround is keepdb. So when you run tests you can add these uh option here and this will preserve the test database without deleting it between test runs. If it doesn't exist then it will be created. first time and other migrations in the few in future test runs will be applied in order to keep it up to date Pros and cons. Well, running tests locally it's perfect because it saves 20 seconds for each test run

Speaker 2: because it keeps your database structure intact Not a real problem, but primary key values are not reset between tests and they can increase at infinitum. So they can become really big and if you are running a lot of tests every single commit, every single code change, well they can overflow Yeah. The worst thing is that it's not easy to configure this in your CI CD because you Either need to cache or create artifacts in your GitHub workflows and this takes time too, so it's not a real solution. Or

Speaker 2: you can use an external test database, maybe a a better solution or thanks to a friend uh you can use Django Migration CI that it make you make easier for you to configure an external database For example, on AWS or somewhere else. Another solution. Thanks to Carl Frederick for Suggesting this solution. Another solution is to use a seed database. So you run migrations against an empty database, you dump the content of the database to a single a SQL file and then you add the file to your code tree and you restore the file

Speaker 2: on your CI database before running test. This is faster than loading it it or recreating it. So what about performances? Well with migrate equal false five seconds more, keep the B in local, zero seconds for creating the test database so yeah uh but on github actions using a seed db and kipdb so without recreating recreating it then you can spend just two seconds to run tests run creation db and then one second for the tests Now we can end here this talk, but we want to talk about squash migrations.

Speaker 2: This is a Another command inside Django Manage PI that squash existing an existing set of migrations into a single new one. Let's try to apply it to our shop application and let's try to compress our 26 migrations into a single one. You can have a couple of warnings. So for example, if you have custom Django function forward and backward, you need to um copy them because it's not an automatic thing with this Django version maybe in the future will be So you need to copy

Speaker 2: paste your forward and backward function function for each migration file inside the my the squash and migration file and optimize it for that. And then inside your migration file you can see that at the beginning of the file it contains replaces with a list of migration names your 26 migration files. The recommended process is to squash keeping old files, commit and release so in your uh Django migrations table it will be added the last line containing all the squash and migration the single file with the squash

Speaker 2: migration. Wait until old systems are upgraded with the new release and then you can remove old migration files, commit and do a second release If everything is fine, then you can transition the squashed migration file to a normal file, removing all migration file it replaces, updating all migrations that depends on on the other one and moving them depending to this one and remove your replace attribute on top of your migration. code. Then if you like you can prune all references to deleted migration with this command here. Let's test performances after squashing.

Speaker 2: Oh oh you spent a lot of time squashing migration, copy pasting code, but 90 seconds, just one second saved from before. Well , this wasn't a solution. So what's the point? Well the point is with squash migration you can move back from having several under migrations. to just a few because if you are like me that prefer to do small changes in your project create migration apply migration and go on then before your pull request to the main branch you will end up in adding a lot of migration files and if you want to make

Speaker 2: it easy for reviewers to review your code you may want to squash them in a single migration file. Oh but you wanted to speed up test well we have a couple of uh solutions from before, so like the C D B. But if you really really want to speed up test in another exotic way, so creating everything from scratch uh using as an inspiration the squash migration file so using replaces and a list of migration removing everything and recreating everything from scratch with a single migration well I tried that road

Speaker 2: don't follow me but if you want it Then let's talk about it during the lunch break. The timing is well you can save seconds, but it's not the best solution in the list because recreating everything with a single migration will save you 15 seconds in this case but the best one is the CDB. So after all of these tries, I've won. I found the perfect way of Optimizing migrations for tests, but at what costs? Well, it tooks a couple of weeks, maybe a little bit more.

Speaker 2: uh on my job to create this so I don't know if spending maybe two three weeks of my salary uh for my company was better than paying for the enterprise chunk of github. I don't know. Not a problem problem for me. Uh because I learned a lot. I learned a lot of strange and exotic way to work with uh migrations and I hope in your case this will save you from spending a lot of time trying things in order to check the perfect way to achieve this. And thank you.

Speaker 2: Thank you for being here. Thank you for the organizers from having me. And if you want to take a look at The example repository, it's over there.

Speaker 1: Thank you. We have time for like one question. Does anyone dare to stand in between a crowd? and it's lunch.

Speaker 3: Sorry people. Don't don't add me on the internet. Um Danny, uh lovely talk, very um useful. I was thinking about the C D B which I think is probably now your favorite, but with C D B you're basically creating a database dump between feels like. And my first thought is how is that going to ha work when like there are like multiple developers doing multiple feature branches, doing multiple migrations, and then who's going to do that C D B reset

Speaker 2: you

Speaker 3: thing.

Speaker 2: Yeah, thank you for your question. Well you need to remember at least to create or update the CDB after every merge to main to your main branch or develop branch if you have multiple deployee branches otherwise you will end up in having maybe a a not up to date C DB but at least it will contain your basic structure. and it will apply just a few migrations and not all of them. So it's better to remember to update it every time you change something. in your migrations every time you change or add a new migration file but

Speaker 2: if you don't remember to do it then it's not a so big of a of an issue but it's better for for us we try to remember to update our cdb after every uh in our case after every merge to our develop branch That is the main develop branch before going to production. But yeah, it's a manual thing, so you need to remember. Or yeah we can create we can actually create oh that's a nice idea. We can create a Git abection for example uh checking for checking if in the merge re

Speaker 2: in the merge request there is a migration change and then it will create it will restore the database, recreate it and commit it to an a new pull request. So that's a good idea. Thanks

Speaker 1: Thank you. So thank you so so much, Danny. another big applause for

Speaker 2: thank you thank you an amazing talk

Speaker 1: thank you um i want to say uh a few things about uh this lunch that you hinted uh in in the talk It's unfortunately cancelled.

Speaker 2: Oh, yeah.

Speaker 1: No. Um it's uh downstairs and it