Speed up your tests with setUpTestData

Good morning, bon dia DjangoCon Europe. It's lovely to be with you here again in Virtual Porto. I'm Adam Johnson and my talk to you this morning is speed up your tests with setup test data. This is going to be a five-part talk. First, we're going to cover unit tests, setup and teardown methods, what these do, and these are setup star because there's a class level variant of each as well. And then we're going to cover how Django extends the unit test framework in Django's own testing framework. And particularly we're going to cover What we go into detail on section three, set the setup test data hook, which is the core of this talk. In part four, we'll cover how setup

test data changed in Django 3. 2 to provide better isolation. making it much easier to use in our tests. And in part five, we're going to cover a whole all-in-one guide to convert a test case that is using the built-in setup hook from unit test to use setup test data instead and how this can speed up your tests. This talk is based upon material from my blog and my book, Speed Up Your Django Tests. And if this talk helps you speed up your tests a bit, then you can go a lot further by covering all the other techniques in my book. And uh during DjangoCon Europe, I'm offering a 33% discount on the book. This stacks with a 50%

discount offer to anyone who lives outside of the top 50 countries by GDP. So if that's your situation you can save quite a lot at current. On with the talk. So we have part one, what are unit tests set up and tear down methods? These methods are intended for you to do setting up of stuff and then tearing down of it afterwards. On a per-test basis, we have the setup and tear down hooks And then for the whole test case, at the start of running all the tests in the test case, there's a setup class hook. And then at the very end of running the tests, there's this teardown class hook. These names are not done in the idiomatic Python style of using underscores

or what's called snake case, but they're done in with uh Every other word but the first using a capital letter which is called camel case. This is a bit unidiomatic for Python and that's because the whole unit test module is a copy of the API provided in something called JUnit, which was one of the first unit testing frameworks in Java at the very dawn of unit testing, and it's been copied into Python and it's survived ever since. So that's why these names like sometimes stand out a bit in Python code. Let's look at an example of using these hooks. So here's a test case that we've written that's inheriting from unit test. test case and using all the hooks.

There aren't actually any tests here, but you can imagine them afterwards. At the top we have the setup class and teardown class methods and then below that the setup and teardown methods. This is the order I'd recommend writing them if you are writing them. because it makes it easy to jump to the test case, see what's general about it, from a class perspective, then a per test perspective, then you can go and read the tests. Let's look inside the methods. So in a setup class method, we're calling the superclasses setup class. If you're inheriting directly from unit test. test case, this isn't necessary because the default implementations for all of these hooks that we're looking at.

are completely empty and we don't need to call super in that case. But then if you do create your own base class and want to share that setup or TED and logic with other test cases, then you will need to call super. So that's why I've put it in this example. And here we can do something once at the start of the test case. So in this case, we're connecting to some ACME API. We can imagine it has this method of calling it and we're storing that object on the class. And equally when it comes to tearing down the class, we're making sure that we close that connection. before then calling any super method. So you can see we've kind of created a sandwich where our test case specific stuff is in the middle.

and then anything pre-existing happens before and after our stuff. We have a similar setup with the similar um setup with these setup and teardown hooks here. So for the setup hook we again call super to make sure we do anything from the base class and then we're going to be creating a user through that connection object. whatever API that might have. And then in teardown, we're making sure we delete that user from the external API before carrying on to do any other teardown logic. So this makes sure that each of our tests will have a fresh user object over on that ACME API. What we've run is not entirely robust and

if the setup fails for some reason then the tear down method is also going to crash. And that's because if setup crashes or raises an exception, then the teardown method is still called in full. And so if we imagine that the make user function crashes, then the self. user attribute would not be assigned So to write this robustly, we can't just call self. user. delete. We have to first check if it was actually created by using has Attra. As you can imagine, this might get annoying if you're doing a bunch of different setup and then tear down. And it's also perhaps a rare bug. that uh most of the time making a user will succeed but when it when it doesn't then you're going to see some quite odd exceptions.

Frankly there's an extended API in unit test for this And that is the add cleanup function and since Python 3. 8 the add class cleanup hook as well. So addCleanup takes a function that will be called later. It's in fact called after the teardown phase. And if you have multiple calls to add cleanup, the functions will be called in reverse order. which is the same kind of stacking that we saw we want when we uh calling super in setup versus teardown. So we get that kind of sandwich effect. And the class clean one is similar, that's called after tear down class. And so we can rewrite that first test case example we were looking at. with both the setup class and teardown class and combine it just into setup class.

Similarly for setup and teardown just combine into setup by using this. So here we go and create the connection again as before. And then only if this line is successful and class. con is going to be assigned, then we um we call add class cleanup to make sure that we go and close that connection. So if the line acme. connect crashes, then unit test can still tear down that test case. But it will only tear it, tear down uh those functions that got registered because their um setup succeeded. Similarly with setup we can call self. user dot delete and this method will be called later after the end of teardown to delete the user.

So to summarize, here's the lifecycle of a unit test test case from the perspective of these hooks. So the first thing that the test runner will do when it gets to our class is to run its setup class method. And then on a per test basis, we're going to run setup, we're going to actually run the test, and that might succeed or fail Regardless, the teardown will be called and then any functions that were registered with add cleanup will be called in reverse order that they were registered. There's no restriction, by the way, on when you add these cleanup functions. Sometimes it's useful to run add them during the test. In rare cases, you might even add them during the teardown.

Similarly, at the end of the test case, we're going to get to teardown class, and then once that's called, all of the add class cleanup functions that got registered will be cooled as well. All right, uh time for part two and Django's unit test extensions. Here is a simple class diagram of the extensions Django adds to UnitTest. So at the top of the hierarchy we have unit test. test case. Inheriting off that we have Django's simple test case class. And from that we have transaction test case. And then we also have the the vanilla test case class, which is the one Django recommends using most of the time.

And then also there's a live server test case class, which I won't be covering here. But this runs Django's web server alongside your test case so you can use a browser tool like Selenium. That's the only extension it adds and it's not relevant to the discussion, so we won't cover that any further. So this first simple test case class that Django defines, this has uh the basics of Django's extensions to our uh to unit test The first thing it actually does for us is it blocks database access. Because it doesn't provide any isolation, it uses some uh hooks in setup class and then removes them in teardown class

to block any database queries. So this makes it useful for testing anything that shouldn't be accessing the database like a pure validation function It also adds a bunch of assertion methods that are useful for testing things in Django, for example, form assertions and HTML assertions, etc. It tries to emulate what Unit Test uh does for you and it does all of the Django's own ProTest setup outside of the setup function. And so this means that when you inherit from it and you you can write your own setup and you don't need to call super as long as you're inheriting directly from Django's test case, Django's simple test case or indeed any of the other test case classes.

That's not the case for setup class though, so you will need to call super in that situation. The next class down in the hierarchy is transaction test case. And this does allow database access. That's its key addition to simple test case. It allows database access and ensures test isolation by rolling back the database. or indeed databases, because Django supports multiple database connections, by wiping all the databases and then re-adding any fixture data you've defined. So this is quite slow, as you might imagine. The more tables you have, the more tables it will have to go and wipe, even if the tables are empty. The main use case for transaction test case is if you want to test code that

goes and commits a transaction. Hence its maybe confusing name. If your test case has to commit and then you want to read the state of the database after that commit for assertions, you need transaction test case. Otherwise you can use the next test case class called test case. What this does is it takes a different strategy for database rollback. That's a lot faster, and that's to set up transactions. on a per class level using setup class and on a per test level at the same level as setup. And transaction tells the database to keep track of any changes made And then when you tell a transaction to roll back, the database knows exactly which pieces of data to undo the changes to

so that it doesn't need to go and touch all the tables and all the rows that you might have there So this is a lot faster. Inside the setup class hook, it calls the setup test data hook, which is what this talk is all about. And we'll see that in a second. There's also a lot more features in test case and I I advise you to go check the documentation I think here it's worth a note about PyTest. So I'm only talking about Django's test framework. You can go and use PyTest to test your Django applications. There are kind of two approaches you could take there. One would be to use Django 's test framework, all its test case classes here, and

PyTest has unit test integration, so that's all supported. Or you could go write functional tests. that um exercise your Django app just with single functions, which is uh kind of a more Py test Pythonic style. If you go and do the second option, you lose out on a lot of the development and work that Django provides for you in these custom test case classes. And indeed you can't use setup test data, which as we'll see is a key way of making your test faster. So for this reason, I recommend you use the best of both and you use Django 's test case classes. You can still use PyTests plain asserts. There's ways to get PyTest fixtures to work, etc. , rather than writing these functional tests which restrict you to only those things that exist in PyTest

or have been ported to PyTest Django. Okay, so we're going to now summarize Django's test case class lifecycle. So this is done on top of the unit test lifecycle you'll see. So at the setup class phase, Django begins one transaction per database that you have. Obviously most projects only have one database And then setup plus will also call the setup test data function if you've defined it in that test case. The default implementation is empty, so it doesn't set up any test data. And then on a per-test basis, Django's extensions are in a function called pre -setup.

So this avoids it being inside the actual setup method, so you don't need to call super On a pre-sell-up level, there's a second level of transactions begun per database. It will then run your test and then in the post-tear down hook, which similarly runs outside of the normal teardown, Jenko will roll back those per test transactions. Once all the tests in a test case are done, teardown class goes and rolls back the per class level transaction. So we have two places we could go and create data. We could create data during the kind of setup class phase, which is what setup test data is for. And you can see that this will get written once and then roll back only after the test case is done. Or we could do it inside the setup phase, in which case you'll see this will get written and then rolled back on a per-test basis.

So this is the key motivation for using setup test data is we can move from a per test to per test case rollback. Time to go into detail on setup test data. As I was covering before, this is a hook called by setup class inside Django's test case. That implies that it also has to be a class method. So did you have to make sure you decorate it as such? The default implementation is empty, so you don't need to call super. It's up to you to use it to go and create test data. Let's have a look at a typical example. Here's a test case that is using kind of the unit test style that often appears in tutorials, which is to only use the setup hook to create data.

Here we're creating some book object in our database. This test case class is Django. test. test case, not the unit test. test case one. And the book is saved onto the class instance with self. book. And then any test that needs that book can access it on self. book. If we even step write this with setup test data, it looks very similar. We have the setup test data method. It no longer says self, it says class, has a class method decorator because it's class method. We sort store the book instead on the class. And then during the tests, the test instances can still use self.

book to access it because Python's syntax allows um or Python semantics rather mean that any attribute that doesn't exist on the instance is looked up then on the class. So our tests wouldn't need to change if we were changing like this. So to compare, as we're seeing in the diagram, anything you do in setup will be done n times where n is the number of tests in the test case. Whereas anything we do in setup test data is done once for that test case. And it's this n to one conversion or uh or difference that means that our tests can be more if we use setup test data.

All right, time to look at part four and a key change to setup test data that came in Django 3. 2 around how it isolates the data that you create there. Let's take these example tests. Using Django's test case again, we're creating a book, much like the last example we were seeing. The book is created in setup test data with the title Meditations. And if we remember how Django's RM works, this will go and create a book instance in Python and then it will also call the save method of that book to go and insert it into the database. So after setup test data, we actually have some data in the database, which will be rolled back

by transactions, and we have some data in memory. So the first test here is just changing the title of the book, but it's not saving it to the database. So it's simply accessing that book through self. book and then modifying its title attribute to set it to a different title anti -fragile. The second test here fetches the book out of the database without touching the self-bot book probably. It's simply calling book objects get, which as long as there's one row will fetch that row. And then it's getting the title field off of that and asserting that it's meditations And the third test is asserting that the in-memory title is meditations.

So when we look at say the setup test data and any test in isolation. We'd expect that to pass because this book is created in memory and in the database with the title Meditations. The first test has no assertions, the second test asserts that the title's meditations, and the third test asserts the title's meditations. So they should all be equal. Unfortunately, on Django before 3. 2, that third test there fails. So when it try when Django tries to read, when the test tries to read the in-memory title, it will actually find that it is modified by the first test to anti -fragile. And it has not been

and that change has not been rolled back after this test. It's stuck as anti-fragile, so this assertion fails. Whoops. How does this happen? All the database changes are rolled back by the transactions that Django sets up, but the in-memory changes to those objects are not getting rolled back. Django until 3. 2 had a whole block of documentation under setup test data about this problem. And the recommendation was that in your tests you always re-query the database. So you don't assign anything to the class in setup test data and instead you write some code in your tests

or perhaps in your setup. to query back that data from the database. So you always have a fresh in-memory copy of the object. This is slow because whilst we've reduce the saving or the insert queries from n to one by moving them into setup test data. We've then added back like n select queries like one per test to go fetch those objects. And it's also more verbose. You want the move from setup to setup test data to be a minimal change, but instead it just requires us to write extra code, which is annoying. It's also unclear. Like if you come across a test and you might think that it works one way and you

instead have to go read these giant paragraphs of text in the docs to understand the problem. Thankfully, a package was created to solve this problem called Django test data, and this automatically copies objects created in setup test data on a per-test basis To use it, we simply import the decorator out of the package module, and that's this decorator wrap test data, and we make sure we apply it to the class method as we create it. And this decorator will inspect the class before and after any objects being assigned to it in the actual setup test data method. and it will keep a track of those objects and then when a test accesses them through the same self.

book syntax, it won't just return that book, it will create a copy specific to that test instance. So the test will access its own copy and the original copy created in several test data will remain intact to be copied into each test in the future. So the state created in several test data remains available to every test. without being polluted by each other. This package was merged into Django in 3. 2. Thanks very much to Simon Charette for creating the package and contributing it. And therefore, we can write our tests in that same style as before on Django 3. 2, no need to use the decorator, and we end up with that per

test isolation. So If we ran these tests again on Django 3. 2, they would all pass. So to summarize, we're going to look at how we can convert a test case that's using setup to use setup test data and use that uh technique to speed up test suites. Um a typical uh level of speedup you would see here is perhaps two to three times, but it really depends upon the number of tests in the test case and the amount of data. It's not unreasonable to see some test cases speed up by a factor of ten also where most of their time is spent creating data and the functions being run on the data are relatively short. This technique takes only four steps.

There's a fifth one maybe if you're on older Django. And I wrote a blog post on this called How to Convert a Test Case Prime Setup to Setup Test Data. You can find this through Googling or I'll try to paste it on the chat. right now. You can repeat this technique across your code base for these great gains. So if it can speed up an individual test case two or three times then if you apply it across your entire code base then you'll achieve that speed up massively. Here are our example tests that will be converting in this section We've got some tests of an index view or something we can imagine. We've got this typical style from

unit test tutorials using setup to create the data. We've got two ORM objects being created, a book and a user. And these are being assigned to self. And then the third step in setup is using Django's test client to log in as the given user. And then we can imagine what the tests are below. I've just left them as ellipses because the content of the tests doesn't need to change when we do this. So the zero step is to go and install the Django test data package, as we were just looking at. It's uh just a pip install away, uh but you should go check the instructions uh on how to import it correctly

Okay, so the first step uh which is what we should do anytime we're refactoring tests is to go and run uh that test case. Here I'm using Django's test runner rather than pyTest, so that's managed. py test. I'm using its keep db flag to avoid rerunning migrations. um when I rerun the tests later on, so I can save a lot of time that way on projects with a large migration history And then I pass the path to the test case. And this looks a little bit different on PyTest. So you should go check the docs there. We see that the tests run and we make sure we get green checks for all of the tests and the result is okay. So we know the tests are already in a working state and we're just going to refactor them and hopefully end up again in a working state.

The second step is to go and add a stub setup test data method to our class. So that involves using the class method decorator writing out this line. If you're converting many test cases, you might have this ready to copy and paste. I've deliberately not added a body to this method yet because we will be moving some stuff. into here. So we don't need to have the code in a runnable state right now. If you're on an older version of Django, as we were discussing, you'll need to use wrap test data. So that means you'll need to insert the import of wrap test data in your module and use it on the setup test data method. The third step is to move all of the data creation from setup into setup

test data. That would mean anything using the ORM. So here we've got the book and user objects being moved. It could also mean other data creation, perhaps if you're creating something like a pandas data frame or a large dictionary of data. That might also make more sense to create once and copy on a test basis rather than go and create. And setup test data can help with that isolation too. Note that we do not move the client login line here. Django's test client is on a per test basis rather than a per test case. There's no copying or isolation done. between the class level and the setup level for this.

So we have to leave this in setup. So our test case will be left with both a setup test data and a setup method. Here's the full view of what it looks like afterwards. So you don't need to just read the diff view. We can see yeah, setup test data is creating ORM objects and setup is just using the client to log in. And the tests have not been touched. And the fourth step, well, that's to go and rerun the tests and see the time saving that we've achieved. Here I have tests that don't actually do anything. So it's my and it's a bit unrealistic and it's also a bit a bit of a narrow use case, but we can already see we saved three milliseconds or 20%

of the runtime by moving those two RM creations to be done once rather than twice. And we get two green check marks. If after this conversion you don't get any green check marks, it's probably best to like take a step back and move some stuff. out of setup test data back into the setup method. So you might need to rewrite CLS to self a few times for example, and you might find some problematic object that underneath is doing something that can't be done once but needs to be done multiple times. Perhaps there's a function call you didn't spot. After this conversion, you can sit back and enjoy your end-to-one performance kit

on these tests. All right, I think that summarizes my talk. Thank you very much for listening to me. I have been and still remain Adam Johnson. You can find me on GitHub and Twitter as AdamChains. My email address if you want to reach out is me at adamj. eu. Adamj. eu is also my website where the blog posts and links to my book are And the slides for this talk are over on GitHub available on HTML and PDF. And I will try and post some of these links in the chat. And I look forward to hearing your questions. Thank you very much.