A minimal Django testing styleguide

You hear me? Yeah,

Speaker 1: thank you. All right. Hello Joseph.

Speaker 2: Hello.

Speaker 1: Welcome on stage. Thank you. It's not the first time.

Speaker 2: Yeah, it's the second time. So shall I take it on from here?

Speaker 1: Yeah, you you absolutely can. Uh do you want any more introduction?

Speaker 2: So, good morning everyone. I'm happy to be here and happy to talk about having a minimal Django testing style guide. So, this talk is probably formed from Years of experience and uh from my perspective because now I'm getting more into seniorish roles and so whenever I join a new project or a new company I am uh joining as someone who is coming to an existing code base. And when we talk about existing code bases, the larger the code base and the more the people have worked on it, then it's kind it kind of gets scarier to join that. code base um and uh uh

Speaker 2: what what is it that troubles us what makes us anxious about it one thing for example that job adverts make about some jobs is That you're joining to work on a Greenfield project. So why is it that working on an existing project is scarier? And the the image here is of broken windows. And uh in 1982 there was a criminology study uh that uh produced this uh broken windows theory. The theory says that if one broken window in a neighborhood leads to more degradation in that neighborhood, and the reasons they come up with are two. The first is signs of neglect lead to more neglect. And the second is that the people in that neighborhood sense a lack of ownership.

Speaker 2: So I think this can be applied to our work environments and therefore our code bases. You might say that it's just a broken window. It's not, it does not affect the structural integrity of the building, but it's the small things that then lead to degradation. And one thing which might appear small is our unit, is our tests. test code and test code basis. It might not be the production code itself, but our uh test code is one of those small things that lead that if neglected lead to degradation. And my answer today to that is to have a style guide in place. So a style guide has rules and guidelines. Guidelines are recommendations or best practices, whereas rules are laws people are meant to follow.

Speaker 2: So the style in style guide can be a bit of a misnomer because the style guide is not only about code formatting practices, about code formatting rules and enforcing them via automation. We have Marica 's talk right after this one. The rules and guidelines in a style guide, for example, do's and don'ts about what to do in your test code require more human judgment and more engineered judgment. So if we have a style guide in place, what do we gain and what do we lose? We lose the individual engineers lose flexibility because their choices are restricted. However, we gain consistency.

Speaker 2: And consistency enables engineers to focus on what to test rather than how to test it. We also consistency also results in expert chunking when we have a consistency. Consistent way of structuring tests. Expert chunking. Chunking is a cognitive process which groups pieces of of information into meaningful chunks rather than keeping track of each piece on its own. This makes it much faster to think about things. For example, expert chess players. think in terms of configurations of pieces rather than uh keeping note of each individual piece's position. This makes it faster to read tests and this results in productivity.

Speaker 2: Consistency also brings productivity because you have engineers who jump from one code base to another and hit the ground runner, uh ground running. And also our codebase is more resilient to time because if the people or players on that on that code base change, the codebase itself doesn't need to change because of the consequence. change in ownership. Finally, uh with this style guide, we have reduced conflict, especially at code review stage, because the engineers can focus On what is being delivered rather than how it was coded. So, what should our style guide be about? It should tell us what to test and how to write these tests.

Speaker 2: So what should we test? And the answer is the Beyonce rule. Test Test anything that you don't want broken. What does this translate to in in our Django applications? We have to test anything that is not already tested by Gort by Core Django or by the tests in the third-party applications, third-party packages our Django project uses. All right, what what how how can we enforce that? How can we ensure that we are testing that? And there are multiple Ways our projects, our code can be tested. So let's get some definitions done here.

Speaker 2: I'm by no means this is a disclaimer, I'm by by no means the authority on this. So, but I will spare the in my opinion in my opinion prefix to sentences. So a unit test, we can say that the unit test so if we're testing a function, that is the test target So it's the target under test. So unit test involves testing that target, and while the unit test runs, the target does not talk to any other component. Integration test, for example, in Django, when we're using the the test client, it's an integration test. Why? Because if we hit the URL and To test form validation, we're hitting the URL, the view that is bound, that the form we're testing is bound to, and the form code itself. So it's an integration test because it it

Speaker 2: hits multiple components. On the other hand, end to end tests or system tests test the whole, may talk to components you have no control over. So for the purposes of this talk We will treat unit and integration tests as deterministic. Deterministic in this case means that their behavior is repeatable. Whereas system tests, because they talk to external systems , they are non-deterministic So this the first pyramid is my constant pyramid. It's from the book Software Engineering at Google. I will give references later on. And they advocated Google to write a huge amount of unit tests compared to integration tests and end-to-end tests. Why? Because as Martin Followers, the spyramid

Speaker 2: shows unit tests run faster. You also develop them faster because they are they they have a narrow focus. They are smaller. And since they run faster, they are easy to run on your own machine, they discover bugs earlier. That's why the cost of both running them and them finding bugs is Is uh smaller. On the other hand, the closer you move to the user, the slower the test become, and the costlier it is to discover and fix bugs because you have to go back. There are some other there are also some anti -patterns here, these two and the from the same book software engineering at Google. The ice cream pattern also has manual tests up here, which is very bad because you have to have a human actually doing tests.

Speaker 2: So that is not scalable. And the hourglass pattern here has unit tests and then two intestines, very little integration tests. This is bad because it doesn't catch bugs that could have been caught at earlier stages And as we said before, catching them at the last stage is the most expensive way to do it. There's also this pattern, it's an anti-pattern, but it's very it's advocated a lot in client-side applications and uh uh user focused applications. Uh for it's by canc dots of React. js frame and uh It it uh encourages a lot of integration tests. So my point about this is there's no one-side

Speaker 2: fits all and your team has to agree on how to how to go about uh what tests to write. The uh the other uh point that I want to make is that unit tests are uh provide the highest ROI return on investment because they are uh faster to run, um probably faster to write. And give the feedback loop is very short with unit tests. However, their main drawback is they don't test component interactions So, what do I end up doing? So, without looking at the death pyramids, I realized that myself I tend to write more unit tests or more unitish tests when it comes to uh code that talks to the database such as code and models

Speaker 2: such as a query set function for example. in modulus. py. Whereas where the closer I get to the user or to the client API, the more my tests start looking like integration tests. As I said before, if I'm uh testing the code in a Django REST framework serializer, I sometimes just use it as client, and that uh hits the view, uh hits the URL server, and then the serializer itself. So that, so we have some definitions done. So let's see how to write the tests. And I would like to go before going through the do's and not's of how to write tests. Through what goes in our mind when writing the test, when thinking about the test.

Speaker 2: And this is uh the this diagram is from a book published in 2000 with research papers it refers to from the 80s and the 90s. About code complexity. And here we can see that this is the flow, the logical flow. For a sequence and whenever whenever you had an if then you have a new logical path. So white box testing is about exercises or exercising all possible paths. that your code uh can go through. And uh it is a very exhaustive uh technique uh And m we we'll see um that if if you have an if you have an extra part, if you have a a Boolean operator, you have an extra part. So uh

Speaker 2: this is from Ray Don, which is a package that uh uh analyzes. Cyclomatic complexity, also known as McCabe's comp complexity function. So these are are about executing all possible logical paths in your test. So let's get a very small function here. We have a customer, email, first name and last name. If the email is set, return the email. If the email is unset, try to return uh both try to return the name if both first name and last name are set, otherwise return not available. So if we try to use white box text testing, we would have eight tests for this simple function. We would need eight test cases.

Speaker 2: So with all the states. So black box testing comes to the rescue and with a technique called equivalence partitioning we decide to narrow down the tests we need just to three. One and here the focus rather on the than on the DNR workings of the function, we focus on the behavior expected of the function. Function. So we just write three tests and test the behavior that way. So We have gone through what tests we should write, what goes in our mind where when thinking about tests. So let's go through the main cores of your of the style guide, which are the do's and don'ts. So when you have tests you have to agree

Speaker 2: about the structure of those tests. So if you have a piece of code and and a module you won't you don't want to think and try to figure out where that where the test for that code is located. So ideally there is a predictable way where your tests are your are located for your code. So a common style is to have tests reflect the structure of the code. So this is my Django app and I have a forms directory with an orders and customers. module in it. So I have test forms and test customers and test order. So anything in customers you can find it here. Anything for models you find in test models. And so on and so forth. There is no best or worst way to do this. Just have one and stick to it so that people don't do not guest lost finding tests.

Speaker 2: So that's one thing about the structure. The next, uh that's one thing about the forzer structure. Now uh we have to agree about the structure within the test itself. And the pattern, which is very useful, uh and leads to the expert chunking I mentioned earlier, is given when then. Uh so a given when then uh give structure to the tests. So we put the given uh code here when And then so we separate the tests into steps. An alternative is called arrange act assert. So in this section here we are arranging the setup data for our function. Here we are calling the target and here we are making our assertions. The more tests you write and you keep a structure is very

Speaker 2: it makes it very easy to when you come to look at a test, uh you know with the combination of comments and white space what each part of the test does. So these since these are docs inline documentation, maybe having a more human-readable string would have been better than when created today is called. Here I'm uh there this this comment here leads me to the next slide, which says use verbose test and function name. So here I have the serializer for the customer model, and the serializer is validating, is validating that when name information is provided, both first name and last name must be submitted.

Speaker 2: So how do we write tests for that? We test the behavior. And it's very useful to get a glimpse, an immediate glimpse of what the test is doing to uh to write about the behavior, the expected behavior of the test um within the test within the function signature. So uh do not be afraid of having long test function names. No one will be calling these. So there is no uh the more information information you provide the faster it is to read the test later on. Also do use test case and test naming conventions. So customer serializer results in customer serializer tests. and validate results in test validate, test underscore validate, and then

Speaker 2: the state or behavior description that you're doing. There is no best way to do this, just agree on one and And follow through with that. This is very important. A test should test only one behavior. If you try to have a test testing more than one behavior, it's confusing, it's unmaintainable. The test itself might be flawed. Um, so uh keep it simple simple it uh uh with when it comes to unit test uh to test code. Then don't worry about having code duplicated. And that brings me to the next slide. Don't try to dry. Dry is don't repeat yourself. When it comes to tests, repetition is not a bad thing.

Speaker 2: Actually, if it makes your tests more readable, more independent of each other, then uh duplication is not bad. Remember, do not put logic and test. It's another point of this because sometimes when we try to test more than one thing in the same test or avoid copy-pasting things, what we do is we stick an if or a for loop in our test And if we do that, it means that our test now needs another test for the test, which uh so no logic and copy paste is good. Another principle that I uh like to have followed in tests and I uh I I do this the mistake myself, is locality of behavior. This is from uh this is an essay from D HTMAXAuter.

Speaker 2: And uh locality of behavior means if you look at a file or if I look at a function I know immediately what that function does just by looking at that function's code. Now, of course, it uses it extends test case, so there are a lot of things. lot of built-ins but if I'm familiar with Django's test case then uh if I look at the function if I look at a given when then structure I know exactly uh with I don't need to think that much uh to see what the code is doing Doing. So locality of behavior, try to locate everything your test is doing within that test. And this something that I have a mistake that I have done myself is to have test data classes With uh setup test data class method. Uh

Speaker 2: if if you're unfamiliar, uh we have um the test the test case class in Django offers either setup which is run for every test or setup test data which is runs once per the test uh per the test case What I have done, a mistake I have done in the past, is having a hierarchy of state classes, each calling the parent, and then if I need to see what state is being created for my test, I need to traverse. manually um through the hierarchy to see what what the state of the data is for my test. So that is a mistake and uh Ideally, your test at the cost of duplication should have everything within the test itself

Speaker 2: and avoid test case hierarchies as much as possible because then you it makes it hard to see what your test is actually the doing. Um this brings me to test data setup. So please do not use fixtures for a test. Fixtures are very easy to create, managed by dump data, you have a JSON file, you can use it in tests. Django makes it very easy to use fixtures, but they are not intended to be used that way. So you have a JSON file. Let's say you are then you filled. What do you do? How do you go about it? Also, fixtures tend to be append only. So if I want to create new test data, I just append to the to the existing fixture.

Speaker 2: And then all my tests use that new state that that fixture uh is loading. And also fixtures are versal, but that's another topic. Um so compare that to using factories instead. So in the Django space we have factory boy and uh model bakery. Which are two very convenient packages to use for setting up data and tests. So here I edited the customer model we had earlier to have a code. And the code here is not null. And therefore I configured the customer factory I had earlier to have this new code column here and whenever I'm not providing it via the factory what uh factory boy does it sets it to have this default value with cast hyphen

Speaker 2: and the sequential generated number. So that's all that I needed to do to have the previous tests passing. In addition, I changed the previous function so that instead of not available it returns the code since it's not Uh nullable and the only test that I needed to change was this. I changed it from not available to return scope to return code, and I'm passing the code so it's hard coded, and I don't need To think about, I don't need to assert again the customer code, I can see what's happening here. So you can see that factories make test data management much easier. And also the point about these given when then strings is that you have to maintain them.

Speaker 2: So this is a mistake I do continuously. I always forget to maintain them. So that uh consider using them only for the most important or most complicated tests. Another two don'ts here. Here I'm using so don't use numbered variables. Here I'm just calling them order 2, order 3, and instead I could have called them order created yesterday at midnight or past midnight. Having the assert queries at equal using that would have been much easier, would would have been more humor readable. In addition I'm avoid having the tests allowing your test to talk to the system clock because it makes them non-deterministic because if they depend If the time, if

Speaker 2: if the system clock is not frozen, your test behavior will vary depending on when they are run. Use time machine or a package like time machine or freeze gun to freeze your daytimes Use clear Ferriel messages. We can see the difference when you use assert in instead of assert through the error message here is more helpful than the error message here. And Django itself provides a lot of assertions. This is from simple test case. There is a whole hierarchy, and I default to use the last one in the hierarchy myself. Because it provides all the goodies provided in the classes it extends. This is very important and probably is why

Speaker 2: I end up using more tests that look more like integration tests whenever I'm dealing with uh URLs and whatever the outside uh world sees of my Django application. So When uh here I'm we have a view, a list view, and it deterns the items for the user. So here I'm creating a couple of users and then I'm creating two items for for the user and then I there and then another item for the another user. I'm assigning the user to the request object, the request I'm instantiating it, and then sticking the request to the view, because since I'm not using the client, I have to do manually what the authentication middleware usually does. So

Speaker 2: then I can test the context. So compare this when using the client. You just for the the user you assign it to the client session. and the code is much simpler. In addition, you can see that with less code I get more return on it on my test effort and investment because I am testing the mapping of the URL to the view and to the con to the buildup of the context. So ideally, test behavior not a not internal logic or state as we have here. A final uh my last two slides. One is about mocking and the other one is about uh coverage. So whenever you mock, you replace the component uh

Speaker 2: the component in your target with some other component with a mock. The more that changes the behavior of the target, and the more you you change the behavior of the target, the less accurate your test is. So mock carefully or sparingly, and ideally use. For example, instead of creating requests, use request factory when when when instantiating requests during tests, and otherwise use autospec. So uh what If if if you don't use uh autospec, what mag what mock does it creates whenever you access an attribute on the mocked object, mock returns another magic mock. So The idea to use Autospec

Speaker 2: is to limit your code, to limit the possible errors to make the target as accurate as possible. Uh autospec has its limitations because it only provides the attributes defined on the class. For example, if an attribute is instant is is instantiated in the constructor that is not available for autospec. So again, this calls for the engineered judgment. Do use a convention for where to apply mocks. So I prefer to apply the curators so that I avoid the width statement context measure. In fact, I like my the code in my tests to be To have no lines of indentation at all, so though so that I need to think less. And also do is a convention for naming mocks. So if you're using the decorator, make sure it's

Speaker 2: you use a mock underscore prefix or a p underscore prefix or no prefix at all perhaps. And if you use you decide to use a package to to uh as an add-on to mock itself uh mock is provided by by python standard library like request mock make sure you use it across the code base not sometimes yes and sometimes no Final word about coverage, do play the tame the game of test coverage, and coverage it on itself is a very dumb metric. Because we could go from zero to 80 % coverage just by executing the test client code by running the test client against some URLs, and so that we have a lot of coverage. um that itself doesn't

Speaker 2: doesn't uh doesn't improve the correctness or there isn't the correctness of of the system. We're we're we're testing So what should we cover? And beyond Cerus apply, we should cover anything that we don't want broken. And if we want that gradual uh improvement towards coverage over time. The way to enforce this is to ensure that no PR reduces coverage. So if we so anytime you open a PR, we don't want we want a non-negative coverage number. So that was quite a drive through the do's and don'ts. These are the books I refer to mostly, but I will provide a whole list of references with the

Speaker 2: talk link. These are more theoretical in nature, these are more practical. This one, while published in the two scopes of Django that I refer to, while published in 2013, is still very much Up to date, even though ten years have passed, which shows how mature Django is as a framework. And this one by Adam Johnson is uh is more about execution of tests, but also has uh uh uh some chapters which are very good about which provide very good pointers about how to write tests. So the takeaways so the style guide So the style guide is a work in progress on which the on which team consensus is required. It shaped, it is no server bullet, so it shape it shapes behavior over time.

Speaker 2: And having a style guide, even if the style you might you agree on is not uh that good to start with, it makes it easier to migrate to a better style going forward. So definitely I hope I convinced you about it. Regarding what tests and how to write them, I hope I convinced you that joining a project which has been there for 10 years It's better to join a 10-year-old code base with a predictable and consistent style than a 10-month free-for-all project. And my final note is that code is read far more than it is written, so make sure you write the tests you want to read.

Speaker 2: That's it.

Speaker 1: Really good talk. Thank you. We are ready for questions. I guess we have room for a couple of questions. Yes,

Speaker 2: yes, go ahead.

Speaker 1: Anyone in the Audience? Yes.

Speaker 3: Um hi. Um great talk. Um one question. You had this one test case shown where you would say um Um I'm asserting the number or the name of the customer, like test customer, and then you setting it in the um factory boy and then you asserting it. And I'm always struggling to put this in a variable or not because you know it's inside this test and it's Duplicate and it feels wrong, but I usually never do it. So what's your take on this?

Speaker 2: Uh I I think uh I I wouldn't put it in a variable because uh I'm not afraid to repeat things. Uh I think having it there, it's easier to see if you if there's a mistake it's e in the unit test it's easier to spot. Um yeah uh I but I did use variables and now I don't do it anymore So it's i it's it's not a clean again uh I'm not the authority on this. I it's a simple it's a minimal style guide with style guide contents that might go into your style guide but you and your team decide. But just decide on one and stick to it. Thanks.

Speaker 1: I I have a question. Uh would you consider naming the style? guide. I'm thinking about maintaining projects and and then there is a contribution race someone fixed a bug or added a feature but they didn't write text. or the test doesn't uh like follow a style guide. Um can make it easier perhaps to refer to it.

Speaker 2: Um so in in my talks references I I put a couple of style links to a couple of style guides. Um one of them is Bioctopus Energy, and it's very exhaustive. Uh so uh if I understood your question correctly, uh is is it about how to install a style guide in how to uh you don't want to enforce a style guide on on anyone, kind of. So I I 'm not sure I I understood your question

Speaker 1: Yeah, so kind of like the contributing guidelines for the uh random open source project and I want to uh say can you follow something like this?

Speaker 2: Yeah, ideally, but uh it's it's a long it's a long presentation, so it would be even longer to write it and provide samples. But ideally, yes, uh do's and don'ts, the format is quite uh helpful at what if you do something you're ah I'm doing this so I shouldn't I shouldn't do it. I sh I should do the alternative. So the do and do the do and don't format is quite helpful in that case. I'm not sure about the contribu whether the Django itself contributing to Django docs have a style on tests themselves. As uh Sarah said earlier, it's it's a huge project with a lot with a huge number of tests. So this my talk is more affected by closed uh projects for businesses.

Speaker 2: That's uh that's where my background is. But yes, I think a style guide, if anything, it helps people improve their uh coding practices. So yeah, it's a good idea.

Speaker 1: Do keep a style guide.

Speaker 2: Yes, definitely. Was the another question.

Speaker 4: Yes, hi. Um you talked about not using fixtures and tests and you provided the example what happens if there's like a change to it, migration or something. Um sometimes you're forced to have fixtures just because it's like a legacy app and that's like a whole complicated thing that you need in the database. to run the uh to run the tests. Um what I've done in those cases is I've added like a part to the readme of like how do I upgrade our test database? Which is basically like delete your SQL Lite database, uh run the migrations up to the point where the where the uh fixture was created the last time. Load the fixture into your local SQL uh light database, run all the new migrations, and then

Speaker 4: r run uh like unload it back into a fixture and put it back in in in the repository. Um but I don't know if like is there a better way of doing this or is like a standardized way of of handling this Because I mean this is just what I came up with myself.

Speaker 2: Um I I how do you how do you manage the transitions when the schema changes? How do you regenerate the the that that's the biggest headache uh that I had when using fixtures

Speaker 4: No, then I run then that's why I I delete my uh my developer SQLite database. uh load the uh the fixture into that and then run all the migrations that have uh happened since the last time. So now it should be up to date and then I like uh dump the the contents back into into the fixture and then I upload the the fixture to the repository and now it should be up to date. And of course this works until next time there's a change and then the same procedure has to

Speaker 2: Yeah, um so the the idea here is that uh the so the fixture makes it very hard to read in my opinion what's in it. So if you use factories, you have if if you use something like factory boy, you have an initial effort where you have to uh model all your data using factories. So you configure something factories with subfactories, for example, to create related uh objects. But the the the migration from fixtures to factories would involve an initial effort where you have the same data created by the factory code. Then that would make it much easier when you have schema changes, for example, because with Factory Boy you either provide a default or not do anything and it it just

Speaker 2: works without any effort. But there is an initial effort to migrate from the fixture to the to use to use factories. And if it's a lot of data, factory function factory functions create uh uh give you provide functions to create batches of data so yeah it it is a one-time effort but I think it pays off going forward

Speaker 1: Quick question.

Speaker 5: Yeah, quick question. Uh thank you so much for the talk, Joseph. Um I uh wanted to ask about where um if you have any more tips around where to put tests. I think sometimes it can be really nice and easy. You know, you have a f a test file already that tests a specific form. You're adding some functionality to that. You know exactly where the test needs to go. But what if it's like less clear? Do you have ideas about like should it, you know, one code file map to one test file? Or yeah. Any more points around um this problem of okay, I've written a new feature now where the hell do I put my test?

Speaker 2: So yeah, so uh yes, it's it's it's a very uh it's it's a very good question because this challenge I encountered it several several times. So uh again, we go back to the definitions. What are we testing? Is it a unit test or an integration test? So maybe unit tests can have this structure where they are located in you have a structure which reflects that of the non-test code, of the production code. But for integration tests, you might have an additional directory that is uh structured around the behavior, the name of the behavior you're testing. So We had a customer, so for example, if we had a customer lifecycle in which we want to execute multiple steps that are that talk to a lot of different uh modules, then we would maybe create an integration uh

Speaker 2: tests directory directory and we call the modules by the behavior they they test rather than by the location of the code. So yeah, that is again that is something the team has to discuss and uh everything everyone should be on board with it because there is no one size fits all for this kind of of question, but it's it's good to separate between unit and that's maybe why we should separate between t between unit and integration test, maybe to locate it and style it differently, the the code that results. Thanks.

Speaker 1: I'm afraid we don't have more time for questions. Um, but Joseph. As maybe you noticed in uh Joseph's slides, then uh there was a reference to the next speaker to do that supporting the other.