Evolution of a Viewpoint – Django Day Copenhagen 2020

Speaker 1: Okay.

Speaker 2: All right. I think we're ready to start then now. Welcome.

Speaker 1: Okay, thank you. I'm sorry, I think I've had some internection issues. Let me start with sharing my screen. Okay. There we go. Um well it's um coach we review as um I've said already. Um briefly introduce myself. My name's Luke Plant. I'm amongst other things, I'm a freelance software developer. I'm also a core Django developer, whatever that means these days. I'm not actually active in Django development itself. um but more widely in the community. And I'm not sure what Ben, I didn't catch everything that Ben

Speaker 1: said just now, but um The background to this talk is a few months ago I published Django Views the Right Way, slightly opinionated title for an opinionated guide that's essentially about when you're writing view codes I'm not talking about the model or the template, but when you're writing the view code in Django, how you can do everything you need with functions. instead of with class-based views and how I think very often, especially with kind of server-side rendered things, that you will be um Better off using functions than class-based views. Now I covered quite a lot of broader topics in there, broader principles of object-oriented programming, which I can't begin to summarize.

Speaker 1: Right now. What this talk is about is is how I came to that position, how my views on views evolved. um you could put it. So there's three parts of what I I want to say. Firstly the um historical journey I went on Second, a discussion of what I think the biggest issues are. And then third, an example of how those issues that might work themselves out in the context. of a single view, how a single view might evolve over time. So first of all, the history I my journey with Django started with a charity called Christian Camps in Wales. It's a small charity I've been involved in literally my whole life.

Speaker 1: I've worked on the camps as a young child with my father and then I was a webmaster in 2005. We already had a website, but I then discovered both Python and Django and had a great time rewriting the CCRW websites in Django. As um as historical interest, this is what my first ever view function looked like. This is September 2005. Now before you Judge anything about this code? This was a this is when I was a very new Python programmer and Django was actually well before version one at this point. So the some of the ORM code here looks a bit funky. You might not recognize that. There's a lot of funky stuff going on that's best forgotten. But I think you should still be able to recognize the view code.

Speaker 1: We've got a function. It takes a request and returns a response and it uses the template engine to render it. Now these days there's shortcuts with these kind of combinations, but otherwise it's it's not too different. This project has some other claims of aim, by the way. A number of features in Django actually started out live in the CCRW code base, such as CSRF protection mechanism and a password reset functionality. These were later moved into Django Core at different points because around that time I became a Django Core developer. As this is a side project with no other developers, I do have a bit of freedom to experiment a bit and try out different approaches.

Speaker 1: So in 2011, Django 1. 3 was released with class-based views. So in the CCIWW website, I rewrote a number of things, not everything, a number of things using CVVs. and started adding new features using CBVs. And it worked out pretty good at first. I found some neat things. For example, I wanted to add client-side validation to a form. And the nicest way was to reuse server-side validation via Ajax. Not quite as fast as purely client-side, but it reuses server-side logic and it's fast enough. So I had a page that was already using the CBV form view. All I had to do was create a mix-in that used the same form defined on on the view

Speaker 1: but responded to ajax requests with just validation errors. So it looks something like this. Maybe you've written something like this before. We can just um look at a query parameter and then we can return um uh form errors as a chunk of JSON and and these were wired in client side um with some generic JavaScript code. So adding Ajax validation to a form was now just a case of adding a mix in to the view. Amazing. Until I found that later on, when I tried to combine it with some other mixings, it didn't work. I think it was a model form for editing an object or something. And there were this really complicated nest of calls to super that's no matter what I did, what order I put the mix-ins

Speaker 1: in, methods got called in the wrong order. But I battled through on through the jungle and eventually I found a solution. Metaclasses. Did you know that if you create a metaclass You can override a method called MRO, method resolution order, and you can dynamically change the order in which your classes appear in the chain of superchorts Wow. This is what my solution looked like. Fantastic? No, I don't think so. Let me just add that to be perfectly clear. Don't do this. And please don't ask me to explain it either. I'm not sure I understand. We mess around with the order of classes and things.

Speaker 1: To what shall I compare this code? In the UK, there's a guy known as Bear Grills who does extreme outdoor adventures. I'm not sure how famous he is outside the UK. He starred in a few internet memes. On a number of occasions, he's run out of drinking water and his solution was to drink his own urine. Which apparently is something you can do as long as you don't do it too many times in a row or you'll poison yourself Well, I can see how that is a solution and probably better than getting seriously dehydrated But if you find yourself doing that, especially if it's more than once, surely you should be reconsidering some of your life choices, right?

Speaker 1: How have I ended up doing this? Because that may be macho, but it is not pleasant. That's how I feel about this code here. Now My metaclass solution might seem extreme, but actually lots of other people are having the same problems Documentation sites like C C B V Chassit Class Based Views. This is a great resource, but it exists to help people navigate this maze of of base classes uh and method calls and there's um diagrams which are map through this maze as well. And recently the official Django documentation has something similar. You've got the MRO and the method flowcharts

Speaker 1: and so on. And and I found this in other projects too. Work for clients, various other places where I came across CBVs, a whole lot of complexity. And I almost always found if I rewrote a bit of CBV code as a function. often to understand it, it not only got simpler to understand, it also got shorter too. But I didn't give up on CVVs just yet. I took another approach, which was to write all my own base classes instead of inheriting from Django 's. Many of the problems I had stemmed from not being in control of the class hierarchy and also other functionality which got in the way. was unnecessary, was it robust?

Speaker 1: So this was a big improvement in my view. And I wrote a blog post about that approach, which I still don't think is a bad solution. My Ajax MRO fixer was killed off, thankfully. But I still kept seeing problems with CD bits. At the same time I was growing as a developer in general. and with a whole range of experiences in different projects. Earlier this year, the last straw came when I read a tweet by someone saying that they were actually scared of using function-based views. They only knew class-based views and they were struggling with others. So then I started writing that guide, Django Views, the Right World. Part way three though, I thought I should go back and practice what I preach. I should

Speaker 1: go to the CCRW website and see what happens if I get rid of those CBVs entirely. So when doing this, I was now armed with a lot more experience in Python programming techniques and had a lot more options in my mind for how to do things. And I found firstly that switching to functions made them simpler to understand for me. It also made my lint up to just play gates um a static analysis tool made them able to fully understand them because instance variables became local variables. So it was catching every single error that I wrote. Unused variables, undefined name. And when I I remember one bit of code I finished fixing all the Linda errors. My

Speaker 1: my code was vote-free and passed all the tests first time. And then I also found that my code was actually shorter, which was not actually the most important thing, but an interesting fact. So after nearly ten years of wandering through the CBV jungle, I emerged. Having written some unpleasant code and regretted some live choices And where am I? Well exactly where I started. The CC codebase now has entirely function-based views again, and I've gone in a big circle, at least with that project But maybe I'm not exactly where I was. Hopefully I've uh learned some things. So the question then is where where did I go wrong?

Speaker 1: I think there's various issues, but I think the biggest single issue is that we, as the Django community, went off chasing the duplication monster Now there's another issue which I'll come onto later which is inheritance versus composition, but I want to spend some time thinking about duplication. I think how we think about this and address it is probably one of the biggest factors in the maintenance of a project. We talk about dry, don't repeat yourself, which I think is a sound principle, but is easy to misinterpret or misapply. I'll give two contrasting examples. This time from my my current clients, MAT Market Access Transformation. I've been working them

Speaker 1: for them for a year and a half now. We essentially do market research for the medical industry. Mostly takes the form of online surveys. Due to the unique needs of the medical industry, we have custom survey software, which is also managing lots of other business functionality. So we have a survey model to capture our surveys, or one along with several others. And a survey has a life cycle, it goes through a bunch of milestones, you scope in question development compliance with the client fielding and then we analyze it and so on. Now our company usually writes the survey and its questions on behalf of the clients but the client do have the option of writing their own And in this case, the question development

Speaker 1: milestone is skips. And this is controlled with a simple Boolean fleck that you might get looks like this on our model many other fields where that's decide whether we're going to do the question development or not. So we could state the requirement just like this. If survey. question development is false for a survey, skip the question development milestone. Now due to the way that this evolved, with uh different things getting added at different times by different people, in our first version of our code, I found that this one requirement had been encoded in at least nine different places. in the codebase. Now this wasn't due to copy-paste coding. There were actually very few duplicated lines of code.

Speaker 1: It was more subtle than that. There were a whole whole bunch of places that were implicitly embedding the same logic. For example, in answering any of these questions, which milestone comes after scoping? Or what milestone comes before compliance? Or do I need to validate the fields? Especially we have forecast dates. associated with each of these um or or some of these milestones. Do I need to validate the fields for question developments in every case? And if I do validate those forecast dates, what other dates am I actually comparing them to? And it went on like this. Now this is a classic dry violation and it causes lots of problems, as we know. If I need to change that one requirement, I've got a whole ton of places to change.

Speaker 1: Similarly, if I need to add another optional milestone, I've got a structure that's going to require adding lots of similar special casing. The solution to this is to have a a milestone abstraction that encapsulates the requirements So you each different type of milestone will tell you whether it is included or not, and tell you what is the next milestone and all the other information about the milestone. You ensure that you only use the abstraction and never bypass it. In our case it wasn't a strong enough abstraction to actually be its own model, it was a bit managed part of a model for us but it would still have its own kind of classes associated with it

Speaker 1: and and that's what the cleanup version of our code now does Now this is nothing particularly groundbreaking yet, but what was interesting to me was that the duplication of logic involved very little duplication in terms of lines of code or structural similarity You have to think actually on a slightly higher level to see that all these different places were actually expressing the same requirement So a second example from our code base relates to certain types of questions in our surveys In market research, you often want to find out how much people will be willing to pay for a certain product. But it turns out that asking them, how much would you be willing to pay for this product actually doesn't work very well.

Speaker 1: So various people have come up with more cunning methods to extract the information. There's Van Westendorp's price sensitivity meter and the Gabber Granger method, for example. Now for both of those methods we would have the following database fields. You have a currency, a minimum price, and a maximum price. And then there's other fields which are unique to each one. Now suppose you've implemented the first and then you need to add the next and you spot the similarity, you'll be adding exactly the same lines of code Don't repeat yourself, you remember. Can't we just reuse this code? Well, there are two ways to do this in Django. You could put the two question types and those fields in a single model.

Speaker 1: You'd have a single table. And then the other fields which relate to just one of them would need to be nullable because they would be unused in some cases. Alternatively, you could use an abstract base model to put the common fields in, which would now give you two separate main models, two separate tables. Now the second of those is much better than the first, but both are actually very problematic. The problem is coupling. You remove the duplication of lines of code, but now join the implementation of these two distinct things together. If you want to understand any code that operates on these fields, you have to be thinking about both ways that the fields might be used. If you want to change one of them, you'll end up changing the both both of them, or be unable to change one of them.

Speaker 1: I want you to notice again that duplicated lines of code is not the heuristic, the thing you should be using to decide whether or not you have a dry problem. Rather, the question is this, what is going to happen when things change? When the requirements change, or when I want to change how I'm implementing those requirements If you think about it this way, if you have one requirement in n different places, then you you obviously have a problem. But also if you have n requirements in one place , you also have a problem So duplicated lines of code may not be a problem. It's very common that if you have you have distinct requirements which happen to look similar to each other.

Speaker 1: And we often happen to choose them in very similar ways. So similarity in lines of code is exactly what you'd expect in some cases, and it's not something to worry about. Okay, so um on to the last section of the talk. I want us um to see this in action with with a single view Now this is an entirely made-up thing, but it's based on real-world experiences and weaving together the kind of things which do happen in real projects. So we'll compare and contrast how an FBV and a CBV evolve, especially how the requirements evolve. So let's say we're implementing the account page of an e-commerce system.

Speaker 1: And for the first version of it, all the account information we need is actually fields on the user object. So we just need to put the current user into the template context. Now I'll admit imports and assume various other bits of code exists just to focus on the main logic. of the view. So the FBV and the CBV might look like this. Nothing really fancy here at all. In both of them we've got a context dictionary and we're putting the user Getting the user from the request and putting that in the context dictionary and the corresponding line in the CVB is here. No great, nothing too fancy here and no great difference really. Now, a new requirement comes along. The main thing people want to see on their account page is actually their list of recent

Speaker 1: orders. So let's add that. Again, there's no big difference. We've added a couple of lines here in FPV we've um we've put in an extra item. The orders are based on some database field from the orders which we get by the user and the similar corresponding thing for the CVV pretty much the same with a few slight changes. A couple of lines added. Now let's say our people actually start to buy things on our shop. Great, business is taking off, and we have return customers, they bought more than one thing, and we get to the point where we actually need pagination on on this list of views, on this list of orders. For the FPV, it might look like this. We can add it to both easily.

Speaker 1: We have a few Extra lines. We could possibly shorten this a little bit, but here's the the basic steps. We need we need to add a paginator if we're using Django 's pagination. We need to get the page parameter out of the query string and then we need to we turn a slightly different object into the context for the view. Now we could do exactly the same thing with the CBV if we wanted. But the CBV approach to duplication is to eliminate duplicate lines of code. And we find there is another place in our code base that has extremely similar lines of code, our product list also has pagination and has it almost exactly the same thing. So we don't repeat ourselves. We use list view

Speaker 1: from Django base classes. We can have it from this G there. I'm done and all we need to do now is we add this paginate by, that's the number of items per page, and we have this get query set method And done this way, a bit of duplicated code between account view and product view disappears. Let's think about where that duplication actually came from. The requirements might have been expressed something like this. For a v-Count page, we show a page with a basic user with basic user info and a list of recent orders in reverse date order paginated into 10 items per page. On the products page, we show a page with a list of products that are for sale

Speaker 1: in alphabetical order, paginated into 20 items of a page. Notice how many similarities there are in the requirements. In addition, there are other similarities. We have decided to implement both of them using server-side rendered HTML, and we're using Django and the ORM and the template engine for both. It's really not a surprise that there's going to be similarities between our code and duplicated lines of code. The requirements are similar. Our implementation decisions are the same. Duplicated lines of code is expected. If we go chasing that duplication phantom and feel we have to eliminate every duplicate line of code, he'll lead us on a wild use chase around the jungle. Now here's another new requirement.

Speaker 1: Turns out this list of orders is getting a bit unwieldy. We're going to have a separate page for the full list of orders. But still have the most common info on the main account page. But now it's going to be split into two. All the orders that haven't been shipped yet. And the first five completed orders. with a seal link. So now we've got two lists on one page. Now for the FBB the this change is straightforward. You get rid of um the uh pagination and now we've just got two lists. We've got the in-process orders needs to go into our template and the shipped orders, well or at least the first five of them. are going to to to go there as well.

Speaker 1: For the CVV, we've got more complexity and choices. But we don't no longer need paging, so we could remove that. We could keep using the list view for one of these lists. So it would look like this. Now we've still got the list view And for the in-process orders, we're using that thing, but then for the other list, we're putting it directly into the context as a separate line. That would be fine. But it's now very asymmetric with respect to the requirements. We've got a page with two lists, which we're for some reason dealing with in two very different ways. You could remove the list view and go back, but that would mean more refactoring again. Now I find this kind of thing comes up a lot.

Speaker 1: in in contrast to to JSON or REST APIs, which are usually very well-defined, very strictly defined and often strictly kind of PRU-based. For HTML pages, it's very common that a page might have two or more things. You might have a page with multiple lists of things. You might have a page which fetches a single object from a DV and then shows a list of related objects to that. Or you might have a form with two or more submit buttons. a preview button and a save button or maybe multiple forms on the same page or you might have a form with most a list of things a page of a list of things under form Now for an FBV

Speaker 1: and the approach I was outlining, none of these cause a problem, but with CBVs, each base class and in fact each mix in establishes its own little framework that imposes a structure on us and I think often gets in the way. The other way of looking at this is is a versus hazer, coming back to something I mentioned earlier. Which of these would we say is a better statement of the original requirement regarding the list of items? List of orders. The account page is a list of orders Or the account page has a list of audits. Well, given that the account page also has other things, navigation and other account information and maybe all kinds of things we might add.

Speaker 1: I think it's clear that the account has a list of orders, as well as other things. But inheritance is really for when you have an is a relationship. We've got a has a relationship. And I think that's a very strong hint that inheritance is going to hurt us. Rather, we should be building up a view by composing all the bits of functionality. that we need and this gets us to the other big principle that I that I mentioned and this is again absolutely nothing original you should favor object composition over class inheritance. And that's what the FBV did all the way through. It simply included little bits of functionality without needing to be restructured. And I think if you follow this principle, and if you don't go chasing the duplication monster, you'll find the evolution of each view in your code

Speaker 1: base will be much smoother. More time on the path and less in the jungle. Well, I hope that has been interesting and useful and thank you very much for listening.

Speaker 2: Thanks very much, Luke. And um I'm wondering if there's any questions from the internet Otherwise I will ask yes, there is a question. Question from the internet. Question from Hans. How about the ability and simplicity of testing function-based views and class-based views? So a question about the uh Yeah, the test the the test ability of these Yeah, I can put it like that.

Speaker 2: Luke, did you hear the question just before? I

Speaker 1: didn't hear that. Didn't catch that.

Speaker 2: Okay, I will repeat it. Uh Hans is asking, how about the ability and simplicity of testing in the question of choosing functions versus class-based use?

Speaker 1: I guess my experience is that they're they're very similar. I think I'm not sure I guess it would depend on exactly um you you may have other experiences. What I would tend to do with with function based views is try and um I would have very few functional tests. I would try and make them include functionality from somewhere else and try and test, do most of my testing at say the model layer. I think it's actually one of the disadvantages of class-based use is that you often say if you have a mix-in, it's actually very difficult to test a mix-in in isolation. Because it you you it has it's making all these other assumptions, kind of implicit assumptions

Speaker 1: about what context it's going to execute in. And you could test it in isolation and it wouldn't work in in some other context Whereas with the functions you can have you might have little utilities or you might have decorators that you can test and usually these depend on an interface. So for example, if you have a view decorator It only cares about the interface that is decorating any view. And it because it's only using the interface, then you can test that thing in isolation. So I would um have a few view level um tests. I would then try and test at other levels, test my decorators, test my model layer. You may have other experiences.

Speaker 1: Well how is it?

Speaker 2: And and uh that uh leads me to uh uh a thing I was wondering uh to about mix-ins. Uh Uh are mix-ins bad? Um are are we d uh s sliding into the wrong path when we use mix-ins for our view structures?

Speaker 1: I think the most useful thing I've I've seen on mix-ins is Brandon Rhodes stuff treatment of mix-ins. If you read my um guide Django Views the Right Way. I linked to him a number of times. I'm a bit of a fan of Brandon Rhodes. I use them, I certainly use them for some things. I try to be very careful about how I use them. They it's very easy for it to work in one context and not in the other. And I would um I think um Django Rest framework by contrast shows much better use of mix-ins. So if you use Django Rest framework, it it occasionally uses mix-ins, but most of the kind of um ex

Speaker 1: Most of the way it includes external functionality is by separate classes which have their own interface. So you have serializers, you have authentication classes, you have filter sets and those kind of things which are all actually separate classes with their own um interface. I think that's a better way of structuring the mix-ins. for most things they still use mix-ins for convenience often in Tesco they use mix-ins quite a bit just to make just to make stuff available on self it's maybe it's laziness i don't know

Speaker 2: And it's not really a a question of whether to use them or not because there's so many different cases and complexities. The next question from is from Marco. Marco says some teams use constraints on how many times you can extend a class. Do you think that this kind of approach can help in solving the confusion created by inheritance in that case? How many times would you suggest to extend the class? Is there like some kind of upper limit to to this in theory and practice?

Speaker 1: Um I'm not sure what the limit would be. I would if you are going for the kind of class-based um method, I would suggest um being in control of the class hierarchy. So even if it means you know starting by copy pasting a bit of code from Django and and so that the that base class you can control yourself. then you then you can control a bit better and you can impose those kind of rules. And you can put if you need if you need some level some code at the top level, you can put it there because you're in control. I think you you get a special problems where you are already inheriting from a set. You've already down you're already at level four and you're adding on levels five, six and seven. And then you get I think you you make life really hard for yourself in that case

Speaker 2: Um we've got uh two more minutes so and and uh I would uh really like to ask you something. I don't know if the answer can can fit in two minutes, but um How how has the reception been for this work? Uh you've done massive uh work on on uh on your website about uh about views um that we've linked on uh twitter and on zulip um how's the reception been and and uh what do you need to like progress with this work from the community

Speaker 1: Um I had a really good reception. Obviously, there's different opinions, and that's fine. That's why I I made something that you know is my opinions, and you don't have to agree with every word. Um I was I was touched by some people who um who were quite um I was quite surprised by the strength of their reaction in terms of being pleased by it when they've really they've actually really struggled with class-based views or they've been made to feel like they're stupid because they they struggled with them. Um and but having somebody else say, actually these are really hard and here is a simpler way and and if you if you don't like class based views And you find them too hard, that's fine, because I don't like them either, and I find them too hard. Some people were actually quite touched by that.

Speaker 1: And I think that's something we should take into account in terms of a kind of an elitist view of you know if you're not using classes, if you're still using functions, then you're kind of you haven't graduated yet, but a lot of a lot of programmers, you know And there's a whole programming methodologies around all functions. And we shouldn't be looking down on them at all. I think we should be more exploring what they have to offer. There's more stuff I'd like to do. I'd like to do something on kind of security, how to apply global security things. If uh if other people have other ideas of what they'd like me to cover on that guide, I'm I'm open to adding more things to it.

Speaker 2: We'd be uh delighted to continue the this conversation and like experience sharing. Uh and I hope the conversation will indeed continue uh with you. Um and uh thanks so much for uh joining.

Speaker 1: My pleasure