This Old Pony: Working With Legacy Django Apps by Ben Lopatin

Speaker 1: Come on, no.

Speaker 2: We'll do a mic check now. It sounds good. All right. Great. So um again, Ben LaPatton. I'm uh co-founder and principal developer at Wellfire Interactive. I'm 50% of the company. We're based in the DC area. I'm out of upstate New York. And I am Benny Lope, pretty much everywhere on the internet, and that's also what I look like pretty much everywhere on the internet that I have for the last decade plus. So uh today we're going to talk about legacy software, legacy Django specifically. And I want you to think for a moment about legacy software. What do you picture? I want you to sort of picture something in your head Might look a little like this. Um but the mind does wander. These aren't on fire. Um

Speaker 2: something else. You know, it it it conjures up all kinds of things. Um No, it's uh well that wasn't also a pony in my first slide, so there you go. Um what uh what we want to do before we start is uh is define our terms. So what do we mean when we talk about not just legacy Django, but legacy software in general? And there are a few definitions that we'll work with. The first is just code that someone else wrote. At the risk of begging the question, a philosophical question about identity, this could also be past you. Okay? Um if anyone's ever looked at code that you wrote like a month ago, I was gonna say a year ago, but a month ago might suffice that this could be considered legacy code.

Speaker 2: Michael Feathers, who literally wrote the book on working with legacy software, defines it as code with no tests. I think this is a good example of a characteristic of legacy software, but I I I don't like this as the definition. The reason it calls it this is basically you have code that you don't know if you're improving it when you make changes. I think that's pretty accurate. Another that I want to go with a little bit more strongly is that it's software that's in production. So if you write code, there's basically there's this two-way fork like uh decision tree of what can happen with the code. It can be used or it can be not used. That's pretty much it. So if you have code that's not being used, it's probably thrown away and it doesn't really matter anymore. It's code that's used. So here we have this bridge.

Speaker 2: It's an old bridge. I imagine it's still there, but this is a bridge that's in use. Alright, and that'll actually be a little bit important as we go. So the reason this is important is because this written software is an investment. There's this literal investment potentially of time and money that went into it. But it's already there. You already have patterns. Um, you already have a lot of information that's that's uh encapsulated in this software. And uh as much as it might pain us to say, a lot of these systems still deliver value, and that's that's the point of the software, is to do something There might be people depending on this. And uh actually we know there are. Does anyone here bank at all? Um You're using legacy software. And the world runs on it. That's not to

Speaker 2: uh forgive it and say, ah, well let's just live with it as is, but is to acknowledge that it's out there and that we need to work with it. So when we talk about legacy Django, let's define this a little bit more tightly. It's really not going to be that different, but there's some characteristics that we can talk about with Django. I think the first is going to be pretty obvious You're likely working with an outdated version of of Django. Your product's dependent on an outdated version. Um you could also have some uh issues around how it's deployed. You know if you've got If you have uh Django 1. 3 uh project, that might be a problem enough and it's deployed with mod PHP, let's say. And so now you have this other issue that kind of compounds the problem. You can have no tests in the project.

Speaker 2: This is not the worst problem you can have with regard to tests, and we'll get to that. So uh you know you can also have some older dependencies when they're outside of uh of Django. And again, these are not by and large totally unique uh issues to Django, but we're gonna be talking about these in a way that kind of that how we work with Django, there's certain patterns that there's certain kind of patterns you might see in legacy software, and they show up in certain places in Django. So when we're talking about this, I want to bring out some assumptions. Um obviously one of them is we're working with Django. Uh another one is that um the soft assumption we already mentioned, you're probably working with an outdated version of Django A really important one, which is not necessary, but it's going to be that we're working with the project that you're, you know, so you you come on to, that it's already running in production, that people depend on it ready.

Speaker 2: So it's not like, oh, here's a project, here's some code, we're thinking about redeploying this. Everything we're going to talk about still applies, but it's a little bit different, probably a little bit simpler to solve. And of course, since it's in production, people are still relying on it. Now the goal in all this is to improve the code base, to let's say maybe fix bugs, add features. That's probably why you're if no one wants to do anything to the code base, like fix bugs or add features, you have to ask why are you doing anything with it It. Those are really the only reasons you're going to be do working on the code. And so the goal is to make these really discrete steps as you go. There's an analogy that a friend uses that, you know, this is uh it's like working at an old house. So this old pony is from this old house. I like the analogy of a bridge. So a bridge is going to span two points and let people get from point A to point B.

Speaker 2: They can fail catastrophically. And the goal is to try and do work on it before it gets to that point. But you have to keep the bridge open as you go. And so that kind of goes into our assumption about people still getting value out of this. Now I'm going to tell you a lot about what to look for and then some solutions. It's not this this is not codified. These are things that I've kind of picked up after, you know, I've spent More time working on existing Django pods than I had creating uh greenfield work. And you know from a lot of this is I've written stuff that you might consider legacy and had to work on it later and I've worked on um code that all kinds of other people wrote, sometimes other uh Django agencies. So you might say that it's just like my opinion, man.

Speaker 2: I take that for what it's worth, but uh it is based on some uh some learnings So the first step when you get to a legacy body is just code review, you want to assess what you have. This is going to be like the first step in the OODA loop. Orient, observe, decide, and act. The reason why is we need to get some context. We really need to understand where the product's coming from, where it's at, before you can start making changes to the code base. So the first step is actually the zero step is to ask questions. This is talking to people. Find out they might be stakeholders, it might be your manager, it could be clients. And you want to get some understanding from outside of the code base. You want to know what is this supposed to do?

Speaker 2: So someone might know that. This is an old code base. There might be someone who's new and say, well, this is what it does. And someone else will say, well, it's supposed to do this. You want to know about known bugs. This is really important, I think. Before you start digging into the code base, find out what you can about known bugs. And then planned features as well. This is going to influence how you look at the architecture This brings you to reading the code. Um this is an art, not a science. Uh basically what you're gonna be doing is looking for um you want to get an idea of what it does. Look for things like I don't know confusing areas in the code base. Look at the architecture. Code smells are gonna be a big one. The style and see you know how is this written? Knowing kind of what some of those bugs are beforehand too will help you.

Speaker 2: You might even see bugs that are obvious as you read through it. You might see some that wouldn't have been, but they are now because someone told you about an issue. And uh right at this point you're just basically taking notes. Remember, we're not we're not making any changes to the code base. Um there are some tools to help you with this. The the main ones are gonna be reading it. There's really nothing more you can do to uh get beyond that. But um some of these tools will help get you some understanding of what's going on in the code. So I pretty much I use Flake 8 and PyCharm as well. They didn't pay me to say that. And you can also use Pylint. And what these are going to do with Flake 8 is going to combine some sort of stylistic analysis. If you're not using Redn, I'm going to guess a lot of people are. And some some static analysis of your code.

Speaker 2: And um what you can , this is gonna this could be a lot. This could show you a lot of errors. And what I've done is I have a script that I've linked on uh a blog post. Basically we would just take this output and give you a summary of what this looks like on a module by module basis and an error type by error type basis. Because if you find out that there's a whole bunch of modules that have like, oh, there's no spacing around an operator That's gonna impact how you read it, but it's not gonna it's not that serious. Whereas you find out there's a whole bunch of names used places that are never defined, that might be a source of runtime errors. So you want to kind of categorize this as you go Now, now that we've kind of done this, it would seem that the obvious next step is to get into testing. But I'm going to suggest there might be an intermediary step, and that's adding a little bit of logging. If you remember, we wanted to talk to people,

Speaker 2: interrogate them about some of the issues in the code. And basically what we want to do is understand production errors. So again, there's another sponsor, did not pay me to say this, Century quite a bit for this. You don't have to use them , could be at a similar service. And you want to just add in this exception handling. It's not good enough to get emails About an error to s like a mailbox that no one's checking. Um you want to get these, you want to make sure you know what's going on, get good stack traces. And um there's also a point which I'd suggest making some code changes before getting uh looking at tests, and that's Code looks like this. When you see this in a code base, it's probably not gonna even look like this. It's probably gonna be like 20 lines. And there's some calls to external services, and then there's like an accept

Speaker 2: clause. And maybe there's like one, but it's just like accept. Chris wrote this. Maybe there wasn't time. Maybe this was a temporary thing. Maybe the code was going to be thrown away. Who knows? But um this is there are exceptions here being swallowed. And maybe it's acceptable to some degree because this is some sort of customer view or they don't see this and so it's not that big of a deal. But this could be hiding bugs So what you can do is you can go ahead and add some logging here. If you're not familiar with um the exception logger, basically what it's gonna do is it's gonna send the whole stack trace out. So if you're using something like Sentry, you'll get what looks like an error message. It's never raised, but you'll get the full status and you can see what's going on. This will allow you later to come back to this code and properly address it.

Speaker 2: And that brings us now to testing. In case you're wondering why testing, I'll give you a little visual prompt for why. I even talked before I had like a whole bunch of screenshots of that from airports. This is gonna help you fix bugs. It's going to guide development. You don't have to do religious test-driven development. You can do a little bit of that. And it's going to give you deployment confidence. You know that We have close to 100% confidence that you know what you deploy is is going to work. And what I want to tell you here is it tests provide information about code quality. Again, that's gonna be key in more than three or four slides. But but that's the goal. It's gonna tell us what's going on. So that's of course if we have tests.

Speaker 2: So what are the test suite scenarios? You you've come to this new project. Basically, if Dr. Seuss were writing about tests, this is how he would write about the scenarios. Okay, you could have a few, you could have great tests, no tests, bad tests, slow tests. Great tests are great. This means you probably have a lot of coverage. They're meaningful tests I want to put emphasis on that. Coverage is a great metric, but it is a false God. Do not worship it. It just means that every line's been evaluated. You could have junky tests that you know don't test every type that can go in, and you can have failures because you didn't test that, because they weren't meaningful. So it's good coverage will tell you where there might be dragons, but it's not going to tell you that this is amazing. So that, but this is, you know, if you have great tests, then you're probably golden. Everything's passing and and it's passing because these are good tests.

Speaker 2: You could have no tests. Again, not the worst scenario you could have, because at least you don't have any noise. So we have signal and noise. Bad tests I think are the worst scenario you could get. Especially if they're combined with slow tests Now bad tests could be that you have coverage, everything works, and it's just a bunch of stupid tests. I don't know. Someone went through and just like mod someone patched the like test case class so that all the asserts always pass. I don't know. It's it's stupid, but they don't hash. Like hey, but these are terrible tests. You could also have tests that are failing because you have test drift. You know, people just started you know no one was running the tests and they were they were making changes to the code base and not you know updating the tests. Um you could have errors for very similar reasons. Uh this is bad. Um this is

Speaker 2: This is not information. This is just telling you that the tests are not good. It's not telling you anything about the code base. So the strategy I would recommend here is you know you any of the changes you're making to the code base, you're making incrementally. We're gonna do the same thing with these tests and basically the first thing you do is silence anything that is bad, especially if you don't know why. Because really what you want to do, if there's a test that's failing, there's an error. It indicates that either the test needs to be fixed or the code underneath it that's being test needs to be fixed. And if you can't figure out what that is at first, silence the noise and wait till you can come back and see if you can get signal out of that. And of course we have slow tests. Slow tests are bad because no one wants to run them. It's terrible. This could mean like it takes five minutes to run the test suite. I came under a project where it took 90 minutes to run. I ran it a few times,

Speaker 2: I stopped it after like 20 because my computer felt went to sleep. And I asked the client and said, yeah, it takes 90 minutes after I ran it on another server instance somewhere So the the solution there is to speed these up. Now the reason those tests are really slow is because they were really fixture dependent. I would argue kind of outside of the this the scope of this talk that using heavy fixture files or fixture files in general is not the best idea. But if you have a legacy project, you might have a lot of them. We're talking like thousands and thousands of records and every single test case loads all of these and that's where all the time comes into play. It can also be because the code underneath the test is slow. It could be because you're making ex the the code or the test are making calls to external services So these are basically all anti-patterns. And what you have to do is figure out you have to prioritize which of these are going to speed up first.

Speaker 2: So with fixtures, I was able to get these down to 90 seconds with like four lines of code by just using Django Nose. and the uh fast fixture test case. Um if you want to use that, it's not currently supported in the latest version of Django Nose, but I have a fork on GitHub that does. But it's not tested.

Speaker 3: You can

Speaker 2: have everything. And you can also just make sure you're not using the database excessively, make sure you're not doing too much I. O. That could mean not saving models when you just need to test a method on a model class that just Just uses something out of the model or mocking. And uh mocking is great for this. So that's your prior That's going to help kind of give you a guidance of what's there. When you're adding any kind of new tests though, we want to prioritize, um especially if you have no tests, how we add these in because the There might be a temptation. Well we'll just write a full test suite. If you have that temptation, you should probably put it aside, especially if you have code that's that's in production you need to make changes to. The first thing you're gonna prioritize are are bugs. This is actually a it's a beautiful millipede um from Virginia. Um so it's not really not a bug.

Speaker 2: But um you'd want to add in uh test for bugs. Anytime you have a bug, you write a test for it and and you you fix it. That's probably not new to most of you. The next, I'd say, are going to be what I'd call um you know the smoke tests or these integration tests where just Load the views. If you don't have any tests, there's a lot of stuff you want to test. One of the basic things you do is just do some client you know using the Django test case client to load views. That's gonna be a it's not the cheapest way of testing, but it's gonna be the quickest way of testing a lot. If you don't have any tests. And that's really important when upgrading Django is urgent because you want to get as much tested as possible. And then anytime you add a new feature, you should be adding this. And refactoring. So refactoring is by definition, it shouldn't be changing how code works.

Speaker 2: It should really just be changing names and extracting code. But this is a really good opportunity to add tests for what you're refactoring. And then that brings us to the upgrade. This is everyone's favorite part of working with uh legacy Django is upgrading to the newest best Django version because within you know with an hour of work you're done and you're on Django 1. 9. Yeah, it's it's an hour and different, you know, relative time. Um the the issue you're gonna have here is backward compatibilities. So, you know, all kinds of things that have changed in each Django version. It could be as simple as, you know, URL patterns is deprecated, or it could be that you're using, you know, get query set was renamed, and you've got to fix this in your code The

Speaker 2: superpower here that's gonna help you do this is Tox. I'll repeat it. Tox is a great it's a testing tool. If you're not familiar with it, it's predominantly you've probably seen it in uh with reusable apps or other Python libraries And it um controls testing environments. You can have isolid testing environments for a matrix of of whatever you want, all kinds of dependencies So you usually use it with a Django reusable app, but it's really useful with your own project when you want to test a small matrix. Like, I want to test this code base. Against the current version of Django and the next version and the next version and maybe see where I have some issues. So the way this works is you set up the TOX file, this is your configuration file. And the reason I want to point this out is that um You'll see that I have uh there's a requirements file that we're installing the dependencies from, and then Django

Speaker 2: is isolated separately. The one thing I found doing this is that you do need to pull the Django version out of the requirements file. Um And so, you know, that would be like a say a root requirements file in this case. And then you can define it here. So we're gonna run these tasks against Each of these versions of Django. And that's a really great way of kind of doing this in place. You can see what code, you know, how code works. You could potentially change your code base, get it working in this version, whatever version you have deployed, and the next. And then at that point, just deploy the changes and then and then upgrade Django. That's kind of a nice way of doing it. The goal is going to be getting to an LTS version. I don't care if you want to get to like 110, you get to the LTS version first. For a lot of legacy apps that are in production, that's probably just what you want to do. That's your baseline is

Speaker 2: go from LTS to LTS. But this is this is really not even the fun part. The fun part's working with all the dependencies that you probably need to use. These reusable apps. These pose a few minor problems. You could have some integration with obsolete libraries in there. Hopefully no one is working Hopefully you don't have to do too much work with SOAP in Django. And there's a lot of new libraries out there for that, but like there was a gap where there was nothing new and you were pretty much screwed. Overspecification, and this is not an issue in your project so much as the dependencies, where dependencies overspecify their version compatibility and you're screwed because you get version support mismatches. So here's a diagram of

Speaker 2: this is what versions look like. I don't know if you guys know this, but this is what they look like. So here we have You know, the bottom is going to be the lowest version of Django that the teal package is compatible with, and the top is the uppermost. Whether or not these are specified by the way. This is just what we actually have. And so the orange is the that's the current Django version. And now you upgrade. And now you have this. You have packages that have not been updated. You have packages that have been updated, and they're like, you know what, we're not supporting that version anymore. And this is the problem you run into. So you have to you have to solve this, right? That's why we're here for solutions. Um the first solution is what I call patch and bread, where it's uh you know an up there's an upstream and it's an you know it's on PyPI

Speaker 2: and you know what I'm gonna I'm gonna be Helpful. I'm gonna you know patches make it work with the next version of Django, give a pull request, and within a couple hours this is gonna be up in PyPI and we'll be good to go. That's not how it works. If anyone's done any work on a project like that, you know that it's the weekend. You have stuff to do. You don't want to do that. Or maybe you don't care about the package anymore, it's the maintainer. So this is not a bad strategy to take, but it should not be your first strategy. It should be like a secondary. Another strategy is to fork. Now this could be forking as another published project, it could be forking to a private um index, or it could be forking and using um your forked version from Git. know, or or Mercurial or Perforce or what I don't know I don't know if you use if you guys use Perforce.

Speaker 2: Um but you can do that. And you can also kind of vendor it and have a vendor fork where you've actually had the code just sort of added to your code base and work on it from there. Related to that, you could just extract what you need. You might just be using one or two modules and it's this huge app of you know it's got models with It doesn't even have south migrations. Um and there's all this other crap. And you're like, I just need this tempo tag library. I just need this. And I actually don't even need to change anything because there's nothing in there that's incompatible with the version of Django I've got. So you can just Just pull that out as a separate app, you know, vendored, and um you're good to go. Ideally with tests, but uh I didn't tell you this, but you just Include it. And of course you can just remove and replace. So you just say, you know what, we're not gonna, we don't need this dependence anymore. We can make do with something here, or we can find an actively

Speaker 2: maintained alternative. Some of the tools you might use for working with dependencies here are Pure and PIP tools. The reason I like Pure is Pure and You 'll forgive me. This is the example from the Pure site. I know it's not Django. You basically point in a requirements file and it will look at your requirements and then read, it'll update the requirements file in place with updated versions. This is good for just kind of testing to see what works. You just upgrade everything and say great. Let's see what breaks. So that's your dependencies. And there's some other issues you're gonna find in the code base. Um I'd be remiss if we didn't talk about formatting. Uh there are stickers, I'll have more stickers later too. Um free. This is not a Django

Speaker 2: specific issue. It's not even Python specific, although we're there since we have uh kind of a standard, see it. Um If you guys saw the talk on readability, you understand that there's a readability issue. I think that a lot of the bad formatting you'll see can it hides bugs, and so that's why it's an issue. It's not just an aesthetic issue. There are a few tools you can work with here. AutoPEP 8 will automatically format. And PyCharm, again, we can do some formatting. I am cautious about doing too much uh formatting like this when there aren't tests in place. It should be kosher, but it it you know you you want to be very cautious. Another settings. And really the issue with settings is not just maybe gnarly settings, but secrets. Has anyone ever here

Speaker 2: Actually no, don't raise your hand if you've ever committed a secret to a reason. Keep your hands down if you have. It's happened. Is there some price where this is kind of just this is the way it But it's written. And so you this is you want to get these out early. This is one of those first changes you make, and maybe with with logging. Get this stuff out There's a tool for that called Bandit. It's a Python tool. It's part of the OpenStack, OpenStack project, and it will look for security vulnerabilities in code. Um it's just gonna build an AST and do this. And it'll look for stuff like that. It's not 100%, but it'll give you some good signal. The other is uh just having one big app. You know, you have this monolithic app And you've got like 50 models and a URLs file, a URL configuration that's got, I mean it's just huge.

Speaker 2: And you can't understand what's going on. The solution's pretty simple. You break this up. The way you start with this though, I think is with the simplest things. You know, URLs and views. You can do models too, but save those for last typically. And the way you move models is you use the dbtable attribute in the meta class and just define the table name so you have the, you know, keep the table name what it is. And then with a little bit of migrations dancing, maybe some squashing, some faking, you don't change the database, but you you make Django think you've moved stuff around. And that will work. Now there is similar to this is these big views, lots of logic and views. This is uh this is hard to test Um it's hard to read through. There are a few solutions to this. This is kind of funny that there was a blog post

Speaker 2: in 2006 by Janus Buck about this in in Rails. So we we know about this issue. Fat models is kind of one of the solutions to this, putting a lot of the logic on the model itself. I'm a fan of humongous managers. If you guys saw the manager's talk, you may hopefully persuaded of this. There's a lot of logic you can put in managers that might be in your views and it's just much better place to manage. It's much easier to test here. And we could say expansive forms as well. And specifically a lot of uh, if you saw the form stock, you do a lot of interesting validation there. Um and there's a lot, I think, that can be pushed to form validation out of views. It's not just views. Um does anyone here use any management commands? Yeah. They're an interface. Views are an interface, those are an interface.

Speaker 2: They should have comparable levels of logic in them. So management command should really just take input from the command line and then go out to another function, some manager somewhere else, pull the data in. If you're doing like CSV munging in the manager class, You're probably doing it wrong. Um and in the end we want to make as few changes as we can kind of sequentially. Refactor as you go. Okay, you don't have to do you don't have to change everything. You have a new feature, you can put that in the new app. If you you're using some old code, making some modifications there, refactor that class or that module And real quickly, just a few items that we didn't talk about. These all matter a little bit, some more than others. Automated deployment is going to help a lot when you're starting out and trying to get some of these changes out.

Speaker 2: But in the nature of time, we'll end it there.

Speaker 3: We do have a few minutes if anyone has a question.

Speaker 4: Hi, I use PyCharm and how do I use PyCharm to do code reviews?

Speaker 2: Well, so the thing I would use there is is really for formatting the code and for reading through it. So Yeah, and um optimizing imports. And it can it'll show you too uh I don't know what you need to have configured, but it can show you a lot of hints about like where something's going on. You see the little red error or yellow or something else like that, like oh there's an issue. Um

Speaker 5: great talk. Um okay, so and this is directly from experience with a legacy app that like kind of what you were talking about. But when you okay, you talk about having large managers, small views. I get that. I appreciate that. I've run into situations though where There is way too much cross-dependency between models. You have huge managers with logic that isn't really related to itself.

Speaker 2: Yep.

Speaker 5: Where do you start with that? I mean I know that's that's part of the refactoring part of it, but I mean it's one of those If you want to keep traffic crossing the bridge, so to speak.

Speaker 2: Yeah.

Speaker 5: You know, and a lot and and this tends to accompany m most of what you're talking about, you know, like with the test suite that is kind of present, but a lot of it fails So I don't know, just what are your what are your comments on that?

Speaker 2: So I think I was gonna briefly include a slide about that. Didn't I think one of the first things you do is just refactor the view. So before you just taking that stuff out and putting it into a separate function. or a separate method, that can be the first step. Rather than putting into a manager or a model right away. Uh for probable functions. Yeah stuff too.

Speaker 5: And and this is this is where this is where the logic was in in managers. It should have been a bit different than models or a different uh apps manager.

Speaker 2: I've also used top-level functions that managers reference. Yeah, do that rather than because I find those easier to test. So last question.

Speaker 6: Yeah, one short question. You mentioned refactoring. How should I explain my customers that refactoring is valuable?

Speaker 2: Ah that's a really that's a really good question. Um and that your customer could be anyway, it could be a client, it could be you know, manager, your boss. Um that uh I I mean it depends on the person. I one of these I like to use some of the physical examples and say it's it's basically having uh you know a messy job site. You're you're gonna have injuries. You need to be able to see what's going on and it's really difficult to do that. You'd also just I'm sure there's some stats out there you say look um I read it in this book, just point to a book they're never gonna read. But you know it's like 20% likelihood of errors and they you know that that that's gonna go on. You really have to focus, they're concerned about um speed of delivery and bugs, usually And so you can say, look, this makes it really hard to work on this. We're spending a little time up front to do it.

Speaker 2: Customers are probably used to hearing that too. But saying, look, there's some bugs and if you can point to issues that were would have been easier to find, let's say, and and better written code, then that can be kind of what your starting point. But I don't have a great answer for that, unfortunately.

Speaker 3: Alright, so it's time for lunch, everyone.

Speaker 2: And uh real quick, there's I have more. There's some I love pep eight stickers up front, and I have more if you want one. And uh I'll tweet out if you want a shirt and buy what's on laughter.