Upgrading EOL Django: A Journey from V1 to V5 with Michael Riley

Hello, welcome to J-GoCon US 2024. My name is Michael Riley and today we're going to be talking about upgrading endless Live to Django versions, a journey from version one to version five. This has been a talk that I've been wanting to give for some time now, and I'm really glad that I'm able to. Also, this is my first talk, so I am also very excited about that. Now let's get started. First, let's talk a little bit about me. I'm the lead support engineer at Platform SH. I'm also the lead developer donation store at night. I've been in the IT field for around seven years now, and I've also been making python applications since 2018. You can find some of those under um right or right softworks that's the company that I publish my work for through now let's talk about

the Django. But first, legacy applications. It's an important term to know and I feel like that I want to clarify it here for those who do not know it, as well as a little bit for those who might know it. Um, legacy applications are typically those that run in ancient versions of framework. Frameworks, a lot of the cases those are end-of-life versions. Usually they receive minimal attention from the developers if any and they're running ancient versions or unsupported versions of Python. Also the best part about that which I'm sure your security love security team loves is that it has security vulnerabilities. um great combination now there is also another way that that definition can kind of go

legacy applications can also be um applications that are But that have been preceded by a newer version and are not as actively used by the customer base. You might have an opt-out feature while letting most of the customer base to use the newer application and it's gonna be phased out soon that is not what we're talking about here though I just want to clarify that so let's continue Now in the cases of where you have a legacy application or you may come across a legacy application, what do you do? This was something that I had to think myself um when I first started working on a one of donation store

And now one the first piece of advice I can give you is right in over here. Don't run for the hills yet. And don't panic. It is not helpful. And surprisingly, what I've learned through this journey is that Django is very robust in how it works. And I don't think in most cases there is cause Panic. You don't do a good job when you're panicking. You don't make good decisions when you're panicking. You need to do this with a calm mind. Now, the next step is to check if there is documentation on development. workflows to the legacy application. Some people might be in on your team might be familiar with your application, but let's assume that this is a completely separate team who has never touched the application before.

was the case with me. This is why I recommend checking to see if there's already some workflows in place. You need to ideally be able to deploy a separate environment than prod or your staging environments. Staging should be f for testing any other changes that are currently not in prod, but you need to your it should test changes that are specific to whatever the version is in prod. You want to have a separate environment for upgrading. Having a local environment will also save you a lot of headaches throughout this upgrade process which is why I recommend doing a local environment rather than a remote environment. We'll go into more other reasons later. If you do not have any documentation on development workflows,

this is you're gonna need to start making some which is rebut my next point if you don't have a local development environment um Talk with your team and figure out what everyone is most comfortable with. Some examples of local development environments are Docker Compose, D dev, and Podman. And more particular, those are a multi-app. development like local app um multi-app local development environments you want to be able to spin up multiple services so then you can test your application and develop it as sanely as you can that replicates what your production environment would look like

You also want to have a remote debugger for this ideally. You need to be able to see what's if you need to set some breakpoints. Now this isn't a hard requirement. I recommend it, but the Django debug is will tell you what variables are present during um an exception. Like an error that occurs. Um, but it is always nice to have that, which is why I kind of try to push for that. I'm sorry um about that. Um so testing upgrades um You're gonna want to do this in a local environment and the primary reason also coming from a pat platform as a service background is that a lot of these different plat

um services will use build minutes And when you're going through the upgrade and you're gonna have to rebuild each time, you're gonna be clearing that cache, that build cache a lot if you're gonna be doing this in the remote service. And it's gonna be retrieving it from the Python package index a ton. You do not want to waste those build minutes and using it in a local environment will not exhaust those and at the same time same time your extra power for your machine will make it go faster. Saves you time during development. This is why I recommend it. Let's go to the next slide. Now let's introduce our legacy application that I worked on and that I'll also be talking about throughout this um

Donation store. Donation store was acquired in 2013 by a friend of mine from an R IR software. um company uh I invested in it and I started as the lead dev began developing and reworking it. Um prior to this I did not actually have any experience working to with the Django. I had to learn that on the spot, which is why this is really interesting. I also had to prepare local development environments because Two out of the three projects that you'll see very soon did not have any Docker stuff and I had to configure that myself. Um Now what are we working with? Um donation store has

a main site which has the point of sales and accounts, the API and the and then the products. In terms of technical specs, the main setting API run on Python 3. 5, which is the Django 1. The product itself that allow that customers can self-host, it's on Python 3. 8. into Jingle 3, which I want to say that is technically still um supported as of today. But the Jingle 3 is going to be End of live soon. All the products itself use MySQL MariaDB for a database, and only the product uses RabbitMQ with some So two out of the three projects are very simple.

The other one kind of has a lot of extra dependencies, which is why multi-app is so useful. Uh let's look at setting up the local development environment. What did we settle on? So we use Docker Compose um primarily because it has historical images for Python on 3. 5. Um not having to jump through obstacles to get a legacy python version run in. Uh 3. 5 is not supported on Fedora 40. So there's just no default binaries in the repository that I could download. you'd have to compile it which we'll talk about some of the hassles later primarily it was my familiarity with Docker because I have used Docker for years It has been out there for a while.

It just did not make any sense to use Ddev or Podman in that case, which I know is a fork. The thing is that Podman has some differences. that just I did not want to have to deal with two different unfamiliar territories than I had to deal with. I didn't want to deal with anything more than I had to, which is just my unfamiliarity with the Django. That was enough. Let's just use Docker. I'm a single dev team. There is no one else really helping out with it. I have someone helping out in front end, but besides that. That was the decision I came up to. Um we also I would also use Intel uh Intel Jarm for debugging during some

really certain circumstances. I'm not going to mention what it is in the talk, but I did have that set up as well. Now let's look at these Docker Compose files. You'll see on the left The left is the product which has the donation source service, n gen X, Celery, Rabin MQ, and Maria DB. In total, that's five services For the API and main site, on the other hand, this only had

two services, which is the web service, which is the app, and then This just really can show you how Docker Compose can change depending on what your application can be. It really showed both ends of it where there's a lot I'm juggling on the product side, but not necessarily on the site and API side. Sorry about that cough. I just I I hate doing those cuts. I don't want to do those cuts. But um Docker files the product Docker file as you can see has a lot more stuff than the API in the main site Docker file.

This is pretty much just your basic the Django file nothing's really changed in it um it here there's a lot of different dependencies that are relied on in the product I could probably have just reduced the steps but during debugging I wanted to know what dependencies were failing to um retrieve or install so I set that up that way for that reason um continue images this is another topic i want to talk about in the multi-app stratosphere this is the last topic before we start talking about getting into the upgrade Um container images were very crucial and I had mentioned briefly that there was some hassle with Python 3.

5 on Fedora 40. Depending on your platform, it can really Make it difficult what Python versions you have access to. Sometimes it's just source code for newer versions of legacy, very old Python versions. After a certain point they stop compiling binaries it's just source code docker images or in more cases container images do not really have that problem usually it's up to date you can just pull it and it's very simple. Um not to mention compiling binaries uh for python versions you don't have to worry about the dependencies it just grabs it it saves time time that you could be working on the application

Not to mention once you have if you have multiple team members and they all have to get this set up depending on their system, it can be a real big hassle. Um and you'll also be going between python versions. Um so it's probably safe to note that you should just be using a container or a uh some sort of container image for your workflow, some minimum. minimize that. Though that isn't really as apparent with Python, let's say 3. 9 or something, which isn't end of life yet, but you can still actively grab that from a repository. Like apps or a DNF repository. Okay, now I want to show you here

how easy it is to find the Python versions for a uh Docker image. If you go on Docker Hub, you will not see Python 3. 5. It is it's because there's a limit for how many characters can be on the readme so they don't show it. You can very well just pull the image uh s Docker pull python and then 3. 5 and it will grab it um this isn't this talk is not about docker um cli stuff so i'm not covering it but i just want to say that your IDE may be able to simply get this information for you. It is very easy to find versions of stuff of container images in a um

repository on Docker Hub whatever other platform you're using as well, presumably that would be there as well. Now let's start upgrading. This is what we um now that we finally got through the prerequisites and stuff that make this stuff um process easier. Uh let's start upgrading. Now to get started with the upgrade, I always recommend that you do it incrementally. Do not go from one to five. It is very much a coin toss. depending on how much stuff in your application depends on the Django um

package that you will have to pull your hair out trying to figure out what's deprecated what um Needs to be updated. Doing it is in these small batches is the easiest. I've done three of them. It is really easy doing it this way. Um also use the latest supported version for your particularly the Django version, if there is not any um if your current version that you're using isn't compatible. Also, before you start, audit the dependencies that you're using using um because if you're not using something

just remove it if you're not planning on using it or planning removing that functionality from the site such as oh I found a really old payment SD from a payment provider that's we're no longer using. Great, remove it because it may have some real um dependencies that you're gonna have to juggle around and figure out Um that's that's a in a couple slides, but that's something that you should look at. Now in terms of documentation You are gonna want to go to how to upgrade a jjango to a newer version. Seriously, it is it has the Django to uh depreciation uh depreciation location timeline as well as the jingle release notes and other relevant links that may be for your situation

this will probably be the best source when even the jingle six is available the J-Go seven and so on this page will be very central and you can always look at that even when this talk is over like three or four years down the line. Something will be be here there may be another page that replaces it but this is a really good central hub of information for upgrading your Django stuff I also recommend the API docs as well as the development environment documentation. The next one might be Seem weird looking into it, but it will make sense later. Get the Python uh package index and open in the tab.

And for those who need the link, the link is right down here. Um Now let's start with the jingle one the jingle two get the load of the local development environment working. This is something you need to do when you have the app even on the jingle one You want to make sure that everything is working fine before you go to the Django 2. And I'm saying this from experience because when you jump into Django 2 and something's broken, you think, oh, something upgrade broke. it but then you find out oh it was broken into Django 1 I need to fix this um and if you're working in a new development workflow you may not be aware that something's broken until you thoroughly test it.

Um which is why I put that at the beginning. If you have any tests um in manage pie tests or unit tests test um run it make sure it's working as well because again it can be really can Confusing why something's broken only to find out that it was broken before the upgrade. Um Upgrade to a later version of Python for local install and multi-app. Once you get to the Django 2, don't use Python 3. 4 five I think as well as for my particular case I did not use python 3. 5 I used 3.

6 because the later versions of Django 1 actually was compatible So use Python 3. 9 and then stay there for a while because um there is something very interesting in the next slide. The last part I want to mention, in your IDE, there may be a um analyze feature that allows it to quickly look at every file for TeleSense or Air displays. This will tell you if something is missing and more often than not, it is missing references to stuff that was removed. Um yeah.

In the IDE, there may be a button or feature to analyze all the files in the projects. I recommend running this for the Python files and if there's a Jiggle feature. enable it. The reason for it is because it will tell you right off the bat if something is missing. More often than not, it's missing a missing reference error you want to fix that then um I also want to preface that if you do not find anything moving from one to two do not be worried depending on your application there may not be anything to change at this early on let's go to the next slide

I'm sorry, I had to move the camera a bit for this. Uh what is the best way to figure out dependencies? Now upgrading dependencies is the next step that you're going to have to go through when you move to a new python version. I recommend doing it and not to mention that you may start getting errors with the dependency related. There are many different approaches to this. I found two. One of them is this very inconvenient and picky. But if you want to do it like this, you can just upgrade as you go. If you don't encounter it, run the program and see if you get an error, then upgrade it. It really is not ideal. And what back to what I said earlier with analyzing, your analyze feature can also, depending on what version of the IDE you're using or if you're using

what which whatever one you're using, if it's like the free community or the professional, it may have a security option automatically tell you, hey, this this this dependency has a vulnerability. Update it to this. You should use that information as well. The second method you could use is to just upgrade when you move Python versions. I had specified that. For this entire talk, I'm going to tell you what versions you're going to be using. and what ones I recommend? 3. 6. If you're on a version of the Django 1 that is not support 3. 6, try upgrading to uh 1. 2 I believe that is the latest minor version of the Jega 1 that supports Python 3.

6. Then you are going to move to 3. 9 and then 3. 12. You're only going to move to 312 when you're on the Django 5. And I will cover that in the next slide where we talk about um Two, three, four, and five. Uh three two three and four. When you're unsure what version of a you need use the Python package index. It has a field on the releases that says which versions it's compatible with for a release. One other thing to mention real quick as well. Poetry does have an update command

which will look at your Pythomil as well as the projects version of Python that you're using you can go ahead and use that that is something I would recommend if you want an easy way to see if this will just work automatically. I have not used it nearly enough to recommend it is reliable. Use that as your own risk. I have not used Puletry much I'm assuming that it might be the same with any other type of Python package dependency manager but just keep that in mind that is also an option If it's not working, you've tried debugging it, you've tried manually fixing

dependencies or looking at the error messages, you may just end up having to go to the manual method of manually going through the dependencies. dependencies and figuring out what is screwing up here. Or you may end up having to go through a bunch of conflicts if the auto update does not is not able to resolve um a way to update it might say oh there's some conflictions between these packages you may have to end up going through that too and go through a lot of other hassle in which case this following information will help you and i already specified that the analyze feature stuff i'm actually going to show real quick what it looks like on the python package index so then you can see this for yourself.

Okay, so we're on the the Django Python package index index page if you go to release history let's say we're gonna go to the latest version of the Django 2 If you look down here, you will see the Python versions that are supported. It goes all the way to 3. 9. This is very useful for figuring out what version to use and whether dependency is acceptable. Because you may not know, okay, this is the version of dependency I need to update to. Another thing to mention, or point of advice that I have. Check to see when the Django version of whatever you're using was released.

You can see this in the release history section. And then I will look whenever I'm looking for other dependencies. that I need to update I will look for something that is close to that date the major version particular and then I will upgrade to the latest one It is a pretty reliable way to upgrade dependencies. Let's go back to the talk. Testing device. Now if you have unit tests, you are good.

What I would recommend if you are thinking about upgrading a legacy application in the future and you have the resources to write unit tests depending on how big your application is and it uses the Django core very heavily the the Django module. I would recommend putting these unit tests to verify that your code is working because depending on how big your application is is you don't want to have to manually test it. I was very fortunate. There is very few crucial workflows for the main site and the API that had to be tested manually. It was very obvious when something was not working or not. I could very much at a glance log in, check to see if accounts are appearing. Does my license appear?

or any other critical areas like documentation, if they all worked, it was good. That's how I did it, but you may not be able to manually test everything. And writing these unit tests will help you out Now another thing, and this is something for develop um any developer, have someone else test your application. End users can break things in ways that developers never intended to. That is something I have learned in this journey. In this field and having another look of eyes for someone who doesn't really know the application can be a really good tool of and source of information.

Um one last thing. Oh, I said the other thing I want to mention. When you have a finished testing, when you finish testing a version. If you have another staging environment available that is basically similar to where your prod is or is at your prod. I don't know where your stage environments are. if you use something cheaper, I don't know. Um try pushing it to that and seeing if it works. There have been some points where I caught errors that were not showing up in the Docker environment that were showing up in my season environment. I very much recommend this for that reason because it saves time.

And when you're going to do a QA test, or if you have a QA department that does testing, oh, suddenly it's just being pushed right to back to the developer, you it's a good step. It's a good practice. Now we're moving forward to the next range of upgrades for Django, I just want to m mention that upgrading will revolve a lot of repetition. A lot of these steps will be repeated and I will just be telling you, hey, you are going to be repeating these same steps for each version. There is a lot of this talk that that applies. So let's go to 2, 3

and 4. 3-9 is supported from 2 to 4. You do not have to bother with any other dependencies while testing this And you don't have to bother with a different version of Python. It is very nice. This is a nice kind of easy spot in terms of upgrading some things that you may say uh you may see is that the URL method is deprecated you may have to use path or repath as well as the MySQL client version might change from this. You may also have to uh update some the Django dependencies. If you need to uh debug something

, if you need to diagnose something further, enable debug mode. It is very useful. I mentioned that. or you use your remote debugger just make sure that you don't enable it on prod or staging when you're pushing to it because An unauthorized hacker or unauthorized actor could see it and then they see your database credentials. It is very bad. I do not I have experience with that, but it is called all over the the Django documentation. Now you will be using for two, three, and four the steps I mentioned for updating dependencies as well as what I had covered previously. Now let's talk about upgrading from four to five.

Four to five was probably the most difficult out of all of them. For this, we moved the Python 3. 11 and 3. 12. The reason why I say 3. 11 and 3. 12 when 3. 12 is the latest supported version is because for the self-hosted um products most Linux distributions because not everyone wants to run this in a Docker container they may actually run this natively in a server service under via a virtual environment or a uh poetry environment they may not have three twelve a lot of the distributions do not have three 12 yet except maybe Fedora or Ubuntu 24. 04.

Meaning 311 is gonna be the the vast majority of it. So 311 is what we can we use for uh the products. One of the reasons that this has been so difficult and why I say that this is more difficult than the previous version is because it removed the internal time zone methods from inside of Django. That's very easy to change. You just have to start using the the uh Python provided date time slash time zone libraries and you're set Um the more complicated fix was UUIDs were previously stored in 32 character columns. Now starting in MariaDB 10. 7, it's required to use a UID.

I had to fix migrations that were using this in the products I did not see this in the main site or the API. But I I specify this here because it was probably the most Annoying and difficult fix to do. The release notes for the Django 5 outline how to fix it, which I will be showing those sources in a minute because There's some details about how to navigate it that I kind of want to outline real quick. Um and as I mentioned before, we had to change previous migrations. Now real quick, I want to go through the

release notes and the deprecated timeline real quick. The kind of to show you where you should be getting the information about stuff that gets deprecated. Okay, so we're on the how to upgrade the Django versions page and You 'll see that it has a release now deprecation timeline right here. Deprecation timeline will actually show you stuff that is going to be deprecated in the future. as well as what is being removed. Deprecation timeline will just basically say what is being removed. In most cases it will not say how to actually fix it. it so what you need to do to actually figure out how to fix it is go to the relevant release notes it will tell you

for a lot of these fixes what you need to use Sometimes. And I say that with asterisks because I've had to Google things before as well to figure out what replaced it and look at the API docs. For big fixes it you for big effective changes Usually I would have to go to the release notes and it would say it. An example of that let's look for Maria D. Bay here. You'll see now. So what actually happened here is that I'm currently looking in the version 4. 2 release notes and It was deprecated here. It was not actually the requirements where you cannot use 32

character uh character fields was not actually Actually, the limitation where you can't use it was not actually put in place in Forkwood 2. From what I can tell, it wasn't until I moved to 5. 0. I should be looking at the 5. 0 release notes, but I got missed sled because the deprecation timeline linked to 4. 2 so lesson here is that if you're seeing an issue on a different version than what the deprecation timeline links and it's not in that release notes, look at the current version of the jingle that you're using or at least look at the version after the one it links, and then you'll probably find it. Okay, so I just want to preface something. Uh that Maria DB change was not actually shown in here.

Um I had to go to the Django 5 release notes. Just another point of fact. Sometimes the information is not even in the deprecation timeline. You have to go to your specific um major minor version of the release notes. to get the updated information about what needs to be replaced. So for example I am in the Django 5 release notes. If we look for Maria DB 10. 7 regarding UID fields. It mentions it right here and in a lot of cases where it is very much difficult migration it has a dedicated page or a dedicated section rather And it will tell you exactly how to fix it.

Okay, now Okay, now let's talk about the Django migrations. They're how you track changes to your Django models and something during this whole entire journey that I wasn't 100% sure and I did not figure out until Finalizing this talk. Isn't it acceptable to be editing the Django migrations? And it took a while for me to find this documentation. page that said it but the quote is you should rarely if ever need to edit migration files by hand but it is entirely possible to write them manually if you need to some of the more complex operations are not auto-detectable and are only available via a handwritten migration so don't be scared about editing them if you have to

in regards to some of the changes in five I had to edit migrations in order to make it work in order to fix the UUID issue the where newer versions of MariaDB require that data type in the database. table and thankfully some of the migrations that preceded it didn't were not relevant. I just reverted it. And put a check in for that migration to where if it finds an existing install, it will remove it. Now if it doesn't find it, cool. Doesn't need to worry about it. And the reason for that is Because self-hosted customers are going to be

potentially installing this on a new version. They're never going to see that old setup in terms of oh I have a database table using character 32 that can store the UID instead of E UID type. They're never gonna see it. But I also have to take in consideration for existing installs and for updates. Therefore, I had to write a manual migration for this. Your situation may be different. You may not be able to do it that easily. But ultimately you will have to write a migration by hand for some cases to take care of some of these edge cases. Just be careful and test in staging. Test, test, test. Because you do not want cyclic

dependency tendency problems where it's depending on one um migration and it just starts going an endless loop. Now concluding the upgrade, when we got everything updated for the site or for one, the J-Go applicant. I would push it to our PSH slash Upsun testing environment. And I it would then create a one-to-one copy of production. And allow me to test it there to where I had the most accurate idea of what would happen when I pushed this to production. This is a very reasonable and very

good recommendation for you to import what you your your uh data if you can or import some of it so that you can Without a reasonable doubt, try to guess what's gonna happen when you hit in the production. There is sometimes where that won't where something can happen, but ideally you want to represent your testing environment for an upgrade as much as you can and make it similar to your production environments. Ideally one to one. We repeated this process three times and it worked beautifully. When you're doing this, back up your database in full before you merge those changes to production Because if something happens, you want to be able to roll back.

Now I put this in bold. Make sure your debug is off in your settings files. Because the last thing you want. . is your database credentials to be leaked or for someone else to find something and be like, oh, what's this? A database credential login Um in conclusion, we had no issues the point at the main site. As of recording this, we're kind of in the process of packaging the products. And releasing it, but that is still in progress. So we actually have some remaining time here that I wasn't expecting. I'd actually cut some topics from the talk beforehand So I'm just gonna briefly mention it. In terms of changes that we've actually um that was applied to these three applications, the smallest amount of code changes from going the one

the five was a hundred lines that was for the API that just really shows how robust the Django is and that's kind of why at the beginning I said do not Sweat it do not panic because I think for a large majority of the Django applications going through this you will not have that much difficulty on the Django core end. it's probably going to be dependencies and other stuff that you rely on that you may have to update. Now this talk did not cover having to update late can see coding other dependencies that is another step you will have to take in consideration. We were very lucky. The dependencies that we used were very solid on these newer versions. In terms of the maximum amount of code changes, I'd say around probably 500 to

1,000 lines on one of the applications. I think that was actually the main site where we had the most on it but that is something to take in consideration it's just that when you look at the deprecation timeline on the Django there is really not much in retrospect and when I showed that To a non-Python developer, that was the first thing he said. Wow, this does not have a lot of changes in it, and it really does not. It's a testament to how well the Django was built. Now we actually did have a outage during this as well that I cut from this talk. Um we had an API outage that led to some issues while we were working on the the main site.

It wasn't because I was doing the work on it. Uh conveniently just just happened during this. I had to migrate the site from a vendor to using a disk image to platformsh slash upsun. We were not initially planning that at all. Um And matter of fact, it was a we deployed it as the J-Ga 1 because we were still working on stuff that I did not really want to deploy a non-tested app. to production let alone I had not even touched the API yet so that had to be deployed under Jingle One. Um platform SH does have uh 3. 5 Python images 3. 6, uh the Django 1 templates. So

we were able to, again, because of container images, which is fantastic. Um we got it up. It also gave us another avenue for testing. We were able to push once we got the main site to like the Django 2, the Django 3 three each time I would push it up to upsun or platform and we would do that testing. This was not initially planned at all either. Um but yeah that's pretty much the talk. I do want to thank some people. I want to thank the uh PSH and Upsun Devreaux, as well as my colleagues in support. Chris May, who gave a fantastic talk on Monday.

Roger Houston. My family and friends, and also the Django Khan US staff for allowing me to do this talk today. Thank you everyone so much for attending my talk And I hope you all have a fantastic day. You can find me on LinkedIn below. You can also find me on my blog site, which these slides will be released on it. very shortly if you have any questions about anything platform as a service platform SH or Upsun you can reach me at this email. Also my outside of work email is michael at a right. org. Everyone, you have a great day and thank you so much again. Purpose.

No, I'm going to go. So random time and number pick up and then button her window and on them driven and report them on the eyes.