Green Coding with Django

Speaker 1: Yeah, good morning everybody. I'm here to talk about green coding in Django. So first off talk about a bit about myself, so my GitHub handle AQ30. You can see that. Just before the image over here to your left I'm quite proud that I have Scottish heritage, so it's an honour really to be able to give this talk here in Scotland. For anyone wondering about the kilt, it's a Clanitalia kilt to a Scottish Italian heritage. I'm a developer. I've specialized in Django for seven or eight years now, maybe longer. Developer much longer than that, probably 10 to 12 years. Lose count

Speaker 1: I like walking, charity walks, and cycling too, so do quite a lot of uh uh activities. uh for charity. And then you see the the the beautiful team there in the middle. This is the uh early team for Django CMS version four. It's uh a new version of Django CMS. There's not really been advertised uh uh massively just yet, but uh it's starting to be pushed now by the Django CMS Association by uh Fabin as the fellow who who uh um uh invited me onto the straight the stage there. I also have quite a bit of interest in uh electrics and uh Raspberry Pis and Arduinos and just just generally just tinkering uh with with electronics So today I'm going to talk to you about what green coding is,

Speaker 1: why we should care, what we can do. And any questions at the end, please. So just before I start, can I have a show of hands of anybody that actually knows what green coding is? Oh that's cool. So I'll ask you at the end and I expect to see hands everywhere, please. So what is green coding? So it's a fairly new term Uh the first article that I could see coined I think was 2018. Uh it's an environmentally sustainable computing practice, so this means writing environmentally friendly code, green code, is not Code that is green, as in the color green, it's environmentally friendly code. So it's split out into two considerations and that is behavioral

Speaker 1: and structural considerations, and we can explore now what those are So for behavioral, it's how we actually create the code. So us as humans, we need to eat, we need to drink. We also get hot or cold, so we need to cool ourselves or may hot, we need to heat ourselves or may cold Uh we we also need lighting to be able to see. So I'm not proposing here that we turn all the lights down. So we're working in the dark. That's not good for your eyes, that we have needs. So there's certain things that we have to do to be able to be healthy. um how we actually uh travel to and from. So uh I'd I'd practice what I preach here. I I made sure that I got the train here and I'm also gonna be getting the train back on Saturday. Anyone that knows There are no trains on Saturday, so

Speaker 1: I'll enjoy that I'm sure. So there's also our personal habits. So these are going to be unique to everybody. They're going to be things that you do that others don't. They might be things that we all do. And there's also the structural side, which is actually the code that we're writing it, the actual code that we're writing, the hardware it's running on, and the kind of in infrastructure to make that happen. So why should we even care? Why are you even telling me about this? Well, unless you've been in a living under a rock for the last decade There's clearly a problem with our environment. We've heard it a little bit recently. And there's obviously something that has to change, and that's mainly because

Speaker 1: We have uh now a sixty-six percent chance that the uh the the globe is gonna uh is gonna warm by one point five degrees by twenty twenty-seven. So I don't know if this is new news to you, but This was actually a lot earlier um than it was originally uh projected. It was 20 th 2035, 2030, 2035. So it's now getting closer and closer. So it's it's worse. So whatever we thought climate change was before is much worse. It's getting worse. So There's also an interesting stat that I found when I was researching this is that the data center usage is 1% of global emissions. Now I couldn't find or had the time to make a chart, which would have been beautiful at this point to actually show you how that represent

Speaker 1: how that relates to everything else. But if you think about all the cars on the roads, the transport, our heating, the oil industry So manufacturing of steel is a very uh pollutant, uh very heavy polluting um process. So as individuals we have a responsibility to make sure that we can fix and and try and avoid uh the the damage and effects of climate change So sorry Python fans, but it's not a very efficient language. I'm not proposing here that we all just run off and leave Django behind, Python and Django, but I think there are certain things that we can do to try and help and make the the the

Speaker 1: the the use of Python um efficient. I think we all have a job to do, we all have things that we have to do on a daily basis. It's just making sure that when we're doing that, we're doing it as efficiently as possible. So what can we actually do to fix the the behavioural side? So this isn't the code, as we remember, this is actually what we we need to be able to create that code as individuals. So we can use our machines more efficiently. So making sure that if we have unused applications in the background or or we have uh Uh we're not shutting down regularly so we just have lots of uh unused things uh lying around in the background that we that we clean up So I'm sure that it's a bit noisy this this image, but I'm sure that we all know somebody or we are that person that has all of the tabs open.

Speaker 1: If you were to zoom out, you would see all the apps running in the background. It's just like really bad And something that was interesting, I left the machine alone for a bit just to see what was happening, and that's what these charts are over the top. You can see that the CPUs are bit active. And this was only the browser open, by the way. You can see the CPUs have been a bit active You can also see uh memory usage in the middle, which is uh we can see that we're using swap, which is pretty bad. You'll see why in a second. And then there's also a bit of network usage going on. So I left this for a while just to see how the machine was behaving. uh with with the tabs open. So that's quite inefficient. So k uh manage your tabs, manage your applications better. So we can understand a bit about why uh the activity was happening by looking at this chart. So we've got the the CPU at the top, so I hope people

Speaker 1: kind of have an an idea about the the hardware uh inside a machine. So this is kind of this is this is the same for your phone, for your laptop, for servers. Any machine has So this is pretty much the same across all devices So the the CPU and the RAM is the kind of the cheapest. The way that things happen is via electricity. I can take take you back to early days, I think one of the first years of college I ever had And somebody, the the the IT teacher explained to me that all that the whole of a machine is just a series of switches and electricity going through those switches. And that is actually true. So transistors, uh a PC uh uh uh

Speaker 1: uh hardware is made out of just switches. So it's just logic gate switching through. I don't want to bore you too much about that So as we get further through this chart, as we're coming down through the different areas of the RAM, uh through the the various chip components, we're having to use electricity to move that that information backwards and forwards Now that this explains why machines can get slower by the fact that when we run out of RAM, we have to then start going to the hard drive, which you can see is further down in the middle. of the chart. So we're having to use more electricity and more chips to get backwards and forwards. And we can also see this gets worse if we go out through to the network and through to the internet. that where once you get to that cloud, you're then interfacing with other machines that work the same as this, that have CPU, RAM, etc.

Speaker 1: And that just goes down the chain, down the chain through switches, through routers, through through to the to through to our servers. So the less we can do through this this electricity network, the more efficient things become. So what else can we do? So the the equipment that we buy, we uh some interesting statistics here that we can't actually uh uh use the CO2 and and actually CO2 is a bad uh metric to be using because it varies all the time. The the the calculation of doing it is not really a science, it's just more a bit wishy-washy and magic Interesting facts that the average laptop consumes 214 kilograms. I think it's just useful just to try and picture what that

Speaker 1: what that means. When we're buying hardware, make sure that we're buying TCO certified. That that stands for total cost of ownership. We have to do that. responsibility so go and look on this website see uh if you're buying equipment that is that equipment listed also when we're disposing The WE stands for Waste Electrical and Electronic Equipment Recycling. It's actually an EU and UK law, so you you shouldn't throw hardware in the bin. It has to be recycled responsibly, so make sure you do that. I also love the idea of sending old equipment over to different parts of the world. uh Samo that had uh uh uh a need for hardware so I think that's a great use instead of throwing it away.

Speaker 1: Because also for total cost of ownership, it's also the fact that end-to-end you're having to manufacture it, then you're having to use it, then you're having to destroy it. Which is really bad. So these are a couple of reports that I found that I thought were really interesting. I'm not a Mac fan, as you'll see, I don't have a Mac, but I thought actually what they're doing is actually brilliant. They have an environmental report for every single device that they manufacture. And you can see here it breaks down like this report carries on, so I'm not going to bore you with all of it. but it's really interesting how they're making sure to source all of the equipment, uh all of the um uh uh materials in the device from um from from from recycled material and also trying to tackle as part of the manufacturing process.

Speaker 1: The other point to the right there, to the environmental um uh uh image that was from Nenovo so these are things they're trying to do to make themselves uh greener Yeah, to a bit more of what we can do and that's assess our personal habits , try and think about what we're doing that's that's potentially not very environmentally friendly. So I've seen we have a lot of coffee cups here, right? So I've also been uh guilty of this because I forgot my reusable coffee cup at home, but we should be using uh reusable coffee cups and not especially not with the plastic lids Another website here that's great for not just equipment but for everyday things that we're that we're doing and buying and that's Ethical Consumer.

Speaker 1: Again, that's just individual to everybody and then talk about things. There might be certain things that we're doing as individuals that we didn't realise that we're doing that other people might find interesting. So make sure to share that on So here's a few things that I identified about myself that I started then trying to fix. So I I I'm remote, I'm a remote freelance developer. I've been working remotely since COVID. uh and and never went back into the office. So there's a lot of things that I found I was doing inefficiently that I improved. So I was heating the whole house in the winter. So uh just to just to be working the office. My my wife is a uh a matron. Uh she's uh basically

Speaker 1: at the uh at the hospital uh and uh every day in office hours so it was just me in the house heating the whole house. Uh I had uh as we saw the many used applications so all of the the tabs making sure that I was closing them And then I found that I wasn't shutting down, so I just had loads of unused things living through the weeks. So I'm actually much more efficient as a worker. by cleaning out and starting from scratch every morning. Uh every every Monday morning, sorry. So I start with a fresh head and I'm not leaving behind old problems. So I'm I'm always making sure that I that I leave a to-do list so I'm not just ignoring and and forgetting that that there 's things to be done. And then I also found that I was just drinking too much. I was drinking so much coffee that it was actually uh really inefficient. But I didn't want to change that

Speaker 1: So I I tried to find a way of being able to drink all the coffee that I wanted, but be as efficient as possible. So here I had an energy plug, uh uh In fact I wanna I wanna just just point on actually why um uh I even thought about uh using the water that using boiling water as a as a metric Years ago I had uh a shed and I put a solar panel on the roof and I I had some batteries from my next door neighbor. He had like a uh a mobility scooter and he had some old batteries so I hooked all this this uh setup uh in the shed and then it got cold, winter came and I was how do I heat the shed that's got no electricity supply in the winter? So I started trying to use the battery bank and the solar panel.

Speaker 1: And I realized quickly that I used all of the power at the battery bank. And I had no heat at all, so it's like wow, this is actually really expensive to as as an energy to heat water. And we can see here uh using an energy plug that I I plugged into the kettle that Uh my kettle was using uh 2. 8 kilowatts, so that's the the first image on the left. And then I boiled uh the minimum that I needed for uh a coffee and then waited till the end and I could see that that had used 40 watts. And then I went for what probably what I did, a guesstimate of just half filling the kettle before, and I found that I was using 100 watts. So there's a a bit of a ugly chart here that I'll just go through quickly, try not to spend too much time on it. But basically the the we just saw there the the quantities of water is the

Speaker 1: the first row. The second row is the power that we've just seen. And then we could see from the power per day that that was 160 watts for the 350 mil and 750 watts for the uh sorry, 400 watts for the 750 mil. So I could also then see what that was for the year by working out that we have forty-eight weeks for the year uh across five days gives these metrics of thirty-eight point four kilowatts and ninety-six kilowatts. So here's quite a uh a pretty chart of over a year and then over five years how much power had been saved. So it's uh quite a dramatic amount when we look at the five years, being two hundred uh this is kilowatts by the way, this is not watts uh versus 500. So over the one year

Speaker 1: uh that just gave a a figure of money just to see how much we could save in the year. But then also the energy, which I think was a much more, much more interesting If we were to take a Tesla and put that power into a Tesla saved over a year, we could do 200 miles and that would take us from Edinburgh to Glasgow. A return trip twice. That's quite far. Now over five years, it's much more interesting at 288 kilowatts. That's an insane amount of electricity just wasted. So the the average home I found uses 8 kilowatts , and that would power that home for 36 days, which is Think about that now, thirty-six days for water, it's over five over five years of savings, that's quite a long time.

Speaker 1: But thirty-six days of electricity, full usage, that's your wash machine, uh uh wash machine, cooking, cleaning end to end uh as quite an insane amount anyway. So then now we can move on to what we can do structurally. So this is our code, this is not now our behaviors, but actually the code that we're writing, the hardware it's running on. Found a really interesting piece from the Django documentation. That most performance problems in well-written Django sites aren't at the execu at the Python execution level, but rather inefficient database querying, caching, and templates Now that makes total sense if you remember the the diagram we had earlier of the internals of the PC. Now we can see that in a in a wider

Speaker 1: picture of the things that we work with every day So you've got the clients interfacing across the internet, hitting your webcache, so that's uh Amazon, um uh cloud front and uh as your uh CDN and then you have your web server and then that's interfacing with the DB and we can see that the chain gets further and further away and all of that has uh electricity it has its uh its o its own setup. uh sorry, its own uh compute power in each of those services. So how do we normally find problems in our code? Well, we normally find problems by bugs. So somebody raises a bug and you have to question at this point of the project, sorry, at this point of the problem being found, how long has that code been running in the project

Speaker 1: It could have been running for a very long time. Running efficient uh inefficient inefficiently, using uh resources inefficiently and costing money actually. So how do we find issues before they come bucks? Well, seems quite obvious really, doesn't it? We just have to look for them But you have to question I'm not gonna get a show of hands because I think everybody would have to show their hands because we all look for problems, right? We don't just ignore issues But I I think a lot of the time we do actually ignore issues. We know they're there. I think companies don't want to pay for the time of a developer to fix these inefficiencies. which I think that has to change. That's not right. That can't that can't continue because it's just wasteful. So the the solution to most of these problems is to throw hardware at it.

Speaker 1: Uh we've all been there. We've all been in the situation where actually we've just gone, actually this is easier. uh on me to do this. So sorry So how can we actually look? What are the tools that we have to look for these issues? So I think uh uh for a lot of this might be just obvious to most people, but there are people here that might not know this. So Please bear with me. I think it's very use it hopefully very useful for them. So we have profiling tools that analyze the performance by time and resources, so it's measuring how long something took. to to complete. So C Profile, Django Silk, the built-in Python Time It module, and there's also a new

Speaker 1: Perth profiler that is used in C Python currently but it's being exposed out uh in 3. 12 to us as developers uh in Python. And then we also have I think what most people have uh most familiar with the uh request and response profile in with the Django T debug toolbar, and there's also the uh colo application which uh Lily uh at the Colo store will uh show you give you a demonstration of and talk to you all about that. And then there's also how we find issues in production, and that's via uh Datadog and Sentry. So that gives us all of the above information. uh about how our applications are performing in production. So you can look back through here

Speaker 1: and try and understand uh and find inefficient parts of our uh projects. So it's just a quick quick sheet a quick screenshot of the Django debug toolbar here for the SQL view. So this is just showing us uh the time it took for queries to run. And really what we're looking for here is any slow queries that are taking a long time or many duplicates. The Django debug toolbar does a good job at highlighting uh duplicate queries that will need to be fixed. We can look at how to fix those in a minute. And there's also the the kind of the time profile of how long scripts took, sorry, how long a request took. uh to complete and it gives a bit of a a detailed information. I think um uh uh uh code tracing quite gets quite interesting here. I don't think the the G

Speaker 1: the Django debug toolbar does a very good job of that, but there are other tools for that such as Colo. uh and data dog that will actually let you walk through uh the and actually step through the the the the the code line by line to see um how long things took to complete. So by fixing issues in our code, we actually become better developers. In fact, I think by finding issues in our code, we become better developers. And the reason for that is because we learn new things. We learn how to fix issues. We then learn a new pro a new a problem that we might not have known before, and then a new fix for that problem. So I'm just going to go through now a list of problems and then some quick fixes to to hopefully fix

Speaker 1: those issues. So here, the cheapest of them all, somebody else has done all of the work for us. In fact, hopefully after all of this, you will go away and fix all the little issues in your packages and then they become releases. So then we have uh the just a couple of commands here to show uh outdated packages and then the the fix is just then to read any change logs uh that that mention performance and patch them. There's just a few more uh uh a few more items to to walk you through. Uh first start off just by running through how I how I actually uh uh tested uh uh and and gave some of the the um the results. So I just created a quick time snippet, tried to run with an empty database where possible.

Speaker 1: Also tried to keep data to a minimum because that would just be wasteful. I actually contradict that in a couple of them I I was annoyed at myself afterwards, but it would have been wasteful to then run it again with smaller data sets. Um for any average times that were really small, I ran it three times just to just to just a show. So here we can see the the time snippet that I mentioned. I could have used uh uh I have used Django debug toolbar in some of the other um uh some future examples and also the I could have used C profile or the Python time it module. So I'll just show you some some modules quickly We've got a question model and the choice model. This is from the Django polls tutorial. And then now we're going to look at slow queries.

Speaker 1: I think most people are going to yawn, select N plus one again, and I'm just going to run through it quickly because there might be some people that don't know it. But basically all it is is it's an issue where you can have many duplicate queries and it's normally caused by looping or nestig looping It's common in the Django admin. It's also very common in Django the Django Rest framework with serializers if you're just using them straight out of the box. And as mentioned, any custom logic where we're nested we're nesting, looping through nested relationships. So a a quick takeaway from this is for reverse lockups use the prefetch related and for forward lookups use select related. So for reverse the foreign key is in the other model. For forward the foreign key is in the current model

Speaker 1: So here we're just looping through some questions and some choices. Not really doing anything with it. Just want to show you that this is what it is. We're looping. And we can fix this by simply adding the prefetch related flag or the select related flag to the to the to the source query to make looping easier. So I just did a quick Stats set here just to see how this was going and the uh for 25,000 records um which was made up of 5,000 questions with each with five choices nested Without prefretch related, you see that it's a shocking 50 seconds, and then as soon as we add prefret related, we get that down to one second. And you'll also see another interesting stat here of the uh 5,000 queries versus the uh two

Speaker 1: uh two th two queries. And that's because of uh every time we were looping, we were having to do an extra query for each loop. So every this gets worse as you nest even further. You make the problem much worse. The Django debug toolbar will solve this for you. And this Uh uh this article from Scott Stafford is from 2015. I was first using Django Framework I think in 2016-2017 and I had no idea that I'd completely violated every rule uh of the uh Select N plus one issue. Reading this article actually brings forward a concept of lazy loading. That it doesn't actually exist in the Django framework and it's just something that we can do to make sure that every time we nest and every time we use a serializer, the optimization is already there.

Speaker 1: So I'd highly recommend uh reading that article. It's still valid today and I will still go and reference it if I'm uh ever unsure. Uh yeah. There's also bulk operations for anyone that's not aware whenever you create, update or delete things. There are ways to speed it up. Every time you create, update, or delete on a set, you will do a query for each one of those operations. We have bulk actions, so bulk create, bulk update exist internally. We can use them to speed up. We can also create a custom bulk delete from raw delete. I don't know if many people know about that It does void extra things, so you don't normally get Django signals, but it does void other other things of checks

Speaker 1: on the database. So be very careful if you do use that. But there are ways to delete bull there are ways to speed up deleting as well. So just a quick point here, use book create use book create over uh sorry, yeah, use book create over um uh create well the bulk actions over standard actions. So again using the 25 ,000 records that I'd already created, didn't want to waste creating them again, so I just used what I had. We can see here that with create sorry, this is where I created them, the select M plus one was earlier. So this is where I'd made the mistake and then I just kept that mistake and used the other stats from it. So it took a a an age with create and we can see with Bolt Create it was less than two seconds, which is an insane speed

Speaker 1: up. We would normally use that for testing, so for performance testing, loading a database, and also for data migrations if we're moving data around. Use Bulk Create where possible And there's also uh even for smaller sets, we can still see a speed up, even with smaller sets. Uh marginal, but still there's a speed up. So I tend to try and use bulk the bulk actions for most operations these days And then we also have indexing. I I think uh you as newbies people don't tend not to use indexing so much. Uh just a a quick I'm gonna have to go quicker because I'm running out of time. So we hear the choice uh choice model from our uh tutorials And then we have when we're creating, uh

Speaker 1: I just created a tho uh 10,000 um to be able to test this because I need to test from the middle. We loop a thousand times to test the index and we can see that when we don't have an index, it's 26 seconds, and then with the index it's marginal. So it's it just that that exposes the uh the need uh uh and use case for for uh indexing. And there's also another thing that I didn't realize I would always create autocomplete in the Django admin myself. So the select all from question was what I found, and I found it from The drop down of when you have a foreign key on your model in the Django admin. Inlines also do this. And the simple fix is to add the autocomplete

Speaker 1: field and that will fix that issue. For any slow views, make sure that we cache. And I think sadly that I'm gonna miss the most important thing. So I'm gonna point you off. So uh something that actually uh pushed me towards um uh fixing all my test suites. It was a point from uh Camilla uh Ma uh Mae Mae Maya sorry um from DjangoCon last year that I thought was really interesting. So We will just skip through, suppose quite a bit. It's a shame because there was a lot to touch on still. So you might be thinking after all this that premature optimization is the root of all evil. That could be the case, but what I'm proposing here is actually to fix the issues in your code and make it more efficient.

Speaker 1: And then it's not then perimeter optimization, it's actually just making it efficient. So we've we've spoke about uh what green coding is, why we should care, what we can do. I was gonna talk about some takeaways, uh but I think I've massively run out of time. Can I talk Yeah, thank you. Um so just takeaways. So there may be things here that I've missed. Uh I would have loved to have gone through Factory Boy. There was some some um speed-ups of Factory Boy of using traits basically make sure that you use traits instead of uh creating uh instead of creating a model with subfactories all the time, only you create subfactories when you need them You can use traits for that.

Speaker 1: I was going to just walk you through the speed-ups of those traits and then just give you a takeaway just to keep up with what we're doing, make make your code efficient. And thank you.

Speaker 2: Thank you, Andrew. Thanks a lot. Great applause. I assume we'll be available for questions and um I from my own experience can tell you Andrew's a great guy to learn from, so ask him questions please. Yeah, thanks a lot. One more sorry.

Speaker 1: Sorry, I just I've just completely missed one thing. There was a a point missed in the there was a point raised in the first talk about working from home and not not socializing, not getting out much. You may have noticed on the t-shirt Django Social. Um there's John at Foxy Talent uh with uh Liam and Josh. Make sure you go and speak to them. They run the Django Social Group in London and that's just to try and get people out for mental health and just to make sure that pe like-minded people are uh are meeting regularly. Thank you.

Speaker 2: Thank you.