Django Tasks with Jake Howard

Speaker 1: Ha hello welcome to another episode Django Chat. Podcast on the Django Web Framework. Carlton Gibson joined as ever by Will Vinson. Hello, Will.

Speaker 2: Hey Carlton

Speaker 1: Hello Well. And today we've got Jake Howard, who's uh Django and Wagtail superstar. How are you doing, Jake?

Speaker 3: Yeah, not too bad. Thank you very much.

Speaker 1: Well that's good, good. Thank you for coming on with us. I guess we should dive right into it.

Speaker 3: Yeah, the kind of my like CV my Django CV, as it were, is getting longer and longer and longer as the kind of months tick on. Um, so for people that have never met me before, um, professionally I'm a senior systems engineer at TorchBox. Um I Torchbox are the people who make the Wagtail content management system, which by extension is built on Django. WagTiles use in very, very many places. Um NHS, Mozilla, Google, NASA, loads and loads of people are using it.

Speaker 1: My house use it here.

Speaker 3: Yeah, everyone's using it. Um on the Django side, I'm a Django Software Foundation member, um, but as of About this time last year, I also joined the Django security team dealing with all of the security things. So if you've seen a, oh god, here's a security thing, I need to go and do something, it's probably my fault. Sorry. Um on top of that, I'm a new parent. Um I occasionally speak at conferences and I've been sort of dabbling in the Django ecosystem for the best part of a decade, but sort of more prominently over the last kind of two years or so.

Speaker 1: sort of been leaning into it more heavily.

Speaker 3: Exactly, yeah. It's kind of the last ever since the um the debt fourteen stuff, which we'll get into, is everything's gone from I'm a person who works with Django to like there are other Django developers who know my name, which feels really weird.

Speaker 1: Yeah, because I mean it's not exactly the highest profile thing in the world being a Django developer. Norman.

Speaker 3: As as you well know, yes.

Speaker 1: So go on then. Which do you want to pick first? Because you mentioned TAS, you mentioned the security team, and I want to talk to you about both of those topics.

Speaker 3: I mean the docs got tasks first. Should we start with that?

Speaker 2: Alright. Well, sorry, we're referring to these these docs we use to guide the discussion. So I'll I'll say let's talk about TASS. So one of the public facing things is you gave a talk at DjangoCon Europe in twenty twenty four. Um I suspect that wasn't the start of you thinking about tasks and what we can do about things. Like where did where did it start, this idea of something should change and I should lead it? How'd you get sucked into being the leader?

Speaker 3: Yeah, so how I got sucked into it, I don't entirely remember. That's kind of lost to time a little bit. The the story of where Django tasks came from, sort of it sort of jumped on the scene, I would say, sort of early twenty twenty four was when Carlson popped up and said, Hey, this thing you've been doing in Wagtail, let's make put it in Django and I went, Uh, okay having no idea how to contribute to Django at all. I don't think even at that point I had an account on track, let alone knew the track existed. Um So long long ago um a lot of the debt 14 stuff started in Wagtail. Um Wagtail being a content management system, it does a lot of things when you like click publish. For example, one of the things it might do is go and build a search index.

Speaker 3: It might yell at your CDN to purchase caches. Currently, or at the time of um developing things now 2022 era, when you click publish, all of that stuff happened inside the request-response lifecycle. And that works and that's fine, but as a site scales, it starts to get really, really slow. And so we were looking, okay, we want to shift some things into the background. As any developer knows, when you're trying to like add in task things, you can Google Django background tasks and you'll come up with basically a thousand different options and you've got to weigh them. Wagtail had a slightly more unique problem in that Wagtail is in itself a package, a framework you install. It can't really go, we're going to depend on celery and everyone is going to live with that.

Speaker 3: Because if your maybe your dev team are more familiar with RQ or Q2 or something like that, then you are sort of forcing um a developer's hand as to which framework they want to use. So what we wanted to do was the original start was we want to kind of build something inside Wagtail based on something that we like, but being very Wagtail specific. Early on we realized that's not really gonna work and we are gonna need something that is more sort of an API layer around existing ones where you can sort of um plug and play with them and go okay well Wagtail just has this shim and it knows how to talk with this API layer. When you install it in your project, if you are comfortable using Celery, you just point it at Celery and it just works. If you want to point it at RQ, you change like one or two lines, make sure you've got a Redis instance, and you're good to go.

Speaker 3: Wagtoe itself on the development side of things. doesn't need to care. And that's where this s sort of sparked on. We had some proposals, some ideas, and then we just kind of went, Here's a good idea. And then it kind of sat gathering dust a little bit. There was an RFC open for eighteen months or so But no one ever had the kind of the time, the energy to sit down and go, Okay, I'm actually gonna write this. Um for about eighteen months. And then someone chimed in on Mastodon spotted this thing going, This is really cool. This should not be in just Wagtail. Um and then this Carlton Gibson person who I'd heard of in sort of abstract appeared and went, yeah, you should do that. And I went, okay. And I messaged some people in work going, this Carlton Gibson

Speaker 3: bloke who's really important thinks what I did was really good. I'm going to sink half a day into this. And they went. Okay. And out of that came the sort of the draft draft draft that eventually became dep 14 Shaggot Background Tasks.

Speaker 2: Well that that's a sign of good workplace that they You you say the name Carlton Gibson, it gives permission, but also that, you know, you can you can work on stuff, right? I mean, because that's the thing is that the best things come out of a real world need. So Carlton's Neapolitan. tasks, right? Like it's not academic projects, it's solving, like Django itself, a real world problem.

Speaker 1: And I r I remember that from the other side, right? I remember there's someone mentioning it or Mastodon and seeing it and being like, oh wow, this is really cool because It's not just that you if you go to Google and type in Django background tasks, you can get you know a thousand options. It's that we would discuss this sort of every six months on on you know whatever the social media of the day was. and there'd be a different option for every single Django developer. Now that's kind of fine, but if you're writing a third-party package like Wagtail or any other one, it doesn't really give you much of a target for Do you know what we want to enable background tasks in a um an agnostic way? And what I really liked was that it was like, hey, and what I really like that it's k it's maintained is it was going to be a shim above the different concrete in implementations.

Speaker 3: Yeah, that's that's where that real benefit was. That was that's what kind of lit that light bulb of A, this is a big deal. Like this is this is gonna quote unquote change the Django world as it were, that you've got this single unified contract where particularly for package maintainers, they can go, I can now put stuff in the background and it just works. You look at other ecosystems in like your JavaScript, Go, things like that. A lot of that stuff is built into the language. Python, you can kind of do that if you are like fully native async. You can trivially kick stuff up into the background. But you don't get the same kind of guarantees that you get from something with a proper Q store and multiple workers and things like that.

Speaker 1: Putting

Speaker 3: this kind of common thing sat there just handles everything nicely. And as a developer you don't need to relearn an API if you're pivoting between projects. If you move company and go from using RQ to Celery or vice versa, you don't need to relearn all the different APIs. You go at task. And it kind of does the rest.

Speaker 1: You change it yeah, and you did the back end is sorted out at the deployment time. As well, there was one more thing as as at the time, um, I think they just announced the Solid Q um implementation for Rails And it was another one of these, oh, why haven't we got, you know, what Rails has got moments? We're doomed. And it you know, it i your proposal came along nicely at the same time. And so actually we, you know, there's a there is a c a community idea in play already that if we just advanced would um right but there was a couple of there was a big gap between that first conversation and then the final thing.

Speaker 3: Well, everything takes a lot of work. And I think the problem is, is that for it's something that as a developer you you go, this is great. Django sort of quote unquote moves really slowly, it's solid, it's stable. But what that means is that when you're trying to develop it, it means you have to put in quite a lot of work and quite a lot of pushing to make anything happen. And like I say, with it's all well and good like going, I found a bug, raise a track ticket, progress things through. This is I want to give I want to add I think the final PR ended up being like two and a half thousand lines. I can't really raise a track tick and go, I want to give you two and a half thousand lines that you're going to maintain.

Speaker 1: Well you can, but it's not going to get the same though.

Speaker 3: Natalia Saring Jacob will shoot it down very quickly. Rightly so, to clarify, like things like this need thought they need a kind of an actual review dog. And I know, Carlton, in particular, you've sort of been trying to think quite a lot on where the line sits between a track ticket, the new new features repo, a dep. Maybe we need like some kind of fourth thing and it there is no like definitive line.

Speaker 1: Well, if I'm gonna just squeeze in there, what I think we really need is easier depths is what I think we we really need. But th but I don't think tasks need an easier depth. It's a very thought um, you know full featured mm major scale proposal. It's got its own um, you know, it's Django dot tasks. It's got its own top part of the namespace, right? It's it's a it's a feature of that kind of size and importance. uh you know, the formal debt process was no problem for that. It's where the formal debt process is needed for something which is a bit more than just a track ticket, but

Speaker 2: uh

Speaker 1: the impedance of that full debt Seemed it does seem to be an impedance.

Speaker 3: Yeah, but in in task case, as you say, it it really helped. The original I suspect if I looked back at like that initial commit for depth fourteen, what we have now is massively different to it And there's a good reason for that. What I had then was terrible because I was the only person who really looked at it. Having some insights from people who have either worked with background tasks or have been maintainers of other libraries and have used it. People that have been burnt by other things and go, okay, this library does it in this way. It's not great. Let's try like let's bring it back to Django and do it right.

Speaker 1: So just from that review process and that that maturing process and that brewing process almost, um, can you call out a few people that perhaps you know, not saying that uh other people weren't, but you know, perhaps a few people that stick out in your mind as, oh, that was really helpful. This they you know, said this and that.

Speaker 3: Yeah, well so I've actually put a a dedicated post on my own website kind of calling out the various different people who are involved in the all manner of parts of things, whether it's sort of you, Carlton, being the the shepherd on the deck pushing things forwards. I've got um the actual kind of one of the people who sits very much in the background but was actually really really pivotal in This progressing through was my colleague Tom Usher. He was the one that looked at the Wagtail proposal and went, we can't just implement a new background task thing. We need a generic API. And quite frankly, without that one piece of insight, none of this would happen. What Wagtail would have done is gone, we're gonna make our own thing. We're gonna make you use this. And it would have been fine for Wagtail's use case.

Speaker 3: But it doesn't progress the ecosystem quite as much.

Speaker 1: You had this um in your talk at DjangoCon, you put up the XKCD for the the computing standards thing. And I think the The important thing is not just to add another one that was the same, you know, another option, RQ, Celery and now Jake's one. It's it was that you were offering the interface above to abstract against them that's the the key change, right?

Speaker 3: Yeah, and I think that's that's the other thing as well is by calling it Jake's one. Like, I don't want to be like the BDFL for celery mark two kind of thing. Like

Speaker 1: Yeah, the last thing you need.

Speaker 3: It's it's the last thing I need. In researching for kind of The ways we can implement Dev14 background tasks, working out what the different API layers should be, what the interfaces are you come across a thousand different people that have gone, I've written a background queuing thing. And what they've got works perfectly for their use case. And that's great. One of the nice things is if you've written it, it's easy to maintain, it's easy to bug fix. You can go and work with it. That starts falling apart when you need to build something on the scale of Django itself, the number of edge cases that you have to account for just sort of escalates massively when you're working at A at scale, but also at Django scale. There's extra things like, okay, one of the big ones that really, really stumbled me up was there are lots of these handy things that work really really well on Linux.

Speaker 3: I need Django to work 100% on Windows as well, which means all these nice little handy things are like, oh yeah, it's great, you just add this one line. And my answer is yes. But what about Windows? And it just falls apart then. There are so many things that just don't work nicely on Windows. They might Someone who knows the Win32 APIs can probably come in, point me in the right direction, and we're good. But I'm not a Windows developer. There are I think there is one Windows machine in this house and that's it.

Speaker 1: Uh no, I mean I I remember when I started as a fellow, like n neither neither Marish nor I had a Windows machine available, so the DSF providers would want so that we could at least um sp uh assure that, you know, the test suite was working on giant on Windows and and all the rest of it.

Speaker 3: Yeah. And so the like one of the things that has particularly on speaking of the differences in Windows is one of the kind of the features that isn't in background tasks yet, but will be, is adding the ability to time out a task. So if a task takes more than thirty seconds, kill it. Implementing that, if you look up how to kill Python code, Lots of people have gone, oh, just do this, just do that. But I need solutions that work on Windows, on Linux, that work reliably, that are maintainable, that you can depend on. And it's really, really hard. It got so hard, I actually ended up doing a conference talk on this at PyCon UK back in September about how to kill Python code and all of these. different weird APIs and the this works for 90% of the cases apart from these 10%.

Speaker 3: And you can kind of overlap the Venn diagrams and find two that work nicely. But it's still a real pain.

Speaker 2: Is that that's the side quest you didn't think you wanted to do that emerges?

Speaker 3: Yeah, it was one of those things. I mean, my personality type can end up being quite magpie-ish. It will be a so how would I solve this and I will dive for like a week into dealing with

Speaker 2: Curiosity runner. I can remember.

Speaker 3: Yeah.

Speaker 2: Yeah.

Speaker 3: It can be

Speaker 2: to dilettance, the dilettante curse, right? You know?

Speaker 3: Yeah. It's a really like it's a really fun way of doing things. It's just that I will just dive deep, I'll learn a thing. And after a week I will probably never touch that mileage again. But for that week I'm having a lot of fun.

Speaker 2: No, it's it's in the neural net. It's in the neural net, you know? It's like cause then five years later something comes up and you're like, oh Connection.

Speaker 3: Exact yeah, that's that's my hope is sort of picking up a lot of these little bits of knowledge. And I can go, Oh, I did that like two years ago I should probably get Renova writing some of these notes down, but that's a problem. Well, this is the

Speaker 2: the Simon Wilson for me, right? Like

Speaker 1: Yeah. Blog

Speaker 2: blog for yourself so you don't forget what, you know. I mean I'll I'll sometimes write it like yeah, back back when people read tutorials, I'd type some forget something, type it in, and be like, Oh wow, me three years ago was like really like really new something, you know? Like what have I been doing since then?

Speaker 1: So I want to ask you about um 'Cause uh about tasks and 'cause th when it when it was released, people were a bit confused about the lack of a back end in core and the messaging around that and because we know we've just talked about the why, but you know, I you know, f from the horse's mouth, so to speak. What how would you describe that? Um

Speaker 3: Yeah, the I don't think I did myself any favours when I initially sort of released Django tasks upon the world as it was. Um the initial Django dash tasks release was a PyPI package that contained three things, let's call it The first one is the kind of the API contract, the bits of scaffolding that are needed, the task class, the task result class, those bits, and the bits that deal with switching out backends. That's the stuff that was sort of required the most amount of time in making sure we got it right. The second thing it contained was the development backends, as I called them. The the dummy backend and the immediate backend. both of which are really really useful for local development, unit testing, things like that.

Speaker 3: The dummy deckend doesn't run anything, it just stores the tasks in a list so you can interrogate them in your unit test. the immediate one still behaves like a back end, but it runs your functions immediately rather than spinning them off in the back end. So it means that if you're using a package that um It means if you're using a package that isn't kind of fully tasks aware, or you don't want to go through the effort of deploying things, your code will just still work, but be a bit slower. rather than these really important functions never getting run. Those two things, the the basic API contract and the development things, those were from day one intended to go into Django in the kind of initial push. That sort of two and a half thousand lines of code, that's what we wanted to get in. The third thing that Django dash tasks contained was a database backend.

Speaker 3: This was this required a huge amount of work because as I say, building task systems is really complicated. And that's when I had to build one sort of from scratch on my own. That's where the kind of confusion came because someone looked at Django Task and went, but it's an API contract and it's some dev backends and it's this production backend, but I already have Celery that does all those things. Why do I care?

Speaker 1: Yeah.

Speaker 3: And so I ended up shooting myself in the foot there. In the initial ones, just trying to sort of get things out, it was the right thing at the time. But as time went on, particularly once the code landed in Django Core and was released in six point zero people got really confused. They looked at Django dash dash tasks and went, What's this? Why is some of it not in Django? Why is some of it there? How does this all work?

Speaker 1: And the other part was why doesn't Django have a a an actual back end. Yeah,

Speaker 3: it ended up being some kind of like, why have you only done half the job kind of thing?

Speaker 2: Yeah.

Speaker 3: Like and I'm like, no, what we have here is great. It's really useful. And everyone's like, but it's missing this thing, I don't care. And so what I ended up doing was sort of in what I should have done probably about a year or so prior, was I wrote down my thoughts and went, How can this actually work? How should this work? And then I shot an email to the steering council and went, here's what I think we should do. Am I insane? And conveniently, and probably for the first time in my life. Everyone went, no, Jake, you're not insane.

Speaker 1: We've been thinking that too.

Speaker 3: We've been thinking that too, exactly. And so that kind of that shone this light on, okay, let's sync some time, let's make this much, much clearer. And so, particularly now as of recording, but also probably for the last two months or so, things are a lot more sensible. So to give sort of the outline if you are going, what on earth is this Django tasks, there are realistically three things you need to care about care about. Django. tasks exists in Django 6. 0 forward. It contains two things. It contains the API contracts. built into Django itself and it contains the development backends. Those are there, those are ready to use. If you're building packages that target only 6. 0 forwards use Django dot tasks and pretend nothing else exists. That is the dream, that's what we wanted, and that's what we've achieved.

Speaker 3: The second thing we have is Django Dash tasks. Django Dash tasks is for people that want all those nice things that are in Django six point zero. but can't upgrade yet for whatever reasons. Maybe their company policy is sticking to LTSs. Maybe they're just on a really, really big code base and it takes a huge amount of time to move. Doesn't matter the reasons, I want to support those people and get them using some shiny new things. If you're one of those people, you can pip install Django dash tasks, change a couple of your imports to make sure that you're pulling from Django dash tasks, not Django dot tasks, And you're good to go. The third thing is, as I say, there is that database backend. That is now separate from Django dash tasks. You pip install it separately. It's Django dash tasks DB. You can pip install that.

Speaker 3: And the nice thing is is currently it still has a dependency on Django dash tasks just because I haven't written the bit of code that swapped. between Django and the Shimlayer. Eventually that package will have no dependency on Django dash tasks at all. If you're on Django 6. 0, It'll pull the things it needs from Django. tasks and you can keep using it in Django 6. 0. If you're on, for example, 5. 2, Upip install Django-Tasks. Django dash tasks DB, configure your settings, write your tasks, and you're good to go.

Speaker 1: Perfect. Perfect. And that you've got a reddish Backend as well.

Speaker 3: Yeah, there is also the so one of the things I wanted to do is it's all well and good having 100% of the code written by me. because I can change everything I want to work exactly how I want. But the whole point of this without going power mad is I need to be able to support third party back ends, things that are not written majority by me And so because RQ is the tool I'm most familiar with outside of the stuff I've written, I thought, okay, I'm gonna write an API contract against RQ. And so what also exists is Django -Tasks-RQ. It fits in exactly the same place in the model I just talked about as the database backend, but it means if you want to use Redis and RQ, you just change the one line in your backend config that says

Speaker 3: rather than database, use RQ. none of your you change that one line and nothing else in your code needs to change. You'll go from using the database to using RQ. and you're done. At some point, and sort of this is the kind of the next step is the I've been working on this API contract which shims between Django tasks and RQ. One of those things already exists. It's called Django RQ. It's the shim between Django and RQ. So I've got an issue open against the Django RQ repo to try and get Django Tasks RQ upstreamed into Django RQ. And eventually that means Django TasksRQ can completely go away. And that's the dream. It means if you want to use RQ, you install Django

Speaker 3: RQ and you do the things. You don't need to care about I need this for the caching bits of R of um sorry, I need this for the RQ bit, but because I want to use Django. tars, I need this extra package. Everything is just there.

Speaker 1: Yeah, and that's the dream, right? It's because the Django Q2 could have th their um integrate their back end for Django tasks, you know, the the salary people they can have their their back end and for each option then they're just implementing against the backend rather than you know Exactly.

Speaker 3: Yeah, that's I think the main benefit is that you can go, okay. For for local development, I don't want to run an entire Redis cluster in RQ. I want a database. So I use the database one, it just works. Then when you scale, you change an environment variable that says, actually this is RQ now. give it the various credentials for connecting to Redis and it just works. Your your business logic doesn't change.

Speaker 1: Okay, super. Again, I've just got one more then question. What are the bits of the Django Tasks interface that you think? Oh, we it would be nice if we could grow this or it would be nice if we could um grow that. Or actually I want to keep this exactly at the scope it is. Because an abstraction layer the you know be People always say of the ORM, well it's the lowest common denominator because it doesn't do all the advanced features that you know you might need. Your Oracle can do that or your Postgres can do that. Well it's going to be the same here. There'll be features from individual cues that you can't use in Django

Speaker 3: Task. Yeah, so there are kind of what we've got at the moment is good for like the simple things you need to do. If you need to run a task in the background, that's fine. It'll do that. The things it doesn't do, which as you grow, as you start needing to depend on your tasks, become really important, aren't there yet. And I do want to stress yet. This is not a I've written Django Tas, I'm gonna disappear off into the sunset. Like this is an ongoing effort. Some of the things that don't exist that it would be really nice to exist at some point is gonna be things like retries so you can say okay if this task failed for whatever reason wait a bit try again things like that timeouts as I've already mentioned making sure that okay if this takes too long kill it start again maybe we want to do

Speaker 3: some other really complicated things. Maybe I want to I've got a lot of work to do. I want to run five different tasks, and once all five are finished, I need to run one final one to do a bit of cleanup. That is possible to do in fancy things like celery. It's really hard to do in other languages and frameworks that don't have this as a native concept. You can shoehorn around it by sort of spinning up other tasks and it it doesn't feel very nice.

Speaker 1: It starts to grow into a s sort of fully featured workflow engine at that point.

Speaker 3: Yeah. So this is there is gonna be a fine line. This is I'm sort of channeling my inner like basically going full feature creep of what could this thing do. Because it's it's fun to work out how could these API looks, what could they do? What are the edge cases? That's the the engineering mindset is what can we make it do At some point, someone needs to kind of basically someone needs to be the bad guy and go, no, you can't have that. I think, unfortunately, that bad guy hat meets the spearing council. They have to be the bad guy in some of this. They at least need to be going or at least for at least saying they we can't do it yet That's the main thing is A, it's we it needs more thought. It's probably not big enough quite yet. And I think the big one, which is one that the steering

Speaker 3: council rightly lean on, is the we don't know how important this is. Build it out of core.

Speaker 1: Yeah.

Speaker 3: Once we know it's really important, then we can include it.

Speaker 1: But y yeah, that's absolutely it. Then the flips the other sort of the other point is, well, it's all right building it, but what about maintaining it? Um i you know, of you know, a fully featured workflow engine to take the, you know, the the whole shebang with, you know, dependencies and retries and branching and loops and uh those kind of control flows within uh tasks flow. Well, there are whole products that do that. And are we going to import one of those into Django, even that they're just the API for it? Or, you know, are we going to have the capacity to to to do that properly. Um I think it's much better to have half a product than half asked product, if that makes sense.

Speaker 3: Exactly. Yeah. And so that's that's why I went with I'm gonna build the basic bits, let's get those in, because they're sort of it's much more obvious that these are necessary. It's very obvious that you need some way of getting the return value of passing arguments. That's fairly obvious. Do you need retries? Does everyone need retries? Probably not for certain cases. If you're building sort of a small internal back-end tool or a pet project, you might be alright without it. It would be good if you had it and if it was like one argument away, fantastic. But let's wait until we know the demand is there.

Speaker 1: Go much further than that as well. The vast majority of projects don't operate at scale, that they see failures on a regular basis. The failures only come up when you're running things at scale and then the the the uncommon things become common just because you're doing it a lot. But if you're sending email back as emails out, say, as a you know classic example, you could be sending, you know thousands an hour, which is much more than you know, almost all projects are ever going to send, and you'll never see a failure. So what's the need of retries to say, oh we couldn't we can't have any background tasks because we wouldn't have we couldn't have retries for the the the two percent of projects that are doing things that hit the numbers.

Speaker 3: Yeah. And it's you're then you're having a large amount of code complexity for two percent of users. And for certain things, if that two percent is ten percent it's really useful. If that two percent are a vocal two percent, it might also be useful. At the moment that two percent is a hypothetical two percent, and at that point two percent is basically zero.

Speaker 1: Yeah. Wouldn't it be nice if well yes it would be nice

Speaker 3: Yeah. And again, I think another big thing is this is Django. It's an open source project. If you want it, come and build it. Like, I do need to work out what the contribution process looks like for how do we design retries, how do we design these more complicated bits? That I haven't sat down and sort of discussed with the steering council and worked out where that line sits. But everything else, like, this is where open source wins. If you have a weird bug, a weird jet, a weird edge case, open an issue. Submit a PR. Let's make these things better.

Speaker 1: I'm yeah. I mean I f uh this is a question. I firmly believe that three or four goes in the ecosystem shows the right way forward because you know, people have go one, go two, go three, go four, and then it it becomes clear what the right route is, whereas you could never have seen that without the the four attempts and it might take a while for those four attempts to play out, but the end result is something that's lasting and correct. How does how does that fit your kind of philosophy?

Speaker 3: Yeah, I think so. And there is kind of we do need that kind of falling on the face a bit going, okay, that doesn't work. Let's try again. That's why I kind of put the initial debt fourteen proposal out there. It got sort of I think at the time it had sort of 150 comments on one PR. Like GitHub got really, really slow trying to load that PR. And that's that's what I wanted. I want that. It's all well and good building something based on either my experiences or what I think other people's experiences are. I can't know what other people's experiences actually are unless they tell me. So I make a proposal, people shoot it down, I make a different proposal, we try again.

Speaker 1: Yeah. Yeah. Yeah. Okay, good. Will you been silent for a while, so that's Oh

Speaker 2: no, this is my dream my dream podcast. I always tell Carlton is I say hello and then I say goodbye. And I just let Carlton and the guests carry on.

Speaker 1: So uh was there anything we didn't cover about tasks then, Jake? 'Cause before we move on to a different topic?

Speaker 3: Not not massively. I do want to address one One thing which people occasionally pop up and go, huh, that's weird. Then it takes I end up saying the same thing again and they go, okay, that makes sense The with Django Tasks you create a task, you create a function, and when you call it, you pass arguments into it. With Django Tasks dot tasks, dash tasks, whatever, those arguments must be JSON serializable. There is absolutely no pickling in any part of Django tasks. and it's very, very much by design. I'm sure we'll get into in a second sort of my security team background on things, but it was very much chosen on a let's do things right. And if I can shoehorn people down the don't touch pickle, Jason is fine, then it solves a huge number of problems.

Speaker 3: Anyone who's done weird things and sort of like gone, oh I'll just pass this request succession object into my task. It will work fine on say the immediate backend because it's the same process. But as soon as it kicks off into like it could be on a separate machine, it could be halfway around the world. That like TCP session that is being represented by the request or session doesn't exist anymore because it's on a machine a thousand miles away. As the stuff just falls over. Whereas, again, as you say, Carlton, you want to implement things based on the lowest common denominator. You can do basically everything you need to do with JSON. The big one that comes up is um passing models between tasks. It's a really, really obvious use case, but you can't JSON serialize a model. What you can JSON serialize is

Speaker 3: a primary key.

Speaker 1: How to look up.

Speaker 3: So if you Exactly, just a lookup. So all you do is you pass that through and then in your task you just retrieve it again. That's got a bunch of nice benefits. It means you're not passing around stale data where things might change between enqueuing and running. It means your task um the data that's stored in the task queue is also really, really small. It's a number or a UUID versus an entire fat model with all of its columns. It means you haven't got weirdness of maybe when you loaded that model, you deferred some of the fields maybe you've only got one of these keys, but the task assumes another one exists. So you've got a bunch of weird performance issues. Those kind of things you still you don't have to think about when you're just passing IDs around or you're just passing these like primitive data types. It just works. And so for the kind of

Speaker 3: it seems to be a very vocal minority that are like, why is this not pickle? There are lots of reasons it's not pickle. I've just talked about all of them. And so by forcing you to think about things in JSON, you avoid a bunch of weirdness for a slight little bit of, oh, I need to like grab its primary key and then do a database query to pick it up. Doing that little bit of work for avoiding a bunch of extra issues felt worth it to me. It felt worth it to everyone else who reviewed the depth. And I'm pretty happy with that as well.

Speaker 1: Yeah, no, it's absolutely the right design decision. We see similar things with people working with channels, racing code, that they're like they're passing it trying to pass a model back from a coroutine, it's hang on a second, but that could be crossing a thread boundary because it's crossing a sync to async. Um the boundary and it's like you can't pass models across roads. This is not you know, just don't do that. If you need a model in a a particular place, look it up there and then. So I think that's a great decision. Well just briefly

Speaker 2: Yeah security. And maybe the interesting point here is how Wagtail and and Django overlap there. Maybe you could shed some more light on that, right? Like do things sometimes come in th to Django via Wagtail, vice versa? How does how does that play out?

Speaker 3: So obviously the the poster child for this relationship is Django Tasks, which would not exist without the Wagtail CMS. I'm Wagtail is quite a large project that has quite a large number of installations And so it gets sort of it exercises a lot of edge cases that a standard Django project might not. You're doing a lot of things in Wagtail. The content management system pokes a lot of internal things in the Django internals because it's trying to do some of those things. Some of that works well, sometimes it doesn't work well. And so there are sort of, I suspect that a lot of the um the core Wagtail team have raised a lot of issues upstream towards Django and even to other projects going this weird thing happens, is it a bug?

Speaker 3: Is it a feature? And so that kind of that relationship works really well. We have also had sort of the relationship work the other way, whereby Wagtail goes, We need this sort of thing to work a little bit nicer. Maybe we put it up less as a like, we've hit a bug, more a Let's change this so it works a little bit better and then it can cascade downstream. It's unfortunate that both projects have quite long release cycles. It's not a like We open a ticket and a week later it's available on PyPI ready to go. Like it's gonna take kind of three, six months sometimes to actually be able to use the thing, but by contributing upstream, they can come back down. It's useful to us and hopefully it's useful to someone else as well.

Speaker 1: Yeah, I think so. I mean I've I've been a um Wagtail user for the last few years and I've uh took me a while to get across cost accustomed to the the Wagtail release cadence. It's all good. It's brilliant. But uh it wasn't it was because um Wagtail will use dependencies that are then pinned because incompatibilities and whatnot, and I'm busy trying to update my dependencies and um finding conflicts with what Wagtail had pinned versus what I wanted to use and like how am I gonna work that

Speaker 3: the number of times a a Carlton Gibson has appeared in some discussions going Can you please unpin the upper bound on this dependency, please? I think we're at at least three or four by now.

Speaker 1: Yeah, yeah, yeah. We're getting it, we're getting it. But I've I've got kind of come to terms here. I think it's only um what's the model tree library that you use that uh um

Speaker 3: Uh tree beard.

Speaker 1: Yeah, tree beard that you've got that pin, so that's um but that's fine.

Speaker 2: Okay.

Speaker 1: So the security team, you yeah, you you joined there a year or so ago, you said?

Speaker 3: Yeah. So it was The relationship for how that started I is kind of a long one. Like I've from sort of very, very early in my professional career, two things have stuck out as being particularly interesting, and that's security and performance. And so naturally Django has a security team that kind of do quite a lot of really cool things around maintaining the security of quite a important um software package. There's no kind of it's not obvious how to get involved in that team, how to join that team. And quite frankly, it still isn't. Um that is an ongoing problem people are thinking about how we can deal with that. There are some things happening internally to try and make that a bit clearer. Um I just happened to basically almost be in the right place at the right time.

Speaker 3: Um At the time of this, um Tibokolast, the then president of the Django Software Foundation, had mentioned that the security team was looking for kind of some more people. And I went, I can do some of these things. At the time I was on the security team for Wagtail. All of our kind of anything internal, uh Torxbox security related, generally passes my desk. And so I was like, I have some experience, like am I helpful? And at some point I assume he went away, talked to some people, and I got a message in Discord sort of three or four weeks later going, Would you like to join the security team? And I kind of I had to kind of double taker that message a bit 'cause I was like You want me? Cause it is this kind of it's really weird. Like Django is used by so many different organizations, so many different individuals, and there's a pool of

Speaker 3: what, eight, ten people in charge of making sure it doesn't have massive security holes and I am now one of those people is a really, really weird and humbling feeling, is that like maybe I am kind of good at what I do. Like there is a list on the Django website that has security team and there is my name next to it. And I can point to that and go, I am making this thing better. And that feels pretty damn great.

Speaker 1: Yeah.

Speaker 2: Yeah.

Speaker 1: I like that Can you give us an insight into what the security team does? Like how does it how does it sort of function?

Speaker 3: It's

Speaker 1: it's a mystery, right? It's one it's these seek It's this secret cabal that no one knows anything.

Speaker 2: You gotta wear hoods when you look at the issues that come in.

Speaker 3: Yeah, I mean conveniently none of us are kind of the like the the hoodie up sitting with bright light screen just tapping away at stuff. Like we are human beings. Like I have met the majority of the other people on the team. We are human beings. We're not like supercomputers sitting in the background. The the sort of the start to finish on how things work is if you're if you are sitting for so on a reason listening to my voice at the moment, going, I have a security hole in Wagtail, but I don't know what to do about it, the thing to do is at the time of recording anyway, is email security at DjangoProxject. com. That goes to all the members of the security team. We'll triage it. We'll talk to you. We'll work out is this a bug? How severe is it? Can we reproduce it? Is it like, are you holding it wrong?

Speaker 3: Things like that. All these things that are really important. The security team gets a lot. of emails. Most of them are spam, unfortunately. Um I would say fr on sort of a volume. maybe a third of them are like people legitimately trying to help. What that means is that two thirds of the time when I open an email It's garbage.

Speaker 1: Yeah, and you have to read every one, right?

Speaker 3: And I have to this is the thing that I think people forget is if you're looking at an email from a Nigerian prince, for example. It's pretty obvious I have not actually won a million pounds by having done nothing. If someone goes, I have found a bug in Django Here's a bunch of code, here's some bits of stack trace, here's why it's a thing. It can take a lot of brain power and a lot of time to go No, this is nothing. We get a lot of things varying from you're calling APIs that don't exist or you're calling weird bits of internal stuff. that is not intended to be used in this way or is not publicly accessible, or even a like we've had people report that, for example, when you call cursor. execute and you pass it arbitrary user input, you get SQL injection. And the response is yes.

Speaker 3: If you take a user's input and pass it to the database, that is SQL injection. That is kind of you have to sanitize your user input. And so I'd say sort of nine months or so ago, um a bunch of the security and got together and we sort of added some extra things into our security policies to try and make some of those kind of you're holding it wrong more obvious so that someone can kind of self-police certain things. So for example, you need to make sure that if you're passing random stuff into the database, you should sanitize it. In most cases, the RM will do that for you. If you're passing it into raw SQL stuff, that's on you. You need to be dealing with that. Things like making sure that some of the stuff you're passing through the Django template language.

Speaker 3: If you're passing the entire works of Shakespeare and complain that it's a little bit slow, we can't do much about that. We're limited by certain things. We can make things faster and we're always up for this is a bit slow, I want to make it faster. Absolutely, let's do that. But does it constitute a security issue? Probably not. And finding that line again, unfortunately, there is no if it takes more than like this amount per character or something, it's a security problem. It It doesn't work that way. It's a feeling. It's a, is this too slow? Could this be a problem? If it's a yes, we'll triage things through. Ideally, the person who's reported it has gone, if you change things like this, it gets faster or better. Great. We love people who actually submit um full kind of proof of concept

Speaker 3: or patches, things like that. It makes our lives so much better and so much easier. But some people don't. And we also don't want to put that as a requirement. Some we'd want to know about any vulnerability, even if you've gone, I don't really understand the internal bits. we get reports about some of the weird internals of the RM. I have never dug into the weird internals of the RM, so I go, I can reproduce this, but I have no idea why it's a problem. And there's someone else on the security team that does know the internals of the RM. They can look at it, they know that side of it. I can help by going, this is legitimate, and sort of shield them from the garbage, as it were.

Speaker 1: Yeah.

Speaker 3: And then they can dig into the internals.

Speaker 1: Yeah, that's good. That's good. Um, I guess we need to talk about um AI and the the the the the the effect that that's had on the process and the team.

Speaker 2: This is can I say this is a new record to make it forty seven minutes before the words AI were

Speaker 1: No, we got through the whole episode with Adam without mentioning it once, no? Anyway, anyway, let's not.

Speaker 2: Apologies, Adam.

Speaker 1: Yeah.

Speaker 3: Yeah, there's AI is definitely making it worse. And I think the one of the things that people kind of don't really realize about the reports like this is that AI isn't magically creating more reports. It's not like there are bots out there just submitting stuff. One of the big things is that people will sort of go, I think I have found something. Dump it into an LLM. Most LLMs are intentionally designed to make you feel good about yourself. So they'll go, so if you ask it, is this a vulnerability? They'll go, yes, you're absolutely right. It's definitely a thing. Well done, you can't. Exactly.

Speaker 2: And honestly, you should report it.

Speaker 3: Yeah. It's they will just support that kind of model, which for some people sometimes they need that bit of like the confidence boost. And we're all for that. But then what happens is the LLM will naturally just sort of vomit out a large amount of text And so what goes from a like, if you call this function with this input, it's really slow comes out at about a thousand words. And they'll sort of talk about things like, oh well it affects these versions of Django, and if it's really slow, it could have these knock-on effects. Really at a certain stage we sort of stop caring about all of those details. The things we need to know is if you do this this happens. That's the important bit. If you want to go into the some more bits of details and go, okay, it's really slow, because it's in this part of Django, this has this knock

Speaker 3: on effect that makes this bit particularly slow. All really, really helpful bits of context. But that can be a sentence or two. That doesn't need to be a paragraph of stuff. And that's the bit that kind of really, really slows things down. It goes from I can fit this in like half a screen and know this is all the context I need versus I need to kind of sit and read and think and digest because it's the same amount of information in about ten times the amount of text And unfortunately, that's not something that is just plaguing the security team. I know in the last week or two, um, Natalia in particular has been sort of getting really, really frustrated at the low quality pull requests that have been coming through to Django Repo, where they are kind of a lot of noise for something that is either

Speaker 3: not actually an issue or it's someone going, this could be better, without any justification. And that gets to be a real problem. Um security kind of it's not unique to just us. It's not unique to just security. But it just pads a huge amount of time and effort into reviewing every single report.

Speaker 1: So you know, I stepped down from the security team at the end of last year after eight years, I think, and it was always kind of, yeah, this is maintainable. This is this is something we can do. This is, you know, okay, we're s probably understuffed slightly, but it was fine. It was a it was a a comfortable pace. But the last year or so I think it's just picked up and it was like ev every day, multiple issues, you know, and it's it becomes too much. It becomes you know, you may need twenty people on the team instead of ten and well, we don't have that capacity.

Speaker 3: Yeah, so like I've I've been a bit slow over the last couple of days uh reading 'cause the Django security team, it's shrouded in mystery, but it's also one of the oldest teams that exists in the kind of the Django foundation, I guess, is it runs really, really old. Almost everything is done by email. Um and so I've not looked at my emails for probably almost a week or so. I've got sixteen unred emails in that inbox. Most of them are probably the internal team going, This is garbage. But that means that some of them are the initial report. giving us garbage. And so I need to then sit down and go through sixteen emails worth of following a discussion, working out, is this garbage, is this not? For some cases, I can just go, they've concluded it's garbage. it's garbage.

Speaker 3: Occasionally, and this has happened a couple of times, is we'll look at something and go, actually I don't think this is garbage. This is there could be something here. Nine times out of ten, the person who reported it hasn't seen it that way. They have submitted garbage. But if you look at it with a slightly different angle, you go, okay, well, if they've done this, it's actually this other thing. Unfortunately, because of some of the specifics of things, I can't talk about the like Oh, there's this API that if you call it in this way, it's a massive problem. Um, and also my memory sucks, and so I can't remember most of that detail. But things like this happen. There's a lot of nuance around security of massive frameworks like this because it's Django, the code base itself is gigantic and the security team kind of has remit over literally

Speaker 3: all of the code, but just some of the kind of the symptoms of things. If it's slow, that's a problem. If it's too slow, it's a security problem.

Speaker 1: Yeah. Just one more question then. Jacob Wolves, the fellow, put a post on the um um on the blog recently about some of the recent work in the security team because there was you know, a number of issues that have been patched over the last month, last few months. And some of those were SQL injections. And they were, you know, perhaps um question you have to go very, very wrong. to to do this. But by the they were legitimate SQL injections and by the the the standards there um an SQL injection is um c category high itself. You know, it's a serial severity high issue. And it's it's kind of a this problem where he's like, Well, we've got a release coming up. It is a high issue And then people put on a team to to do it. And then they're like, oh, but we would never just pass this, you know, these keyword

Speaker 1: args to filter in this way. And it's uh And it's kind of difficult. How do how do we balance that when there's so many reports coming in and we're trying to do the best and we've got our procedures and

Speaker 3: yeah, so this reminds me sort of Many, many years ago in the Yang and Thailand, there was the release around the kind of hit a large number of web frameworks around the like uh email issues with like the dotless eye in certain things.

Speaker 1: Really, really famous hijacking.

Speaker 3: The account hijack one, exactly. It was a problem around password reset where emails that as a human aren't the same, the database would look at and go, these are the same, and you can take over accounts. It's been fixed for like eight years now. It's not a problem. But what happened is that affects basically everyone. It had a because it was very well known, the time between there is going to be a security release and it being released upon the world. was expedited. It was one day rather than the standard seven. And so naturally, I, getting that email going, there's a release tomorrow, lost my mind. Because I was like, what the hell is going on? About a month, a month and a half later, there was another email that came around. It was again, it was a security release. It was marked high, but it had the standard seven-day embargo. I was like

Speaker 3: Oh God, is this gonna be another kind of I need to spend a day patching all of my projects? Somewhat conveniently, I happen to be out of the office on the day that um that it was due to be released. So I was there, I was sort of sitting um in the passenger seat of a car, panicking, checking my emails, going, okay, is this a big thing? Am I gonna need to like triage some things in Slack? And it was It was a small SQL injection specifically on Oracle, which if you're using Oracle, is a big deal. We weren't using Oracle, we were using Postgres. And so to us. it was not even a low, it was a complete non -ish. But if you see an email saying there is a security vulnerability, it's rated high, you just see that you have to react based on that. And I don't actually think there's an easy way of fixing that.

Speaker 3: you can't give more you can't give too much information because someone will try and reverse engineer it or work out what it is or try and kind of beat that embargo. We also can't do um we can't sort of sneak little bits of information to certain people. For example, Django has got, I forget the formal name of it, but basically if you are a big enough important organization, you can ask us very, very nicely, and we will tell you the specific details about security release when we send that early warning. rather than when we sort of do the final release. And what that means is that for for certain companies, where it's really, really important they get patched as soon as that information is out. They need a little bit of warning. And we want to accommodate that.

Speaker 3: But we can't give that to everyone. At the time I was working at a small Django agency. Our projects are not big enough and important enough to need that kind of information. But where does that line sit? Who is big enough and important enough? Is it based on The size of the team, the number of developers, is it how important the project is? That's generally the way we've gone. So we're going to do that.

Speaker 2: And do they sponsor Django?

Speaker 1: To give you a concrete example, um the standard people be Linux distributions redistributing Django, right? So if you've It it they need some headtime to be able to get it in that that to patch their version of Django because they're not they're not necessarily shipping Django six point oh, they're shipping five point two point with all the patches and you know, they they need time to be able to update their ri their re um repository. So when you apt get or at update or unattended updates, it automatically pulls in the Django update for you.

Speaker 3: Yeah, those those are the easy examples. You're so we have security context at Debian at Red Hat, they get the emails. There are some organizations that get a similar thing. Um, it's really, really ad hoc. We kind of we don't have rules for it. There are some listed in the docs, but they're not perfect. Um It's very much a if you ask, we'll tell you. In the last three weeks or so, we had two companies come to us who I will not include their names. One we said yes, one we said no. And it just comes up that way sometimes.

Speaker 1: But to come to sort of swing back to the original point, like if you're not using a um an API in the way that's affected, then even a high a a high severity issue is a non-issue.

Speaker 3: I exactly yeah. And there is there is no way around this. Did learn recently in in the Go ecosystem, they have a kind of security release um configuration basically that encodes as part of a security release. Under me release, the issue is in this function if you are calling it in this way. And it's programmatically defined. So it means you can run like basically static analysis on your project and know not only am I using like the standard one, am I using dependencies that have vulnerabilities in, I can also know am I vulnerable to it? And that would be massively helpful in an ecosystem like Python. But the reason you get some of that in Go is it's got static types.

Speaker 3: Python does not have quite so static types, let's call it.

Speaker 1: Oh indeed, yes. Any

Speaker 2: well, I yeah, I've got we have other things in our notes around Self-hosting, like you write quite a lot on your blog, but I do want to respect our roughly one hour time limit. And so that leaves times for books and projects, the important things. Do you I believe you have one queued up, Jake? Is there a book you want to recommend?

Speaker 3: Yeah, I've I wouldn't say I necessarily recommend it. My brain works in very, very weird ways. I've I'm not actually a massive reader, sort of from I would say in the last fifteen years, I have probably read two fiction books at all, and I don't think any. non-fiction books from start to end. I've tried to trick myself into reading. Like if I buy myself a Kindle, I'm kind of playing with some tech, but also reading. It worked for a couple of months and then I got bored. I'm trying to pick that up again. And one of the things that's helped is conveniently at the same time there's sort of been a tech reading group that's formed at work of people that are kind of interested in reading and learning stuff. And so the the first book and the book we're reading at the moment is um designing data driven data intensive applications.

Speaker 3: And so conveniently that's also very well aligned to kind of the systems programming, performance, background tasksy stuff that are kind of interesting to me at the moment. And so I've quite enjoyed I'm a couple of chapters behind the the rest of the group. But kind of slowly catching up, being involved just talking about some of this stuff has been really interesting to me at least.

Speaker 1: Tech reading groups are really good. I'm um had a group with some people from the Janga community reading um Rust in action back a few years ago and it was really good. We'd meet up I think it was might have been been during the pandemic time so it was Nice to have some sort of social interaction as well as, you know, how are we going to get through this book on

Speaker 2: Carlson, you have one uh

Speaker 1: Oh I do, I do. Well I've been um I picked up Back up off the shelf um solar power finance without the jargon by Jenny Chase. Um Jenny Chase is an analyst that works at um Bloomberg and back with oil over $100 a barrel. I thought I'd uh pick it up. It's It says it's about solar power finance. It's sort of more really about solar power. There's a little bit about finance and prices and supply and demand and guts guts and things. But it's a a really interesting book about um how um well about the the the economics of solar fi uh of s solar power and then how um renewable energy has grown and will continue to grow especially you know and um alternatives are inflated due to geopolitical events. So I mean that solar power finance without the jargon.

Speaker 2: Well and I I believe this is more recent. There was just some study showing that solar panels can they thought they had a lifespan that was quite short, but in fact Almost indefinitely for decades they can continue.

Speaker 1: So a lot of the a lot of the estimates that um, you know, it'll take you ten years to the solar panel has a lifetime of ten years and it will take you five years to recoup your initial investment and then the second five years to rec It turns out that the that those estimates were um a lot shorter is exactly as you say.

Speaker 2: Yeah, which is

Speaker 1: a nice uh

Speaker 2: some nice news for a change. I mean one I'll just say one interesting thing as an American with what's happening. Um you know the US we're the largest producer of oil and gas and because of that um We can't make our mind up about what we want. So China, China can't produce enough oil, so it's forced to invest more in alternatives, even though it seems fine to deal with polluting things. Like it just knows it has to. The US doesn't really have to. And so we're much more prey to do we want high prices, do we want low prices? There's a whole discussion around that, but it's a little more complicated in the US, I guess, which is one of the reasons why we're so reticent, at least this year, to uh invest in alternative. I'll just say

Speaker 1: No, it's interesting. I think

Speaker 2: I'll check that out. Um for me quickly, just fiction The passage by Justin Cronin. This is actually a reread for me. I think it's ten years old. Um and this is catnip for me, because he's like a literary author who won a bunch of awards and then decided to write something that people wanted to read. So it's Essentially a vampire book. Um and it's actually a trilogy.

Speaker 1: It's a literary page turner.

Speaker 2: Yeah, right. It's you know, um, it's just really good. I just really like, you know, a plot and to me this, you know, little quick sketches and portraits and It's uh yeah, I need that. So highly recommend it. Um projects, I'll just keep going quickly. So um this is well, as we record, which is March 10th, this will come out a little bit later, but um Andres Karpathi just released a gist uh micro GPT dot py, which is an entire um way to run a GPT in under 200 lines of code. So uh train, inference, tokenizer, autograd, initialize parameters, optimizers and buffers, repeat in sequence, and then do inference. And it's not uh optimized at all for scale, but it's meant as a teaching tool.

Speaker 2: And I've been sort of circling around how do you actually build these things. I recommended recently there's a book, Build Your Own LM from Scratch. This is really great to just see in one file. Oh, this is you know, academically how it's all happening. So I would highly recommend check it out. It's commented. Um very cool. Check.

Speaker 3: Yeah, I mean sort of side projects for me, I'll few and far between in certain places. I found myself sort of originally sort of when I was much younger and had a lot more energy, I would kind of I'd be having thousands and thousands of different things. And slowly my GitHub is slowly filling with these kind of the things I've built. And I'm going, yeah, I'll I'll maintain that. And then I realize, oh, it's been like two years since I've had a commit. That's been not so great. Um One of the things that's kind of particularly in the last week or so that's been on my mind is actually one of the things um that Carlton and Gew's been working on is which I'm gonna hand over to you quite nicely, is around type safety and your project mantle.

Speaker 1: Oh yeah. Well that was gonna be my project of the week. So go on you

Speaker 3: I'm not gonna stew your I'm not gonna stew a thundering light. I'm gonna let you talk about it. But this It's a way I've been writing typed Python code quite a lot, is that by its nature a lot of Python and Django is hard to type. And so writing a kind of type safe layer around an untype safe API is a really, really nice thing. I do want to kind of rein reoperate a little bit in it's a really, really nice idea, but I've also seen it in a couple of other places.

Speaker 1: That's okay.

Speaker 3: Actually, funnily enough, it's also a really, really common model in Rust. Um so I mean at least you can say you're copying this thing from Rust and pulling it into Python, which is uh something.

Speaker 1: Okay, so I mean Let me talk about then. Django Mantle is my project of the week and um also its partner um project Mantle DRF, which is uh DRF um add-on for it, so that you can use it with your existing DRF project now. But what it is is um So it's it builds on atters. So you talk about a a type safe layer. So Jan Django doesn't you got your Django model, you're never gonna make that type safe like and there's some new pep about um can't remember what adaptive typing or can't remember exactly what the the the word is even if that comes in we're not going to be able to retrofit that into the art we're just not like Django is dynamic to the core. It's built on, you know, d um Python's dynamic capabilities and we should just embrace that. But for my work mo code, for m I I want

Speaker 1: to to have like modern Python, modern uh simple, plain, typed Python classes. So I use At as for that. I can talk about why at some other time. So I use map at us for that. And then the challenge isn't writing the atters class, right? Anyone can write a data class. You know, it's got a name and that's a string and it's got an age and that's an int. And you know, those classes aren't hard to write. The problem is mapping that from the RF And so what Mantle provides and it uses Django readers that I've talked about on the show a billion times. I love it. It uses Django readers to m create efficient RM queries from your your outer class. And then the those queries will have just the fields that were in it. And it works with nested data doing prefetches related automatically. And then it serializes that. So you just do a query. You give it a query set and you

Speaker 1: the shape you want and it gives you back typed objects that already did all the fetches and then you can pass them to your um API or you can pass them to your templates or you can add the you know the domain logic that you want and it's all independent from the ORM. And so it's a way of decoupling your business logic or your domain logic from the your ORM models. Your RM models are busy enough already. They're busy do dealing with the relationship to the database. And one of the big problems we have is when we slap on loads and loads and loads of business logic into our models and they become this big, difficult to deal with monstrosity essentially. Where it's a way of m cutting cleanly between Django's dynamic core and then the type safe layer on top. Now the the the bit that Mandel

Speaker 1: Mantle brings is that flow. Atters , not unique to me, not original to me. Katas, the serialization tool I'm using, not original to me. Django readers, not original to me. It's a synthesis of how I've been using those tools over the last you know, few years basically. And Mantle DRF is, as I say, it's a it gives um versions of um DRF's generic views, its mix-ins, its viewsets, so that you can just change your imports and, you know that use a a mantle shape class in your DRF API. So anyway, go check that out. It's on my website.

Speaker 2: We'll put a link into it. And uh

Speaker 1: I'm glad you've been looking at it, Nick. Like you've Yeah. I d I haven't really talked about it yet. It's well

Speaker 3: to the best of my knowledge you talked about it in one place, which happens to be the one place that I Happened to see it, which was the um the Pi T V conference last week.

Speaker 2: Oh yeah. Yeah, good for you. Yeah, yeah. I'll put a link. Yeah

Speaker 3: Yeah.

Speaker 2: World premiere, yeah.

Speaker 3: Exactly, yeah. And conveniently the two talks I happened to see between meetings and other bits was yours and Sarah's, and they were both fantastic. And Mantua in particular was just that kind of it's nice that for a while I found this kind of type safety on top of untype-safed stuff as a really good way of thinking about stuff. You can kind of hide a bunch of weirdness, particularly if you need to kind of do a bunch of weird things to make it type safe, you can hide that stuff in a function and you can just use this public API that make things nice.

Speaker 1: Yeah. Doing

Speaker 3: we do that for like operating system abstractions. Rust does it for safe and unsafe code.

Speaker 1: Yes, yes.

Speaker 3: Why can't we do it for safe and unsafe types? It just felt like Mantua in particular obviously is the Jenga way of doing it. But that that thought process of kind of facading stuff, I think, is the tool that works for developers in general, even outside of Python.

Speaker 1: But it also means that you can make use of Python's dynamic powers when those are of use and you don't have to abandon all that power and leave that power behind. Um and I think that's really important as well. So anyway, and this was like you remember that thread on the forum about Django needs a rice story? And I said, Oh well we need a serialization story. Well that's at us and catters we c But for me, but the point I wanted I I said was, well, it's got to be ORM aware. And that's the that's what Mantle gives you as well, is these is the type safety and the modern serialization story, but it's ORM aware. And that's So anyway, I've just I'm putting it out now, so I'm glad you've seen it all.

Speaker 2: And a and you're gonna be at DjangoCon Europe and PyCon wait, PyCon Italia, do I have that right? Are you going to do that?

Speaker 1: Yeah, yeah, I'm gonna so I'm um Keynoting at um TangoCon Europe, so I'm going to do a long version director's cut talking about, you know, sort of typing and mantling a bit more depth than I was able to do in um and the Pi TV event and then I'm going to give a shorter version again at um PyCon Italia. So if you're anywhere in Europe this spring, uh anything, you can come and see me talk.

Speaker 2: All right. Uh Jake, I think we're almost out of time. Is there anything we didn't ask you or that you want to highlight

Speaker 1: out?

Speaker 2: Oh yeah. All right. Okay. I'll hand it to you. Digitally hand it to you. Here's your magic wand. You can't change tasks. But something else in Django.

Speaker 3: I mean I can change tasks. It would be very weird for me to go, actually I want to rewrite all of this. It's like it's It's terrible.

Speaker 2: Well it's just this uh generally we don't have guests who've actually changed something in Django. So it's just this idea of code, community, just Oh, wouldn't it be nice if we could just change one thing?

Speaker 3: The uh Yeah, I think one of the big ones is the Django call codebase seems to be a little bit allergic to dependencies. That's Possibly a spicy hot cane

Speaker 2: box.

Speaker 3: It's a fact. It's allergic to dependencies. My hot take is I don't think that's good. I think I don't know if I go as far as to say it's bad. But I think if we were less allergic to things, some things could be made a lot better. So, for example, if we I'm not even thinking like on the mass of things like oh we should just depend on Django Restorate and work and be done with and we can solve the rest story once and for all I don't think that actually solves anything. But there are lots of things inside Django where we've had to not so much reinvent the wheel, but we've had to kind of write some stuff ourselves, because we can't use a third party library, or we've had to vendor little bits of it to work for what we need. If we could instead pull in a dependency, that works out great.

Speaker 3: If we can um It's it was really restrictive in Jagger Tasks, because that's meant to be a backport. Jagger tasks has no dependencies. It would be really nice if for certain things I could pull in. various dependencies for other things, but I know that if as part of shipping Django Tass I also shipped five additional dependencies for Django, it would be a real problem. And I don't I don't really know what the solution is to this, but I think sort of possibly the easier way is making a good argument for this dependency in particular brings us a bunch of benefits. um particularly around like some of the like utils and the core HTTP bits of Django. They're not unique to Django. They are used by basically every

Speaker 3: web framework out there, why can't we have we need one unifying implementation to bring them all together. I don't want to start this meme again that basically everything I do is making a unified implementation. But at least if we if there are implementations already out there, maybe we can use them. Maybe we can remove a bunch of stuff. inside Django, maybe we can shrink the code base a bit, shrink the amount of work that the fellows need to do, and use trusted, not just pulling in a random thing that a person has written, third-party tools. ideally as hard dependencies rather than sort of the optional ones we've already got that bring in those extra benefits without making our without giving ourselves a ton of extra work.

Speaker 1: No, that's a really good one. That's a really good one. It's the whole hyper project around HTTP and Python, which, you know, they got some good stuff, good code in there. And Mm. It's trustworthy enough. The you use the word trusted. Right. That's the sort of the the that's the historic counter pushback is why is it very few projects have the stability requirements that Django would need to, you know, pull them in as dependencies. But yeah, those kind of cool cases I think is a real good case. I like that. That's a good good spicy hot take.

Speaker 3: All the best takes are the spiciest.

Speaker 2: All right. Well, Jake, thank you so much for coming on. Thank you for thank you for all the work on tasks. I mean, Carleton was sort of whispering to me, you know, like, oh, there's this thing coming, this thing coming. And it's still coming. I'm glad I'm glad you had a chance to address some of the questions and tasks is not going away and in fact will be easier and more powerful to use going forward.

Speaker 3: Yeah, it's not going away and it's not done either. Yeah. We've done a chunk of it. It's usable today. If you're on 6. 0, use Django tasks. Get on with nice easy background tasks. as time goes on, it'll just get better and better and better. It does also, it's I really hope it's not just me working on this for the next ten years. Like if you are interested in Background house, API design, the kind of the systems programming that you need to do for building background workers. Please sort of appear, contribute to things, get involved in discussions. Like I don't want to have to maintain this entire thing on my own, please.

Speaker 1: Cool to action folks.

Speaker 3: Exactly.

Speaker 2: All right. So we're DjangoChat dot com. We're on YouTube and we'll see everyone next time. Bye bye.

Speaker 1: Bye bye.

Speaker 3: See you later.