Django & Celery: A love story of async proportions with Hugo Bessa

Carbon Hey folks, uh I'm really happy to speak at GenCon US 2024. I'm going to talk a little bit about Django and Celery and their amazing love story, right? My name is Hugo I'm a partner at Vinta Software, where I've been working for the past seven years. Some of you folks might have already crossed with Vinta before uh by using one of our open source packages uh for Django like uh Django React boilerplate, uh Django World Permissions, uh DRF read and write serializers uh Django virtual models in the newest one uh which is really

fresh uh which is uh Django AI assistant you should really have a look uh We are a company that really cares about uh the open source world and especially the Python community. We contribute in uh with a lot of uh open source software and also uh we are we've been present in most conferences all over the world so uh that's something we really love and we've been uh been there for a while uh already so uh about myself uh I've been working with Django for the past 10 years in very different kinds of projects, most of them using celery, and that's what we're going to talk about today.

So To start this talk, I'd like to talk a little bit about Django. Django is almost 20 years old and its philosophy has uh made many projects very successful so far. What I really love about Django is it has batteries included uh there is kind of the right way of doing things like authentication, authorization, interacting with data. security things like that uh Chengui's school of good opinions on all these topics so and But that basically allows us to work together and make a whole ecosystem uh in an integrated way, uh built on top of Django, uh like

using the same base. And that's because of that we have tons of packages uh that are really useful. We have events, we have meetups, uh, we have and especially There is an active development in the main framework which introduces exciting new features on every new version. But there's a known problem of Django that couldn't still be fixed. And that's like Jingle performance, right? Jingle uses Python, which is not known for being a fast language, like compared to Rust or C or Go. And on top of that, Shango

wasn't built uh to work with multi-thread. Uh its asynchronous features are still uh being developed. actively. They are still being still maturing. The RAM itself can be misleading sometimes with regard to performance. But there are workarounds to make Django a bit faster. In the database side, you can avoid N plus ones, for instance, like being more cautious on the way you write queries You can use caching, you can make smart database indexes, you can use data normalization, you can run operations in the background.

All these techniques could make our application run smoothly and serve a ton of users simultaneously. That has already been done in the past by big companies. But I want to talk a little bit more about the last one running operations in the background. Before that, let's have a look on how Django works in the in the uh in the in the background right uh basically django exposes a ws gi interface uh or web server gateway interface. Uh that's what a server like Unicorn or WISGI uses To run a Django process, right?

They kind of connected that W WSGI file in your Django project. These servers themselves have a request load balancer. Basically, they res this this uh part of the the server receives a lot of requests and they map these requests to uh to processes they have right so each uh Django process has like the whole Django environment loaded there in memory. Each of these processes speaks a request at a time uh and only gets the next one when the previews has finished being processed, right? Uh so long running requests are kind of expensive, right? Each Django process loads

the whole framework core, right? Uh so it also uh it's expensive to have too many processes right because all of them have like a a cost to to be loaded right uh And requests hold a Django process each while being processed. So there's no way to run multiple requests at the same in the same process, right? There's no multi-thread. uh in in Django processes uh usually right there there might be workarounds but uh we're focused on the on the main thing today. Why so why running operations in background, right? Requests should be processed quickly so we don't have to roll the process for too long.

We want to be able to process the next request. as quick as possible. Otherwise the queue is going to be too long. And we also want to give feedback to the user as quick as possible as well So we can retain their attention, right? Attention is very valuable in these days. So we don't want like the user to switch tab while they wait. uh for an operation he he started on our app right we want to keep him in our tab and like maintain uh retain his attention uh their attention in in our in our uh platform right uh and salary it's a tool that can help with that right

uh salary is a an a an asynchronous task queue or job queue queue uh which is based on a distributed distributed message passing right so it's what the the most important word here is distributed right uh it separates a syntax execution from your main application. So it doesn't run in the same process that Django runs. It's a completely separate process that uh uh just worries about these as asynchronous tasks. Uh it is also very fast And it's very well integrated with many tools, right? Including Django, right? So with Salary you can have access to your models, to your services, your functions, your classes.

and all that jazz, right? Uh and celery loves Django, right? Uh celery basically has uh dedicated documentation for integrating with Django. Actually there are ways to configure salary using Django settings with no extra boilerplate. Django uh is it's allowed to be accessed within tasks, so you have full access to your RM and your managers and your uh query sets and uh your services, classes, etc. And there is also an important community around uh Celery and Django building packages that like enhance these integrations. uh

that uses Django database uh for for managing some of uh salary things uh and things like that so there's a whole ecosystem that uh helps us integrate them both right And uh here's how it looks like to integrate Django and Celery. This is like a very basic use case, so I'm just showing you how it can be done, how quick you can integrate them. Basically here I'm writing this process order view. Right, it receives a request, it creates an order object based on the user, and it calculates it kind of call the calculate user score task. So suppose you have like the uh this

this function called calculate user score that is very slow uh but it isn't that important that you run it immediately after you create an order uh it can have some eventual lack of synchrony uh and so that's the perfect use case for us to move it mute move this task move this function to another process, right? To running background because it's just going to bloat the request, right? So in in here We are moving it to a task function that receives the user ID, it picks the user from the database, and it actually calculates the user score. But uh before all that happened we actually give a response to the user.

So this delay uh method here it's just going to add the tasks to the queue it's not going to execute it uh immediately right it's going to be executed in the background uh so this is kind of a hello world for for salary right uh And what this can give us basically, what async task can be used for. You can delegate long-lasting jobs like the one that you uh we we've just uh seen. Uh We uh we can execute remote API calls like APIs can fail, right? Uh so we can use like salary to run to like kind rap kind of wrap these API calls uh and

give like have retries running without blocking the request, right? Uh we can also prepare and cache values right uh to make queries uh a bit uh uh lighter right uh we could spread book database insertions over time for instance like to not to avoid overloading our database uh we can execute recurring jobs things like that right there's many use cases for uh async tasks right uh and how does it work with salary basically django uses a salary client uh that gives kind of that that delay function for instance uh and it basically queues a message to the broker a broker could be

many platforms, right? Many, many uh uh kind of uh Q uh Q managers right it could be like reptinq uh sqs uh it could be uh read ready 's itself right uh So we can use multiple uh queue manager uh as a broker uh but like uh we we choose one and like when whenever we call this client uh Django is going to send a message to this broker and put it on the queue uh and celery is going to be watching this queue and uh It will mark a task as started. It kind of depends on your settings there. It may or may not mark as

started, it may just start it. But uh and it marks it X succeeded after it finishes uh processing it. So the the second part is like after salary finishes uh the the processing of that message of that task uh it will also deliver the result of that task to a kind of database. We call it results backend uh and it's basically storing task results. So uh you can query these results with uh other tasks or even with Django. You may want to wait for a task to run to get its result. In in the case that we showed may not make a lot of sense, but there are use cases for that.

If we're just sending an email with salary for instance you might not even want to to wait for the result so the result backhand will kind of be useless in that situation but there are situations where it can be used Right. So when you configure seller, you actually have to give give it a results backend uh so it can store the results there. Uh it's just a database, a normal database. Uh So uh but like n it's n it's a good love story, right? uh Django and celery but it's not always rainbows and butterflies uh we can say it's a tough love right uh There are many things that can go wrong. Whenever you're introducing a distributed system into your application, it makes things a lot more complex, right?

Like all connections may fail. There are a bunch of connections now. You have Django to the to the broker, the broker to salary, the salary to the results backend, the results backend to Django. So there are many connections that may fail, right? You also have a lot of concurrency being added there, adding a lot of complexity. So uh we can have like tests that never finish, outdated data, um issues that only happen in production. It's it's kind of Introduces a lot of noise, right? Let's say salary has a strong personality, right? Salary is a distributed system, which means there are many points of failure

Right? And here we are going to focus up on these four problems that I've been through in the past in my my ex whole experience with salary, these things always have uh caught me uh in the past and I have to handle it uh correctly. The first one is outdated data, right? Sending complex data as parameters to salary may result in unexpected stuff, right? In this example I'm giving here uh we are passing the user model to a seller task right and we are basically calculating the score here and saving the user And why this is kind of tricky? Why is is it might not work as expected? Because the user model may change

uh between the salary task being scheduled and it actually being executed. And as we are passing a model object Uh the mod the model may be outdated. I can even have deleted that user. Uh if the let let's suppose the task queue is sh is is full, right? It has a lot of tasks to be executed there uh the user may have been deleted and whenever you run this the user may not uh exist anymore Right? So you calculate the score of an exit and an existent user. In this other case here, the right way to do this is basically passing a reference instead, right I'm passing the ID of the user.

I'm getting the user by the ID. If the user doesn't exist, I just don't run the task anymore right uh but if i can find the user i'll get the the up to date object right and i will calculate the the user score accordingly So this is the first thing. You have to be very careful to what you pass to the salary tasks as parameters, right? Usually you should rely on references, not on like complex objects, right? Serialization is necessary. If under the hood by default it uses pickle which can actually pass like complex objects through But that's uh far away from

ideal. You shouldn't rely on like passing complex objects, you should like pass references and remount that these objects on the task itself. Okay The next problem is duplicate runs, right? Depending on your seller setup and task configuration, it may not be guaranteed that your tasks are only run, are only going to run ones, right? Multiple workers uh may pick the same task um at the same time, for instance, right? Uh you have multiple workers looking at the same queue, both of them both uh can get the same task at the same time. uh and uh this this could happen depending on your on your configuration uh a task may be interrupted uh and requeued for instance

right if you like a worker fails and re the machine restarts, uh you may have a task that was interrupted and was requeued and will be executed again. So you have to ensure that your tasks are atomic and eidepontent. Right? In this case that I'm showing here, this example, we're storing a reference to the latest order we processed in the user model. Right, you can see in the line uh 25 whenever we update the user score on the line 24 uh after it we just store the the latest order id uh and we save the user, right? We save the user. Uh that means uh

if we call this task again with the same user and no new uh order we're just going to pick again this this last order ID right uh and And we are not we are going to exclude that user with a reference to that order ID. So if they if there is a new Order we are going to recalculate. But if the order is the same as the last one, uh we are not going to do anything. We are going to just return early, like in line 22 Right, we are not going to be able to pick the user and we are going to return early. So this function it also has this transaction atomic decorator so if something fails here or if this function

stops uh in the middle uh it will just roll back everything right roll uh everything uh back and will not l leave any any uh like data uh uh updated uh not not completely right uh so the other problem we can see uh in salary uh uh salary projects is uh a little bit more complexity on error feedback right because whenever you pass the bannon on to to a salary worker If it fails there, you may not be waiting anymore in Django. So you may not be able to return an error result to your user.

So user may receive a success message from the request but the home operation may still fail, right? So whenever we are developing a flow that like relies on th sync stuff and async stuff We need to take this possibility into consideration and give like a feedback, not like a success feedback immediately to the user, but a feedback saying that this is still being processed uh and do some sort of polling to check if this was completely uh successful or not. Uh you may also need to be able to undo operations if partial it it has partially happened uh synchronously and the other part is happening in asynchronous in

in background, uh you may need to undo this first part that happened synchronously. So uh it adds a lot of complexity for error feedback and error handling in general you might you you need to take that into consideration when developing uh features that will touch both sync and async at the same time, right? And especially if it involves user feedback. Okay Uh so uh there's this uh fourth problem which is conflicting operations. Why uh uh right so while a task haven't still run another operation uh is triggered by the user for instance right this operation conflicts with the one that's still pending

uh and creates like uh a an estate that is not uh it's not predicted right it's not predictable so For instance, let's consider these use cases. User can add nodes to our system, right? User can also delete a node And user can book create copies of a note. So you can choose a note and create a bunch of nodes based on that existing node, right? So let's look at this following flow I have on the right So let's suppose the user starts creating a new node. It creates then triggers the creation of 10 copies of the node This will run in background because

creating 10 copies may be a little bit longer. We don't want the user to be waiting on that. So uh it happens in the background. This task is kind of queued, it goes through queue and it's it will be executed by a salary worker. Uh Then the user deletes the original note before the task to copy the 10 notes run. Right? The the tasks the task runs, but the original note isn't available anymore, so it cannot be copied. Right? You cannot like copy something that don't even exist anymore in the database Right? There are many solutions to this. Like you can implement, for instance, soft delete on the note. So whenever you delete a note, you don't actually delete it from the database, you just mark it as deleted.

Uh but you can still uh find it on the database uh and you can st you are still able to create these copies. Uh you can also cancel uh all painting tasks before deleting so whenever I try to delete something some note uh a query for painting tasks around that node that uses that will do something to to that node and cancel all of them uh all the painting ones uh and after that uh we can like other like other alternative is just locking nodes with painting operations so whenever we want to do something uh on the on an existing node we before creating the task we are going to mark this node as like

locked And whenever we try to delete the locked note, we'll get an error, for instance, because it's still locked, so you have to wait for it to be unlocked before you can do other operations with it. So this these are alternatives for these conflicting operations. This is just an example. This can be a lot more complex depending on your use case, but like uh async May create a lot of issues. You cannot hike like have this just do a rollback, a database rollback. You don't have like atomic operations. So There's it's hard to to have like atomicity between the sync part and the async part. So uh basically you have to handle it by uh storing state

or even uh canceling painting stuff right uh Now let's talk a little bit about how to make things a little bit smoother with celery in Django. There are ways to work around some of these issues that we talked about, right? There's some tips that can help, like we're calling couples therapy here. So some tips to help the relationship to flow between Django and Celery. So if you're going to retry tasks for instance, you need to use exponential backups, right? You don't want to like give a fixed number of seconds or milliseconds between the retries Because for instance, if you are making a request to an API, this API

probably won't be back. If it fails, the request fails, it probably won't be back. uh within a a fixed number of of seconds or milliseconds, right? You probably want to wait a bit more between the latest calls Right, so um it's important to keep uh your your backups exponential between retries, right? Uh Also, tasks shouldn't raise exceptions the same way like views don't uh uh are not allowed to raise exceptions You if you run an exception in a Django view, uh it will raise a 500 error and the user won't know what happened, right? Uh so The same thing applies to salary. If you raise an exception, it will just fail

in a in a terrible way, right? In an uncontrolled way. So ideally you should handle all exceptions. Uh and if there is nothing to be done in the code for like adjusting uh the state because of that exception, uh you can just like send a report through email or through a monitoring tool Something like that, right? So exceptions should not be unhandled uh in salary tasks the same way they should not be unhandled in Django views, right? Monitoring is also essential, right? Like salary adds a lot of complexity. You need to uh kind of be able to see this complexity, uh see how it's going. So

uh there's celery flour this is a package that can help right it can help you see the current state of uh your your Stellary setup, right? You can see the running tasks, you can see the tasks that have already run. Uh so it can help a little bit for you to understand what's going on. There's also this flag called Always Eager. You can do it by task or you can do it globally for all salary tasks. And this is very useful for development, right? Whenever this flag is active The tasks uh kind of run synchronously. So whenever you call a task, it will run immediately instead of running in another process, right? It will just run, uh which makes debugging a lot easier.

uh easier right uh if you don't want if you are actually uh wanting to use uh the remote thing having a different process for salary RDB could be your best friend, right? Uh RDB is basically a remote debugger. So whenever you you set up a breakpoint it will stop the execution there and it will open a connection so you can uh have a terminal connected to it with uh telnet And you can actually send like debug signals. You can check like the variables values. You can like send a next signal or a continuous signal. things like that you can debug the whole thing even with remote

uh remote workers that are not in the same process as your django django application Right? So RDB could be very helpful. It's kind of a built-in thing, so you can import it from salary library, right? It's available within salary uh you don't have to install anything external uh so it's it could be like uh a really good friend for you it could really help uh The second part of the couples therapy, I would like to mention a good tip for long tasks, right? Long test dumb may not work exactly as expected with salary, right? Because like Imagine that you were uh the

the manager of uh this task. Like you cannot know if the task has for instance uh Frozen, right? It may have frozen, it will not give any result ever. It's just like locked in there. So in that case, you probably want to have a timeout, right? So this is configurable, but you probably don't want tasks to run that long like hours. You don't want that So uh in that scenario you probably want to split the task into smaller ones uh to make sure that uh you have like a sense of progress, right? Uh and you you can know the things are running and there that there's no time out like

uh killing your task while they are still running, right? Another thing about celery is that monitoring tools, monitoring tools are very limited. I talked before about uh celery uh flour Right or flower. I don't know how you pronounce it. I think it's flower salary flower by flow, right? So Uh you may need to implement some monitoring yourself. Like for instance, I had in the past to implement like Q 's heartbeats, right? To know If my queues are well balanced, I have multiple queues uh for different sorts of tasks, right? Uh and some cues were like being very full and other queues were being very empty. So uh

If a queue is full, having a heartbeat may be helpful because if you're scheduling a new task, it may take like too long for it to run. So Uh this is something you might have to implement yourself. I had to implement myself in the past, right? But like there are some paid monitoring tools that might also help in there right like uh things like new relic or a data dog may help a lot in monitoring your your salary tasks and uh uh uh having uh useful logs and understanding the state and all that jazz right uh Another thing is that salary is an excellent tool for asynchronous tasks and for doing simple jobs, but like for complex workflows

It might not very be very reliable, right? It may be uh it has a lot of open issues and many complaints of lost tasks and unpredictable behaviors, but when you're running very complex stuff. So it may not be the the best tool for that. But don't worry, like there the celery and jangles relationship is no monogamous. Like these tools Other tools can live together with salary, right? Uh for instance you could use salary for doing simple stuff Because it has like very little boilerplate, it's very easy to use, it's uh the the the learning curve is very little uh but you could use like temporal IO

temporal IO to to run like uh more complex stuff. Uh it has like a more robust infrastructure. Uh it's kind of done like created to run these complex workflows. And even include like better monitoring tools like built-in that you can rerun tests and do stuff like that. So for very complex stuff, you may want to look into other tools for that, right? But if you're dealing with simple stuff that needs to run async, uh Celery could be your best friend, right? So other than that, we also have this that checklist site. Vinta has put that that up, but it's an open source project as well.

uh it has a bunch of uh dev checklists and there's one specifically about salary uh like my my uh co-worker Felipe Shimenez has put this up uh and it has a lot of best practices you should follow when uh uh uh writing uh s integrating Django salary So basically you should have a look at that. It can be very useful if we're starting a new application or if you're configuring a new integration Right? And that's it folks. Thank you for hearing me. Hope you you enjoyed the presentation and I'll probably be on the on the uh event channel if you want to talk about uh celery or django or anything uh related

uh I'll probably be there if you want to talk just ping me okay uh this is my email if you want to uh uh talk about it uh as well and that's it. Thank you