Building and scaling a live event platform with django-channels

Speaker 1: Hi, thank you very much for the introduction. My name is Rafael. I'm a software developer, entrepreneur, and previous co-organizer of this conference. I'm very happy to be back here seeing you all again after a way too long break. It's been really lovely for the first uh one and a half days and I'm looking forward to many interesting conversations in the next days. Today I want to talk to you about building and scaling a live event platform with Django channels. You've probably all heard of Django channels, but looking at the schedule, I'm surprisingly the only one this year talking about it as far as I can see. So I'm gonna do a quick reminder of what Channels is and what it can be used for. Channels is the extension of Django

Speaker 1: into not only the world of asynchronous code, but mostly into the world of protocols other than HTTP. So the architecture of channels is really similar to the one of Django itself, and I'm gonna do a quick comparison what changes if you go from pure Django to Django and channels So as I said, it's about other protocols than HTTP, but mostly it's used for going from HTTP to WebSockets. So very quick reminder what's HTTP? HTTP is when you send a request to a URL with a method and some headers, you get a response. And as it's already spelled out in this example response after that, the connection is closed, it's over, and you start from scratch for the next request.

Speaker 1: The difference to WebSocket is that WebSocket still starts with an HTTP request, and the request asks the server to upgrade the connection. to a WebSocket connection. And the server, if it can do that, responds with, okay, we're switching protocols to WebSocket. uh and some security header stuff that's way too detailed to explain right now. And after that, both the server and the client can send messages messages to each other. And the interesting thing is that they can do so in arbitrary order. So the client can send a message and the server can respond with two messages, or the server can send a message to the client without any request from the client at all. This means this is mostly used for all kinds of websites where you are

Speaker 1: or web applications where the server needs to notify the client about something that has happened On the Python side, this means moving from WSGI to ASGI WSGI is the web server gateway interface, which is basically a translation layer between HTTP and Python. It makes different web servers work with different Python frameworks. So you can use Django and Flask and so on with the same set of web servers. I won't go into detail. It's basically the definition of a function signature. It says you take an environment dictionary and start response callback function and then defines how you pass back the response.

Speaker 1: And there's different WSGI servers, U Whiskey, Gunicorn, Mod WSGI, and so on. In contrast, the asynchronous server gateway interface is a translation layer for More or less arbitrary protocols to Python. Um it's a weird feeling talking about it with the creators sitting in the front row. Hi. It's basically also the definition of a function signature, but I'm not gonna go into in detail. And there's different ASGI servers out there by now, DEFNI, UVCorn, HyperCorn, and so on. On the code level, where we're actually writing our code, we're moving moving from views to consumers.

Speaker 1: So Where we have a Django view that takes a request and returns a response. We now have a consumer, which is a class. That handles different kinds of events. For example, in the simplest case, the WebSocket receive event, and it can do something after it receives something on the WebSocket. For example, send a WebSocket message back. But the interesting thing is that unlike a view, it's not over after we send something back. We can then do something else and later in our code again send something back. So we have more flexibility on the protocol level here. We also move from Django

Speaker 1: URL configs to routers, which is basically one layer higher where we can root incoming requests. based on the protocol and if they're HTTP we're just gonna pass it to the standard Django URL stack and if their web socket we're gonna pass them based on the URL to a specific consumer for example So this is what Channels does in moving us to other protocols, but Channels gives us one more main ingredient that we're gonna need, which is groups. Groups is something that we can subscribe to, we can add a consumer, so we can add A class that is instance that is currently connected to a user to a group. The group has a name

Speaker 1: And we have a name for the consumer we're in that 's not too relevant here right now. And then later we can send messages to the group. For example, um We can send a message to another user by addressing the group name sending the message and every consumer class will have a function called when that happens And to do so, to implement so channels you use a concept that is called the channel layers. Because our Django or Channels code might be running on different servers and they need to pass those messages on in some way so they can communicate with other people who are connected to to other instances And it's used the channel layer.

Speaker 1: I think the only well-known implementation of channel layers is using Redis, but theoretically it could be AMQP or any kind of of message bus. So when you look into channels, the tutorial case, either in the official tutorial or in most other channels introductions, is to build a simple chat server. It's always the one example that is used. So back in twenty twenty, when the pandemic hit and events just like this one were no longer possible to run like we used to before, that's exactly what we did. We built Venuless, a live event platform to make conferences work fully remote. If you attended last year's DjangoCon Europe remotely, you've used it.

Speaker 1: It's basically a combination of a chat server with embedded live streaming, video calling, and other forms of interaction like Q<unk>A's or a well-known applause button. But even though it chat service, the tutorial case for channels, little has been talked about actually doing that in production And when PyCon Australia ran Venulus in 2020, apparently even the creator wasn't yet sure it was a good idea By now we've hosted about 400 events with participation ranging from five fifty to five thousand people each, so we're certainly far away from Amazon scale. But there are some interesting challenges in having 5,000 people interacting with each other at the same time already.

Speaker 1: And I want to talk a little bit about what we've learned on our way here. Before we get started, one more thing. Every piece of code on my slides is highly simplified to fit on the slide. But luckily all source cut of venulus is available on GitHub. So if you want to dive deeper, feel free to So let's start with the architecture of uh of venulus and how we started building this. One of the first things we did is to define a common wire protocol for all messages that we send over the WebSocket connection. When the client application wants to do something it sends over an array with a command A command ID, which is just an integer that we count up with every command, and a

Speaker 1: payload, which can be any JavaScript any JSON encodable type, usually an object of some kind. And the server then either replies with success or with error, the same ID, so we can match the response up to the request, and a payload object again. And when the server wants to tell the client about something that's not in response to a request, which we usually call broadcast because it's usually triggered from somewhere else. The client sends over a two-piece array with a broadcast type and a payload. So the fact that these are arrays and not objects is completely irrelevant. The relevant part is to define early How you encode messages onto the WebSocket and then stick to it

Speaker 1: and only use a very simple protocol in order to have a manageable code base in the end that handles those messages. On the back end, we basically have one big consumer. As a reminder, that's basically our view. So this is weird because in a regular Django application you wouldn't have everything in one view usually But it's all hooked up to the same WebSocket URL that the client connects to, so it's one consumer. And the main thing that this consumer class does is to parse the wire protocol we've just seen. It splits the name of the command at the first dot, and everything before the first dot is what we call a module, which is something invented by us in this architecture or defined by us

Speaker 1: and then it calls a function on the class associated with that module name. So we don't have many consumers, we have many modules. And we've defined a number of decorators to make working with these modules a little easier. Every function that has the command decorator will uh be called when the client sends the code. a command like you've seen before. For example here we have the chat module and this function will be called when the chat dot join command is sent by the client application and then What do we do? We add it the user to channel's group, we add them to a user list, we inform other people that someone joined the chat, and so on. But in the simplest case, we add them to a list. And we've also defined an event decorator that is called

Speaker 1: whenever we receive an event from a channel's group and then call this function often. This function is very simple and just passes on the event from the group to the client connection. Sometimes we need to check if their user has actually enough permissions to get that message, if sometimes we have to reformat the message some way, but often it's that simple. So that's a quick overview of how we structure our code with channels. One other thing when talking about architecture and starting all those scaffolding is to always plan for failure. For example, if our channels, if our Django application goes down, obviously the chat will no longer work, but the video streaming will.

Speaker 1: And we don't w people want to interrupt their video streams and hit reload again and again to reconnect and interrupt their experience. So the front-end application needs to be built in a way that gracefully and automatically reconnects. And handles all those error cases purposefully. This is also very important when we do deployments of our backend code. Sometimes we have to deploy an update while thousands of users are connected. And in this case we replace our application service with the new version one by one. And then we we have users bucketed into small groups using the channels groups feature. And when we send those small groups a please reconnect now message and the

Speaker 1: client application will reconnect so we're having small groups of users reconnecting after uh one after another which will take some time but saves us from everybody reconnecting at the same time because we just took away the old backend And especially the reconnection with all the authentication handling and so on, it's basically the most expensive code path we have in the application. So how do we make sure that our application can handle many simultaneous users? And the first thing when talking about performance is always to test before you optimize To do so, we use K6. io, which is a load testing tool published by Grafana, but there's many like it out there. And in K6, we can model user behavior as JavaScript code.

Speaker 1: For example, in this case we make the assumption that ninety percent of users in a chatroom will never write anything, and the other ten will write one message per minute randomized with a normal distribution but on average. And this is of course a very simple assumption that needs to be validated in some way, but it gives us a starting point to test our software against And we can use K6 to spawn up as many users as we want from using this behavior and see what happens. And while we do so, we can profile our code. For example, using Yapi, which works similarly to the standard library profilers, but is more geared towards multi-threaded

Speaker 1: and asynchronous applications. And another one I really enjoy using is PySpy, which um works a little differently. It can hook into running Python processes. I show you which function the Python process is currently working on and sample that multiple times per second and give you statistics on that PySpy has therefore quite little performance overhead, and it's sometimes really interesting to even run PySpy in production and see what code gets executed a lot. So what did we learn from doing those load tests? And one of the most interesting issues is what I'm going to call the n squared issue in Doing something like chat. Because if every user sends one chat message

Speaker 1: in a room with ten users, that's one hundred WebSocket messages. Because I need to pass 10 messages to ten recipients each. But in a room with 100 users, that's already 10,000 WebSockets messages. Which means that the load of our system is often not defined by how many users we have connected to our system in total, but more important for us is how many users are communicating with each other in the same room. Luckily, social behavior doesn't scale like that. If you have ten times as many people, there won't be ten times as many chat messages. But for other features, this does really apply. For example, if we have ten times as many attendees at a virtual conference, ten

Speaker 1: times as many people will hit the applause button at the end. So how can we deal with that? Um the first thing that we do is really simple is that we keep our broadcast handler code simple and just like you've seen, oftentimes when we get a message from a channels group And this is the code that is executed so many times it often just passes on the same message to the WebSocket. Or does only very simple checks with it. A more complicated thing that we do is we try to deduplicate some events. For example, when hundred people hit the applause button at the same time, there's no use in sending 100 broadcast messages, someone applauded. We would much rather send one message

Speaker 1: that says one hundred people applauded. But this is hard because we don't really want A background worker that has a loop that goes on every second and collects those things because we don't like things that Have additional constraints on high availability. We'd like our Django servers to all be made equal, and if one of them dies Uh nobody cares, another one spawns up. But if we if we do this with a background worker, we would need to make sure that only one of these is running and that it's replaced quickly if it dies and so on So what we do is a trick where when you hit the applause button we set a key in our Redis database and we set it using the set an X command

Speaker 1: Which only creates the key and returns success if the key does not yet exist. And then we increment the count of the count of how many people applaud it. And then we tell our client, okay, we've handled your command, go back doing other things. And then we check if we actually set the new Redis key. And if we did, we are the first client to hit the applause button since the last cycle. If another person now executes the same loop through a command in the client Um the set and x will return false and we will not enter this if branch because it's uh the the the the the tick key already exists.

Speaker 1: And what we do is we send our consumer to sleep for a second, which nobody notices. It might feel a little bit lacky if the same person executes another command just after that, but it's really not usable noticeable in the UI. And then after that second we get the number of applauses from Redis and we delete both keys that we work with. So now we have the number of people that applauded in the last second, and we're sure that this is only executed once for the same number of people. And we can send that information into a group and we're done and the cycle continues automatically the next second if the action is executed. Now, let's talk about the elephant in the room and looking at channels and performance.

Speaker 1: The ORM doesn't do asynchronous yet. People are working on it, but we're not there. Instead, there are workarounds like the database sync to async helper function or decorator, which creates a synchronous thread to run the database queries on. And we've noticed in our profiling the switching between the asynchronous and synchronous makes profiling pretty hard, but we're pretty sure that the overhead of this is quite high. And we avoid using it and we use it as little as possible. Especially we use helper functions to bundle multiple ORM and database calls together in like business logic units and keep it to at most one call per user interaction

Speaker 1: and if possible not use it in broadcast receivers entirely. There's another thing that I'm not going to into detail in this talk to avoid this. We keep cached instances of models we need often. For example, our model of which rooms exist at a conference. We keep them cached in every consumer and whenever there is a change to the model we increment a value in Redis and then all the other consumers can revalidate that they need to Refetch that model from the database. But ideally, if nothing changes while you're connected, we are only asking the database for the room configuration once. And to be successful with this, not only the code needs to work well, but we also need to have the right infrastructure to run it.

Speaker 1: And just the same as with code, we should make sure that we have good information on where the bottlenecks in our infrastructure are before we start optimizing it. So we're adding instrumentation to our servers, to our databases, and to our software itself. We use Prometheus to collect data from different sources and Grafana to visualize it. And adding instrumentation to your software itself can be really, really easy. We use the Stats D protocol in a piece of software that is called Stats D exporter. Stats D exporter receives signals In the stats D with a stats D protocol and then exposes them in the exporter format defined by Prometheus And whenever we want to count something in our software, we just have to add two very simple lines

Speaker 1: with a short piece of what we want to count. And the beautiful thing about the stats D protocol is it that is just one UDP message. And there's little to none overhead in doing so because it just fires it onto the UDP socket and there's no additional logic or threading or whatever um associated with actually sending that. And then you can run your load tests in production. Someone earlier said they'd like to keep it separate. I would if I had a separate setup just for the load tests, but before we use we led actual users onto the platform after we made infrastructure changes. We can use the same load test that we did on our development setup on the production setup

Speaker 1: to see how it behaves. So, what did we learn from the infrastructure and the code combined? The first thing is that we do like channels. We mostly like it because it's so similar to Django in its concepts. It's very well aligned with how we think about software after working with Django for so many years. And we also believe channels technically can scale well because we have never encountered any roadblock. That is based in channels design. However, when compared with other WebSocket solutions, similar with Django itself If you have a lot of users, you're gonna need a lot of CPU power to handle it.

Speaker 1: We've also learned that load testing is really hard. The low tests have all been really useful and insightful for us, but I've never seen a low test that actually resembled reality later on. Because it's so hard to model user behavior to predict how users will interact. So it's really important to also measure with actual users. For those who've played with channels before, you might have seen there's different implementations on how the channel layer is implemented in Redis. The original one is based on Redis sets. There's an newer one that is based on Redis PubSub, we found for groups with many consumers it works a lot better.

Speaker 1: What we've also figured out on the way there is that Redis is single-threaded and can only make use of one CPU core So at some point we introduced sharding by running Redis, not on more servers, which would have been expensive, but just multiple times on the same hardware on different ports, which works absolutely fine. And we did that under pressure shortly before we have the first event, the largest event we ever hosted. And the night before the event, the client called us and said, the chat is broken. If I write a chat message, only some people receive the message. And we were panicking and searching through the code base all night until we found something that is really insightful and I'm not sharing this too

Speaker 1: To talk negatively about the code in there, but because it's very, very something you need to be aware of when you're writing stuff like that, and it's really helpful to know. Because there's a function in the channels Redis project that figures out which shard the channel goes to for the PubSub implementation. And it uses the hash function from the standard library. And you've never heard of that, the hash function is randomized with a new seed every time a new Python process is started. So across our servers, on every server a different sharding scheme was applied. This has of course been long fixed by now, but it's been one of the more interesting parts of this journey The same way we searched for months for this one particular issue

Speaker 1: where after being connected to the system for a while the connection would just drop. And even under no load at all, it would drop almost exactly after fifty minutes. And we searched all our infrastructure for any kind of idle timeout, connection timeout. There was no timeout at fifty minutes, nothing even close. We spent a lot of time testing this from different endpoints and so on. Until we found that our Gunicorn invocation line had the max requests option enabled, which kills a worker after it received 1200 requests. This is incredibly useful with a traditional Django application because it prevents memory leaks. But with the channels application, it will just happily kill a worker that currently has many clients connected. So yeah, get rid of that.

Speaker 1: In the end, we are ending up with the infrastructure where the client interacts with the load balancer. We use HAProxy for that. and then is connected through to one of our Django channels and Django application servers, which communicate with a PostgreSQL cluster and with the sharded Redis cluster in multiple ports. I hope this little story was insightful here and there if you're planning to put a channel's application into production, and at least a little entertaining if you don't If you have any questions, I took a little bit longer than planned, but we might have time for a few of them. Otherwise, please find me in the next days. I'm gonna be here until Saturday. And um yeah, or write to the Discord channel Thank you very much.

Speaker 2: Thank you very much, Rafael. That was an amazing talk. If anybody has any questions, we have microphones on the right of stage and left of stage. Yeah. And uh Hannah can run around to the audience with a microphone. Thank you.

Speaker 3: Um hey. Do you have any insights on when to exactly use web sockets compared to say um server sent events for the receiving part? Like um The chat messages usually come in and then when you send a mess message out, um you just make a separate post request. So Um what was your decision process to go with WebSockets instead of for instance server-send events?

Speaker 1: I have never worked with server-send events. Specifically, I've previously in other projects worked with approaches like long polling or so on. It seemed that WebSocket is the more versatile, more flexible option. Where we can handle all the communication between the client application and the backend through one protocol. So for this application, it doesn't make sense for all applications. Like if I were to introduce one live feature into an existing application, I would probably use it very sparingly just for that. But in our case the entire application is all about things that happen right now And we b need the features of channels for basically every feature and the server-side events for almost everything. So we just handle all everything through that one WebSocket

Speaker 1: for simplicity. I think

Speaker 2: that's time actually. Sorry about that.

Speaker 1: Okay, sure.

Speaker 2: Um if you if anybody has any questions, uh where can they reach you on their Socialism or whatever, but will you be around for other questions in the hallway track?

Speaker 1: Yes, absolutely. Thank you very much.