Building real time applications with Django

Speaker 1: The next speaker is Jacoba Spalletti. And he's going to talk to us about well, we all know the old web, like the you click and then Even if you're Germany it's still the thing where y the page loads and loads and doesn't load and doesn't load and you may get internet and Maybe you get to the point where your website actually loads. And he's talking to us about application or web applications that load in real time. Wonder how that works in Germany Hi

Speaker 2: everyone, I'm Jacopo Spaletti and I'm CTO and founder of Nefila and we build stuff with Python basically and I've been using Django since 2009. Montainer of a few uh Django applications and one of the co-developer of Django CMS. I'm very thrilled to be on stage for my first time as a speaker at Django in Europe. And I'm scared of talking about channels w with Andrew in uh in the room. But yeah. So uh For some time now, uh web has evolved into being much more than just a click request response uh cycle of the good old days. and the richness and complexity of modern web application has grown and most of them requires a lot of different tools to build uh

Speaker 2: to be built and run. This is awesome as the little nerd in us is eager to learn new shiny tools. But you end up hitting a complexity wall in uh in limiting the amount of stuff you can throw at an application to run before it starts to crumble There are already a lot of great Python frameworks to work with the synchronous web and the asynchronous communication, but would it be nice to have the Django way to do this this stuff? So channels. And channels has been created to go beyond the request, the HTTP request response. without losing the the Django awesomeness. It's not part of Django framework but is still maintained by the by the Django project and is a framework built to deal with all the asynchronous protocols so it's not limited to to WebSockets

Speaker 2: and you can use any asynchronous protocol with with channels. And but we will assume WebSockets for the rest of the talk, which is also part of the channel score. And so channels 2 is the shiny, shiniest, the newest uh version which is just a few months old It changed a lot of things from the Channels 1 version which served us uh served us for uh a lot of years, and I will assume uh version 2 for the rest of the talk. The talk will not assume any previous knowledge of channels. Probably Andrew will go more into the details of channels. I'll explain so I'll explain the basic concepts. And a start

Speaker 2: might be boring, but I found it very important to understand them to actually being able to write sensible applications, sense sensible channels application. I wanted this talk to be very actionable, very practical. So we will have a sample application we will see together and I will drill down into the code to show the intricacies or the pitfalls of of channels of building channels application. So let's see some concepts. The first one is asynchronous. What what means asynchronous? In this context, let's say that asynchronous means event-driven. So if some event happens and our application responds to it. For example, a client connector with a socket endpoint

Speaker 2: and a function is called, or an object is created in a database and another function is called. And this is a big change from the usual synchronous HTTP world. If you are interested in how Django works in the synchronous world, have a look at uh Jacob Kaplan must talk at Jungle 100hood 2015 about the whole request response cycle which is very uh it's been very interesting and insightful. A problem with the synchronous code is that it's generally more complex to understand and write because it's non because of its nonlinear uh nature. And uh so it it's it's in uh

Speaker 2: uh yeah it's more complex. And channels one decided to hide the most of this of its complexity While channel still embraces it and exposes the asynchronous core, uh you can still write uh synchronous code, but you can switch to to the synchronous uh if you if you need One of the important concepts and base of channels is the EasGI protocol. which has been uh developed together with channels but is actually broader than than channels and provide an implementation uh independent specification of an asynchronous interface is basically has the same role as WISGI interface in the asynchronous world.

Speaker 2: And one of big change in channels too is the SGI all the way down approach. Every building blocks of channels is an SGI application, so you can Mix them, compose and pipeline basically them to create your own uh pipeline and your own application The first building block of channels is the protocol server, which is the one that uh implements the SGI specification for a specific protocol. So you have a protocol server for WebSockets, you can have another one for I don't know some IoT protocol or whatever. And uh

Speaker 2: the protocol server is the part of a software that interacts with with the connection, with the network or with the external world. Uh for channels has Daphne as the reference implementation for a protocol server which speaks HTTP WebSockets, but yeah, you can have any kind of protocol server and you can uh have many of them to speak different protocols. Behind the protocol server there is a routing because when you have the connection established you need to route this connection somewhere. And so it pops incoming messages to consumers. And again, being routers

Speaker 2: uh SGI application, they can be nested and composed together, and we will see some examples later You may have different protocols which will have different kind of routing. So you will have uh basically routers according to the different protocol server you have. Scope is also one of the big changes in channels too. Every time a connection is established, a scope is created. and all the information regarding the connection, all the information the protocol server wants to deliver to you are put into into the scope. So you have the connection and then the scope is created and the application instance is provided this the scope

Speaker 2: But what's and uh channel. Uh channel used to be basically the the building block of channels one version in channels two it changed a lot its uh its nature. It used to be the the basic uh a communication system b uh between the the the protocol server and the rest of of the application. In channels too is basically an IPC mechanism for the diff for consumers to to speak to to each other. And it's where events basically happen. As channels one is a first in, first out, at most ones queue And then we have the consumers, which are the main abstraction of building your SGI applications.

Speaker 2: It handles events that happen on a connection or in a channel layer. And the big change is that consumers are stateful. So whenever a connection is established, an instance is instantiated and the connection and the instance will live together. And so uh uh each connection is managed by a single consumer instance. Um group also changed a lot uh in in version two uh they're now basically just labels attached to the to the consumer to group them into into sensible uh sets of of consumers instances. Events are the base concept of synchronous programming

Speaker 2: and they are triggered along the the lifetime of our instance basically. They can come from the from the connection like when a connection is established or a user send a message or they can come from the channel layer where you something uh put a message in the channel layer and uh and the consumer will will respond because in the end consumers consume events Uh and last but not least, uh WebSockets is mostly used uh consumed by by JavaScript. You are not limited to it. Python implementation of WebSocket client or any language actually. But if you need JavaScript channels, ships

Speaker 2: a very lightweight library to make easier to use. uh web sockets in the in the browser. So a brief recap. Channels do expose the asynchronous uh uh its asynchronous interface the uh it and it uses asynco so it only python 3 and up uh only but and but this allows you to write synchronous and asynchronous code in uh in in channels so you are m much more uh free to do your own implementation. It also changed the way it uh handles the the connection because now everything runs in the same

Speaker 2: uh in the same process. So Django and the consumers run in the same process. This allows to make consumers stateful along the other advantages. Then you have middlewares which are similar to the whisky ones and as I said the SGI all the way down approach which makes much more flexible in building application But okay, enough theory. Let's see some action. And this is a sample application I I built for this for this talk. It's a very simple one. It's simplified version of a of an application we actually built is not present one hundred percent of the channel's features but is is a good subset to start with. Uh what's the features of this application?

Speaker 2: Let's call it Poor Folks Google Docs because it can count active users It can tell if other users are reading or editing a document, and you have browser notification when something happens in the into the application. Application logic is very very coarse and very race condition prone is just yeah just to expose the the channels uh features not not to how to build a real application. Uh as I don't want to sacrifice kittens to the live demo god, I prepare a video. And I will explain as it runs. So here on the right you have Lila with the dashboard open and on the left you will have Fry

Speaker 2: At the moment Lila is the only one on the dashboard. You see maybe is a little not very readable, but you have a one here Fries login, there is the notification permission check, but the users are now two and you have these green labels on the each document that says that nobody hit as doing nothing with it. Now prize has opened the document so Lila knows that prizing because the label is yellow. And the fries name is on is there, then fries open the form and the label turns red with the fry name. Fry saves the document and that's the that's the

Speaker 2: notification. Lila clicks on it, the document opens. And Fry 's uh and on Fry 's dashboard the the the label turns uh yellow. So it's it's very basic and and stupid, so to speak, but it allows to to see a lot of different things regarding channels. These are the three parts of channels you are going to touch when you build your own application. So channel layers, routing, and consumers Channel layers is basically where uh the is the backend storage of uh uh of the messages that runs between the between the consumers. So you

Speaker 2: basically have just to configure it and use it one of the uh existing channel layer. You can build your own. This is one of the default ones. And you have the SGI application setting which will tell the protocol server where to send the messages. I configured routing but this as uh everything is an SGI application I can put any SGI application here, not necessarily a routing. But what's the routing? Routing is similar to to the Django Euroconf and uh so you have uh um A lot of different SGI applications here that route the messages along the

Speaker 2: uh along the decision tree basically. And let's see more in detail how it works. So this is our top-level routing. You have the middlewares, which is this one. as you said is similar to the WISGI ones basically they aid they add data onto onto the scope object according to yeah some logic channels have some that pulls data from the HTTP protocol on protocol upgrade and put and make it available into the into the scope. And then there are the routers themselves. Here you have the protocol type router which routes the messages according to their protocol because channels is multi-protocol. And you have the URL

Speaker 2: router, which is the WebSocket-specific router to route messages according to their path. Note that this path function is exactly the same, is exactly the Django uh URL conf uh function. One thing to bear in mind is the routing is established whenever the connection is established. So uh Each consumer is tied to a single connection. In this application you will see a lot of different consumers. It's probably advisable to have more uh less consumers because each consumer will have a connection, memory footprint, etc. And so it's less efficient to have too many cons uh code scattered uh in many in too many consumers.

Speaker 2: And then you have the application routing. As you see here , I link the status path to this documents routing, which is this one. Which routes the path to the different consumers. Again, this is exactly the same syntax as Django Euroconf This is specific to the WebSocket implementation of channels. Different protocols don't have the concept of path, so they have to come up with their own uh routing uh syntax. But This is the general uh idea. So let's see some consumers because there uh is where the stuff happens Consumers are the building blocks of channels application.

Speaker 2: They are protocol independent uh protocol dependent, sorry, as uh the entry points are defined by the protocol server because the protocol server defined the events that happen. In our example, we are going to use the WebSocket Consumer, which uh it's which is kinda high level class. There are a lot of uh classes which allows to work with more um uh low level uh uh stuff if you need In case of WebSocket we have these three entry points, connect, disconnect, and receive according to the different events, client connect, disconnect or send a message. They're optional. The default implementation of WebSock on the consumer does something with it, like doing the accept of the connection, etc.

Speaker 2: And you are going to need to write your own events manager, your your event handler, and we will see why In this example, this is a very basic consumer, very stupid one, you have on connection there is this thing we will see more in detail later, which sends a message to a group uh to the users group and there is this method defined on the same class which matches and handles the the events In this case it just relays the number of users, which is static files, so not much useful, back to the back to the clients. And we will go uh we will see the details each step seta.

Speaker 2: So counting users. In our case, counting users is very safe because very simple because we have a stateful connection. So we when a client connect we just increment in some way the the number of users, we create a message here with the actual number of users connected. and we send them back to the users group. This this group, each consumer is attached to a group according to this to this attribute. This is current I I set it as as a static user's string. Actually, please note the comma. This is a tuple, not as not a simple string.

Speaker 2: But I can use I will use a property here to make more dynamic in other in following example. So I send this message and in this same consumer I just read the message with the number of users and I'll send them back to the to the clients. I can write JavaScript, this is awesome This is very simple JavaScript code I uh I've been able to write by using the WebSocket client library shipped by by channels. Here I connect to a path defined somehow somewhere, but it's basically the the endpoint I configured in the in the routing Then I listen to the incoming data from uh from the connection

Speaker 2: and I do something with it. In this in this case I just read uh the number of users and I update the the label on top of the of the window. So um we know how many users we have, but now we want to check uh who's who's doing what And in this case we have two different consumers, one to track the single document and one to track the list. This is not very different from from the other example. Here I count the number of users I I have, but in this case I I also need to check the the status and the and the slug And then I send them uh

Speaker 2: I send back sorry the the number of users. Actually it's a more complex structure which says okay there are these users in this document with this phases so read or or edit. And please note that in this case the group I'm sending a message to is is a slug, which is something I captured from the path. And I defined, I attached this one into this number of groups defining the property so this is dynamic. When the consumer is instantiated. is attached to the according to the slack of its of its path. Again I send a message and this uh this method sends back uh sends back the message

Speaker 2: But I want to go more into the details of what's happening here because this is basically the concept, the entire concept of of channels So in the connect I call this async to sync function, which is a helper function that maps the asynchronous part of channels, which in this case is the group send method. With the synchronous part, which is the connect. The connect is a simple synchronous function, so I can use REM here and another synchronous function The channel layer is an attribute for on my on a consumer instance which will map the the current channel layer And the group send is the function that actually interacts with the synchronous part and send the message. As I said, the the self-slug is the current

Speaker 2: is the group I want to send the message to. And what's the message? The message is uh basically a dictionary. In this case I using uh JSON WebSocket, so it must be a JSON storizable um dictionary, but you can uh if you go more into If you use different base classes, you can have also have binary messages. And the important part here is that is the type key. which I can actually uh can invent basically document. status which is mapped to uh to a method uh in this case document underscore status which must be implemented in each uh consumer which

Speaker 2: uh is attached to a slag uh to this specific slag group And the important part to understand is that the even if those two methods are implemented in the same uh class, They are not executed into the same instances because this is in uh called in the single instance where uh the the haven't uh has been cached or happens. And this one is executed in all the consumer resistances listening on the same on the same group. So that's that's really important to understand that uh by using events you can propagate uh things from one group to

Speaker 2: another uh to what sorry to one consumer instance to to another. The consum the document list is is more simple is basically the the same it just dot doesn't count the user and um uh and and is attached to a different uh to different group but the logic is is the same. And the front end is very similar. I connect, I listen and then I do different stuff according to the uh to the different uh to the different data. Uh one important thing uh to to know is that uh you can use channels functions outside the channels world by using async to sync You can, for example, in this normal model, I can just send a message to some uh to some group

Speaker 2: and some consumer will handle it. And this is awesome because uh channels two doesn't ship data binding uh anymore because it was uh to uh I'd say to binding to the con to the community basically to to so it it's kind of sad but it's it's nice because community can come up with other ideas. And uh I try to I make an experiment by resurrecting a very m stupid uh application I I built called Django Nokia and we'll quickly go through it. It's based on Django Meta, which is a SEO and social media, social network meter tag library. But it basically hooks into signal

Speaker 2: and send messages to a default consumer. So this is the basic class, uh these are some methods and this is the message we are going to send and these are the methods we are going to to use. But again the idea is that in the in the mixing because it's mixing based you connect on uh class distanciation you connect to the different um events pre-save post save etc same for for delete to a notification function And the notification function which in turn is called this method, which can decide if a knock or a message should be sent. serialize the message and then send this. This part is executed in Django

Speaker 2: in a normal Django model because it is a normal model mix-in And in turns calls this very simple consumer who just listen on a on a group, uh just uses language to have different language notification and is sent to them uh the message to the to the user. Uh it is a very horrible idea, so I I expect that next we'll we'll happen something, I don't know maybe based on junk rest framework sterilizer or another another thing kind of thing it would be nice thing to to talk during during the sprints if you are if you're up to So uh in the end channels made very easy to to make use or even oriented a paradigm.

Speaker 2: I can I I've been able to build an asynchronous application without actually knowing the how to build one and uh so makes uh the entry barrier lower and allows you to interact with the Django API which is obviously awesome for the people in this in this room So thank you all. And any question?

Speaker 1: Thank you, Jacob. We have time for a few questions if you have them. There's a microphone right up at the front here.

Speaker 2: Okay.

Speaker 1: Andrew.

Speaker 3: Hello. Thank you for doing half my talk for me. It's very useful. No, so um sorry, my voice is gone. My question for you is like, you know, with with All the different things you've seen with channels, um, what what part would you say still needs work? Like what what do you think is the rough edges with it with that kind of stuff still?

Speaker 2: Uh well for maybe no I've not alighted it. But um for example one pitfall is that if you don't define um an a a method that handled an an event, an event happen, you have You don't have any way to check this unless you write tests obviously, but there is no safety check in that uh regarding that. So you can send a message. into a group and then some consumer may fail because it doesn't handle that that kind of event that that band. That's that's but uh It took me a while to wrap my head up uh around channels two because I I made some experiments with version one. And so uh basically the the whole concepts part is how I try to structure my mind around channels

Speaker 2: too But uh after that is no uh I find very easy to to work with it. So great work.

Speaker 3: Thank you.

Speaker 1: Are there more questions? One, two, okay.