A Python-Driven Web App Framework... by Kendall Chuang and Henry Olson

Speaker 1: Hi everyone. Um so as uh mentioned I'm Kendall and this is Henry and we're at IazT. Uh so today we're gonna talk about um The motivation for our app framework that we developed at ISD, the React Django architecture, and we'll have a brief demo at the end. And so IST is a artificial intelligence company developing software for enterprise. And we have an enterprise software platform which we provide for uh healthcare, finance, and government customers. And so the motivation behind the app framework we developed is called which is we call Envision.

Speaker 1: was to empower data scientists to build apps quickly. So we want to enable them to build proof of concepts for our customers as well as to provide the data scientists working at our at our customers companies to also prototype and deploy apps. Quickly using our platform. And also we wanted to build many reusable components as well. So as you can see, here is a demo of an app made using Envision on the right. And so why did we choose why did we decide to go with Django? Um there's a great open source community, as you all know. Also, data scientists like to use Python. So this is a huge benefit because they have experience with the libraries like ScikitLearn, NumPy, Pandas, and

Speaker 1: so it was it's more familiar for them. We also have support of WebSockets using Django channels, which I'll talk about a bit later. Django provides form validation, so since we have to take some sort of input into our application , being able to do the form validation on the back end is very helpful. Um also there's a nice abstraction over the database in Django models. And also there's so many third-party modules as well. And now Henry will talk about React.

Speaker 2: Thanks, Kendall. Okay. So I'm sure a lot of people are wondering here, you're developing an app framework, isn't Django an app framework? Why wouldn't you just use views? Like why why bring this complicated React thing into the mix? Um and there are a lot of reasons why we decided to go down this route. So first of all, when thinking about um using client-side logic, we realized that it would be a lot more flexible than using uh server-side templating and rendering. Um and when we kind of started thinking down that route We realized that React was really the best option to go with. Some of you may know it as kind of the JavaScript framework flavor of the month right now, but it actually has a lot of staying power. And um it's proven that it's very powerful through its use of the virtual DOM.

Speaker 2: So for those of you who don't know about what the virtual DOM is At a high level, basically uh React's internals allow it to efficiently re-render content so that when it's updating the page with new data , Um it does so in basically a very efficient manner instead of reconstructing the entire page. In addition, uh there are tons of libraries and tools that are super useful for us. Um so one of them that you may know is Redux, which is basically a state management Management library for React. And there's also some React router stuff and ways to make WebSockets calls, which is really the meat of this talk that we're going to get into pretty soon. So our last reason was when we were developing an app framework, we were thinking, okay, we want our data scientists to be able to use these Python components.

Speaker 2: and write out things like a page or a form or a chart. And actually the way that React is constructed, for those of you who don't know, it's basically these JavaScript components. So it very closely mirrors Python classes so that when we communicate and serialize that data, it matches really well to the front end that we designed. So now we're going to talk about uh WebSockets, which is basically the glue between these two worlds. So let's just get a show of hands out here. How many people know about WebSockets or have worked with WebSockets before? Okay, a lot of people actually. That's really awesome. So for people who don't know, I'll just give like a really brief explanation of what WebSockets are and why you might want to use them.

Speaker 2: So when you think about um like traditional HTTP communication, uh it's kind of called a poll paradigm. So what I mean by that is basically you have the idea of a client. And a server. The client makes requests to the server and receives a response. So imagine for a second that I'm React over here and Kendall's Django. So if I want to get some information from Django, I have to go and ask Kendall like, hey, can I get a chart? Or hey, can I get some data?

Speaker 1: Sure.

Speaker 2: And yeah, so Kendall can send me a response. Uh now if Kendall wants to talk to me, I don't really care. I'm I'm not gonna listen. Um so Yeah, it's it's not very friendly. Um and it's not very efficient because every time that we want to talk we have to open a socket communication. Web sockets, on the other hand, allows for a push paradigm and a persistent connection So basically how that works is we first have our little handshake upgrade. So we start with HTTP. So I'm like, hey Kendall, do you want to use WebSockets instead? Like this really sucks. Sure. Okay, cool. So now we're using WebSockets. And now what this means is I can send Kendall messages and Kendall can send me messages whenever he wants. So you can do really cool stuff with WebSockets

Speaker 2: where the server can actually push messages to a number of clients. And this is called broadcasting. So you no longer have to operate on this model of making a request, waiting around for a response. Um now the server can send information to clients whenever it's ready. And we wanted to utilize this for the asynchronous loading of big data. So for Invision, I mean you can imagine data scientists. working with uh pandas and numpy and these huge data sets. Um it's gonna take a while to process things on our Python back end. So we want to basically push that data to the front end as soon as it's ready. In addition, we realized we could use this for notifications, progress messages, loading spinners, basically a bunch of different use cases for keeping the front end very responsive and interactive.

Speaker 2: So you're not just waiting around for data to load. And now Kendall's going to go ahead and talk about Django channels, which is how we basically bring this architecture to life.

Speaker 1: Cool. So here's a diagram of the channel's architecture. Uh as you can see on the top, there's the the web browser and Um on these blue lines on the left are the HTTP uh messages and on the right you can see WebSockets. Um there's this interface server which handles the uh Basically ASGI and WSGI communications which connect to Django and Django channels. And then there's this channel layer which kind of routes the messages. to the consumers. So you can see the consumers here. There's the HTTP consumer which can route messages to your standard Django view functions. And in addition to this, you can see the

Speaker 1: WebSockets consumers on the bottom for WebSockets messages. And here on the right you can see background processes as well. So this is one of the other nice features that we have with Django channels is you can actually have the background processes send messages back up to the client. So in our case, for example, if you're using uh a background worker like Celery, you can have Celery worker uh send updates all the way and push uh notifications up to your client when a long-running machine learning model is done training, for example. And so the libraries that we're using here are include include Django channels for the channel layer and

Speaker 1: consumers, and also Daphne, which is a uh package developed along with uh Django Channels 2 for the interface server. So the interface server is uh Daphne. And on the back and optionally if you like you can run celery as well. Um and so Let's see. And Henry will now talk about the front-end architecture.

Speaker 2: Cool. All right. So we're back to more front-end stuff. Uh I'm just gonna give you the warning that we might get a little in-depth with uh some React Redux stuff. So um if you're not familiar, um don't worry, there's there's a like a ton of boilerplate, but we'll try and stick to just the core logic here. So before we dive into that, we want to talk a little bit about the object model that we designed for this application. So uh maybe think to yourself really quickly, if you're writing a framework for people to write web apps in Python, uh what kinds of things do you need to have? What kind of things do you need to expose to them? And for us, we are thinking of these things that you see in this hierarchy model here. So, first of all, you have the concept of an application.

Speaker 2: I mean this is really basic. This is your overall app. Maybe just has some configuration and a theme. And within an application you might have uh a navigation or tabs, ways to move around the app and visit different pages. So in our world, the page is like the most important object. This is what data analysts and data scientists write in our application. They write pages. And within pages, they use forms, which are very familiar to Django users. Forms have fields. They can be submitted. They can be validated. They're very useful for interacting. And uh you have blocks, which are like charts, tables, anything that you want to display to users. So with this combination of objects, you can present data, you can get user feedback, basically everything that you would need in a

Speaker 2: A simple web application that displays some information. So we use this object model both on the Python side and on the React side. It can be kind of annoying because sometimes You know, if you create a new component, you have to create both a Python component and a React component. But as I mentioned earlier, they are very similar, so it's a pretty easy process to have them available on both sides. Now, when we're actually rendering things on the front end, we use, as I mentioned, React and Redux, and we use JSON RPC methods. Or basically you can think of these as ways of making WebSockets calls. So we're gonna go ahead and take a look at some code here. Um so we have this demo repo, and we'll show you the demo at the end, but basically um we'll start with a really simple example of how this JavaScript is instant.

Speaker 2: And then move into how the actual flow works. So for starters, we're going to go ahead and take a look at how Django is serving up React. So if you wanted to do this yourself, it's actually pretty straightforward and I'll show you how it's possible. So is that a good enough size for everybody? Can I get like a thumbs down if it's not? Okay, great. I see some thumbs up. So here's uh basically our views function. And we have one view in this application, and it's really straightforward. It's an index view. So uh our URL our URL routing, excuse me, is um set up so that this is basically loaded every time. that you hit the running Django server. So we hit the index view, right? And this is really the only line that I want you to care about here is we basically just render a template.

Speaker 2: Like that's what Django is all about, right? It's pretty straightforward. So let's go ahead and take a look at that template and see what that's doing. So if we go here into our index. html. Okay, this is pretty straightforward. We have uh just basically a thin wrapper around base. html here. So this is really the meat of where this template is lies. And here we have some meta tags and we have some configuration information. There's a few things I want to point out. First of all, we uh use this to load in any static files. And the static files that we're loading are our main CSS file. And our index. js file.

Speaker 2: And uh, you know, you might be saying, yeah, this is this is pretty basic. This is like Django 101. Uh but the fun thing about our Envision app is this is the only view that we have. And the only set of templates that we have. So this is all the front-end logic that's done or that's handled by Django. After this, we basically pass it off to this index. js and that handles everything else. So that will take care of routing. making calls to the back end, all the user interactions. So uh think of this as a very basic handoff uh from Django over to the React side. And this index. js here, this is our fully compiled JavaScript repository, just minified and compiled using Webpack. So all our React code is loaded up as soon as you hit the Django app.

Speaker 2: Okay. So now we're gonna go ahead and uh do a brief explanation of how the React Redux flow works and then we're gonna tie it into how that works with Django. So once again, let's just do like a quick show of hands here. So how many people have worked with React and Redux together on the front end Okay, so less people, but still actually a really good amount. I'm very happy about that. React's awesome. So uh for those of you who don't know, we'll walk through this really quickly, and maybe this will be a good refresher if Uh for some reason you forgot how it works. But um basically the main concept behind uh React and Redux when they're combined together is you have this thing called a store. And it's a data store. It holds your data for your application. Pretty straightforward.

Speaker 2: Now, your store contains pieces of state, and these might be like, um, you know, what user you have logged in, or what page you're currently viewing. Now that state defines the UI. So you have components that are rendered based on properties or props passed down through the state. So when you visit that page component, you can see, okay, now I'm going to show this page for this user with this color. The UI then triggers what we call actions. So you might do something like click a button or select a dropdown, and this might trigger a Redux action, which can can basically modify some data or make an RPC call. So this right here in actions, this is where we make the call to the Django side. When we get any data back, we send that to a reducer, which basically just transforms the data and passes it back into the store.

Speaker 2: And then the magic of React and Redux is once the store is updated, it updates the state, and the UI gets the new props. So just like that, we can go from clicking a button to receiving entirely new data. And here's how that flow looks like for Envision. So a little bit more specific. So we start off with rendering our page that the user has defined. And at the start, the page might not have any information. It might be really basic. But then once you click a button or trigger some request, we kick off that WebSocket call and we call it Git page in this case. This call goes to the backend and hits a Django consumer, which is basically a listener on the WebSocket channel. And that's called get page as well. So that's how these two relate to each other.

Speaker 2: Django does its magic and we'll uh process some data, maybe use some uh Python libraries to do some cool stuff there. And we return that data and send it to our git page action. This basically goes and tells our reducer to take that data we got from the Django side and add it to the new state. Then the components are automatically re-rendered for us, and the page now has some new data. So this is a pretty standard flow, and all we've really added is this piece here where we send a little message to the Django backend. And so now we're gonna go ahead and do another brief code example of what this looks like on the JavaScript side. So again, I apologize if you're not interested in seeing any JavaScript, you're gonna have to sit through a little. But um it's pretty cool, I swear.

Speaker 2: So if we go into our app here, here's our app component. And again, we can excuse all the React Waterplate. I'll just highlight the parts that I think are interesting. So in our render method, this basically tells us, you know, what kind of HTML content are we going to put on the page? And these are the properties that we're receiving from the back end. So here's some really interesting ones. Uh socket and socket node are basically uh components that instantiate the WebSocket connection. So they make that handshake that I was doing with Kendall earlier. In order to upgrade the HTTP connection to a WebSockets connection. This next piece of state here is called page loaded, which is basically a Boolean. It tells us if the page is loaded or if it's currently loading.

Speaker 2: And lastly, we have our page data, which is a dictionary, which basically has all the stuff that we want to display. So that's the information we get from the state. And then we can go through down here and say, you know, maybe we'll make some title text and we could maybe add some images to the page, but only if we have page data. So if we don't have any page data here, we're just going to skip this section entirely. Now when we get down to rendering the actual HTML content, this here is pretty important. So this is where that um socket component actually instantiates the content. connection. So as soon as you hit the page, it's going to go ahead and do its job to connect us to the Django server with a WebSockets connection. Once that connection has been established, we can come down here

Speaker 2: into this block. And basically we're just gonna render, you know, a little little header tag, little button, maybe another little button. And they're gonna have some on-click handlers. Now, when we click these buttons, we're gonna dispatch that action that we talked about. So I can just really quickly go ahead and show you what that looks like on the JavaScript side We go back to our app container here, and this is what contains the business logic. And here we have the function called by the buttons. So when you click a button, it's going to call load page. The first thing that it's going to do is send a message to our, or it's going to send an action, which is going to basically tell us, hey, let's Set that page loaded to false. The page is no longer loaded. Don't display anything. And then it's going to go ahead and dispatch a socket message.

Speaker 2: So this is the WebSockets call here It's gonna dispatch a message called git page and it's gonna pass along a page ID. Pretty straightforward. Once it receives the response from the Django consumer, Then we'll go ahead and dispatch an action to set that data in the state and reload the page. So when we do the demo in a second, you'll see this happens incredibly fast, but this is the entire round trip that's happening in order to basically Render some information from Django in React. And now Kendall is going to go ahead, I believe, and talk about how this looks like on the Django side. So what actually a Django Consumer does.

Speaker 1: Well thanks Henry. So we have uh this protocol called JSON RPC. And so JSON RPC lets us have a nice interface for sending and receiving messages and what it provides us is uh an example like this. Um you have an ID tag which lets the client, uh the web browser, uh set a ID for which when you send a message Then the response you get back should have the same ID. So if you send a bunch of messages, this helps you organize the responses. The JSON RPC is the version, and you can look online to see the difference between the two different versions and here we're using 2. 0. The method is the name of the Python method which we're going to call in the back end and the parameters are the page ID.

Speaker 1: And so we're using this nice library called Django Channels JSON RPC, which is and we're subclassing from the Django Channels WebSockets consumer. So this is the main consumer which I showed in the block diagram above, uh previous slide. And the Django consumer will then return the JSON. response. And so here's our actual getPage method. As you can see here, we have subclassed from the JSON RPC consumer. and made a demo RPC consumer. And so all you have to do is create a decorator at dot rpc method. and then define your WebSockets consumer method. So basically here we define get page. It takes in the page ID.

Speaker 1: And it returns a dictionary of the information you need to render a page on the front end. And so now Henry will show a quick demo of the WebSocket.

Speaker 2: Alright, so we're gonna do a live demo, so bear with me. Hopefully everything works. Okay. Um so the page is loaded, this is good. Um so what I'm gonna do while I have this up here You all can see that right? Okay, cool. So um we have open here basically the developer tools console. And the reason why this is open, actually, sorry, it's uh Oops, make it a tiny bit smaller. Is that still all right? Okay. Just so that we can uh click through it. So basically what we're gonna do here is um show the actual WebSockets calls that happen while this application is running. So you can do this yourself. This is available to you. It's a handy little tool, and if you're writing any WebSocket applications, it's incredibly useful to see the responses you're getting.

Speaker 2: So when you open your developer tools in Chrome, it's in the network tab under WS here, WebSockets. So first things first, when I went ahead and loaded the page, we already got something in So let's go ahead and investigate this and see what this is. So this WebSocket message um is actually We have here a 101 switching protocols. So this is that handshake upgrade that I was talking about. As soon as we load the page and enter the application, we switch over from HTTP and open that persistent WebSocket connection. And here you can see that indeed we're connected to Daphne, the server, so everything is working as intended. So now what we're going to do is go ahead and click a button and start that flow that I was talking about earlier.

Speaker 2: So just again, what's going to happen is I'm going to click the button. We actually have a time sleep on it, but it it's still really fast, so um it'll be pretty quickly uh like blink if you miss it. But basically when you click on this button, we're gonna go ahead and set the page to loading Go get some data from the Django consumer and return it to the front end and render it. So bam, we got some company logos. Um and if you look here we got some uh WebSockets information. So we can go ahead and inspect that. that. So here, sure enough, this is our git page web socket method that was sent out. You can see like the JSON RPC number, the method name, and sure enough, we sent over the page ID. And then if we actually go to the terminal, we have printing in the consumer

Speaker 2: get page. So indeed our Django consumer picked up the message, did some processing, and returned us some image links and a title to display. And you can see that here in the response. So if we look at the result, bam, there's the the images and the page name. And uh this actually it works on the other page too, so we can kind of go ahead and toggle between pages And this is like the basic functioning of a standards WebSockets loop. Now you're probably wondering, um Isn't this like extremely basic? Like couldn't we just do this with HTTP and views? Like why do we need WebSockets to to load images? You know, I can do that in my sleep So if you thought that, you wouldn't be wrong, um, because yeah, this is a pretty basic example. But uh as we'll explain as we go on, where WebSockets really shines is in those other kind of interactions

Speaker 2: where The server can actually push data when it's ready to the front end. So here we're kind of doing a standard request response. But for example, because we have that persistent connection open, we could actually, you know So send the request to the server and the server could go do whatever it wants for 10 minutes and push a response when it's ready. And the front end doesn't have to sit there pulling for it. But we'll get into more of that in a little bit. So now Kendall's going to walk us through how state management works in our Envision app.

Speaker 1: So here's the uh diagram we had again. Um with Envision we have a stateless API. So uh Basically, what this means is we're not storing this state in the Python code. We have the date data backed up by Django models. Um so basically the nice thing about this having the a separate data store, um, for example, Postgres, is that we can allow the view function, the consum the WebSocket consumer and any other background processes to talk to the same database and uh share data through the database. And uh also as Henry mentioned, we store minimal state on the front end as well and re-render when we have a new

Speaker 1: information on the page. The other nice thing about having state on the um having the background processes is basically we can Send notifications up front after running a long task as well. And also another place where we're storing another database that we're using is uh Redis. And so the channel layer is actually backed by Redis data store, basically. So this'll uh it manages the recipients. So when you have the channel layer Um you want to know who you're sending the messages to. So this channel layer allows us to um broadcast the messages to multiple web browsers, for example, if you're connected to the same page and they all want to be updated at the same time.

Speaker 1: So now Henry will talk about some of the challenges that we faced in developing Envision.

Speaker 2: Alrighty, so this is what I was alluding to earlier. Um this is asynchronous loading of charts, tables, and other big data. So um in the example that I gave and in our first draft of Envision, our WebSocket's calls basically rebuilt the entire page. And the reason why that was happening is is we had a page object written in Python and we sent a WebSockets call, get page, to go get that information. We then serialize that page and send the entire thing up. And um while that works great for really small pages, you can imagine that that might be uh really painful if the content that you want to display on the page is extremely large or might take some additional processing based on the request that came in.

Speaker 2: So it took us a while but we thought about some interesting ways around this and this is really where we decided to take advantage of WebSockets. So we created this publish subscribe model, and I have subscribe in quotes because we we took inspiration from this kind of model, but it's not actually how it's working. There's no poll or subscriptions going on. Instead, what we do is when we send that data to the front end for the first time, we might send up, say, a pie chart and it has um no data associated with it originally. But we can define a data source, which is basically like a key. So let's say we set up a pie chart and it has this data source key, pi. So when we load up the page, we might have some content that is displayed, like a title or some fields.

Speaker 2: And instead of trying to wait and display this pie chart when it's finished loading or processing. we can just display a blank box or maybe a loading spinner, something that indicates that the chart is yet to come. And then what we can do is we can actually manage that processing asynchronously on the Django side. So this is where we can use uh sub processes. Workers, additional threads, or even something like celery to basically handle some additional processing and Maybe it'll take a really large data set and condense it down, use a library like Plotly to create this beautiful chart, and then serialize it and send it up when it's ready. And the way that it's going to send it back up is through a different RBC method we created called refresh widget.

Speaker 2: So this call is kind of the inverse of what I was demoing earlier, where the server is just sending a message to the front end without any previous um request. This wouldn't be possible in HTTP. So the server, whenever it's finished, can just send up this serialized chart information along with that key pi that we mentioned earlier. And the front end will receive this data and can basically go through the DOM, find the element that was rendered with that data's source key. and repopulate it with this data. So you can have a case where you see some of the page loaded instantly and then other pieces come in over time as processing is finished. This is incredibly useful for the kind of work that we do. The software that Iazdi is built on involves a lot of machine learning algorithms that can take hours or sometimes

Speaker 2: days to run when we're talking about data sets that are a million columns by a million rows. So it was essential for us to be able to have this asynchronous loading and it could really only be accomplished with WebSockets. Okay, so do you think we have time to do a quick demo of uh what the actual application looks like? Kendall's gonna go ahead and show that to you.

Speaker 1: Cool. Oh thanks Henry. Sure. So here's a demo of our app at Iazdi. Um here you can see on the top we have well we have the application, which is in this case sample app. Um we have this Uh navigation bar, it's uh step-based workflow. So we're just showing, for example, a fields page, a blocks page, and containers page, and then mixed fields and blocks. And so I can type in something here, uh Kindle, and then fill out a form. I can upload a file and This is basically the React version of a Django form that we would instantiate on the back end, basically. And then

Speaker 1: We click OK and then you can see some progress being sent up and then loading the next page And here you can see an example of some of the other components which for which um Henry described basically. For each of these components, for example, a table, uh another table, an image. Markdown, we have both a Python component and then a React component. So the The difficult part is the first time is we have to develop this pair of components. So it's a bit extra work, but then in the end, the result is that the data scientist who wants to instantiate these just needs to write some Python code to instantiate So this is really powerful because then the d data scientist doesn't need to know JavaScript to develop these

Speaker 1: components. Um and here's an example of some other charts that we can show. And some containers so we can have different styling of the containers as well and um showing hiding Enabled through so in a lot of cases w to enable these we have to provide extra parameters in our Python objects. So if we want this container to be um Uh be be able to show and hide this container, we have to have a parameter which lets you set this configuration on the front end as well. And then on the last page.

Speaker 1: Let's see, mixed fields. You can see here's an example where we can intermix like both the Django form field as well as some um images and uh table. So I think that's it for the demo. Um yeah, do you guys have any questions on the demo or the yes

Speaker 3: With connections dropping or you know like a web socket I don't know, the connection being dropped and and the client like waiting for a response that never never got here or or what if the client would like close the tab? after or s or something like that.

Speaker 1: Yeah, so uh yeah a lot I mean this happens qu so the question is if uh what hap what do we do when a web sockets drops. So yeah a lot of times like you you know you have to sometimes refresh the page and so then it'll reinstantiate the WebSockets connection. But then the nice thing is since we're database fact, if you go to the page, okay so Henry mentioned this. previously, but we're using React to route for the URLs. So this actually we're still loading the um index. js here, but you can see the URL is objective slash 14. So this will tell React, okay, we need to get this objective for and maybe this page number 14. So then if you refresh your browser, then you can go back to the same page because the

Speaker 1: the um you're fetching the data for the page from the database and you know which which page you're on basically.

Speaker 4: Hello are there are there any special considerations? Regarding authentication and session that comes into play with uh channels.

Speaker 2: Um yeah, so there's there's a lot, it's a really, really complicated topic. And maybe maybe next year at DjangoCon we can talk about authentication with channels. But um, you know, in brief, um the one thing that you want to keep track of is basically when you create a session or get some cookie you will want to recreate the WebSocket connection using those cookies. So that's um kind of one of the issues that we've had to face is um Um you know, making sure that basically you instantiate the WebSocket connection, you might hit a specific endpoint or um enter some data on the the page and it it authenticates you and creates a Django user object and provides a session or or some cookies, basically some way to track that you're logged in. And you actually have to recreate

Speaker 2: create the WebSocket connection with that and make sure that that information is passed with every request. So that is still something we're kind of building out on the envision side. But another thing to note is that that's actually the only other place where we use a Django endpoint. So I lied earlier when I said that that was the only view uh we actually have one more which is for logging in because um that's another thing that Django provides us that's great is um the user system and authentication and so we actually use a standard HTTP authentication endpoint for Django. And then once that is established, we recreate the WebSocket connection and make sure that those cookies, so that authenticated a trace of that authenticated user is present in every single

Speaker 5: I'm kind of curious how you deal with load balancing in a situation. Let's say you've got uh two clients that are accessing uh the same server and all of a sudden one client starts up a task that's gonna be like really demanding on the server. Now the other guy's gonna be waiting and And I d I don't know how you can dynamically load balance basically in a channel type environment.

Speaker 1: Yeah, so for load balancing as far as the tasks, we we 're The the tasks between the front end uh browsers and the Django app should be pretty light. Most we have a separate um API uh a separate machine learning platform where most of the uh big pr uh data processing happens like in Hadoop. So this is kind of asynchronous. This is why it's nice to have WebSockets because we can have like, for example, a salary worker waiting on a big task. And then when the when the task is done on the And and so there's an HTTP API, so maybe it's it's it's actually waiting on an HTTP response. But then when the salary worker is k gets the response back, it can send a WebSockets

Speaker 1: notification back to the two browsers that okay this long running task is done. So we we offload the work to our our our um platform um for for big tech Yeah.

Speaker 6: Hello. Uh thanks for the great talk. I really think this is really cool. Um my question is for all these data components, are they uh served the markup and the logic for that are they served initially or are they served as a response to the uh RPC request and uh somehow that component is injected into the reactor

Speaker 2: Yeah, so it's more like the ladder. Basically, as Kendall's mentioning, we use the React router. So when you visit a page with a specific route, it makes that get page call to the back end. And essentially, if you were to break at that point in JavaScript or pause on a breakpoint, you would see a blank page. And that data basically you would send the request to the consumer side, it would process the data and serialize the whole page as a result, and then that's returned up to the front end. then load it in there. So essentially that's why we try and keep the pages as lightweight as possible because if you were to refresh the page or hit another page, you're doing another round trip through the consumer side.

Speaker 6: That's really cool. Thank you.

Speaker 7: Thanks for the talk. So how how do you write tests for uh this kind of thing?

Speaker 1: Okay, uh good question. So let's see Okay, so for testing we we had yeah it was it's kind of it is kind of tricky to test. We uh we did a couple of things. For unit tests, we mock out the web sockets. server so we have this mock object which uh mimics the server. In addition to this we have this uh we call it the driver so uh python web sockets client to also test out against a live like Django app, which is which you can test. And in addition to this, we have Selenium tests. So it's it is pretty complicated to test, but we we are able to test using uh these three methods. And it is important to to have a lot of testing.

Speaker 7: Assuming with uh WebSocket you have a kind of a back and forth. um different states carrying over uh between the conversations. Uh how do you emulate those?

Speaker 1: Um so the questi sorry, could you uh

Speaker 7: like I

Speaker 1: don't quite understand the question. Yeah.

Speaker 7: How do you emulate those back and forths and maybe even uh emulate in the interruption of the uh a connection?

Speaker 2: Yeah, so we for our unit testing we actually mock out all the um calls and we'll typically test out, for example, um like one method at a time. So like using unit test principles, we can basically uh mock out the RPC client. And there's um some boilerplate that you can look into that basically sets up a fake web socket connection. So you can just just kind of test that indeed the right parameters are being sent over the connection. In terms of testing the full back and forth connection, that's what our Selenium tests handle. So basically we have a bunch of sample applications. that test that, okay, if I do indeed hit this page, then this thing gets called and the page gets re-rendered with this data.

Speaker 2: So um we kind of if that makes sense, yeah, we we test each method individually with all the RPC stuff mocked out, and then we test the full end-to-end application. Using a live selenium test.