Django's Data Science Makeover: Integrating D3.js and Bokeh for Data Visualization

Firstly, hi everybody and today we have been learning a lot over the past three days about Django and the wonderful capabilities that Django has. But equally important is to acknowledge the data science world that's happening around us. All the buzzword about machine learning, LLMs, including all of that in our applications. So, why don't we learn things such that we are able to use the capabilities of a few other languages as well and integrate it well into our Django applications? As I dive in, I'd just like to introduce myself. So I'm a software developer. I'm getting started a lot into the machine learning space as well. I have worked at product-based organizations as well as in the fintech space, worked at Adobe and City.

And I'm also a social entrepreneur. So currently I'm based out in the US since a few months, but otherwise I'm originally from India. And back in India, I have a nonprofit organization that works across 11 cities. We are a 400-plus volunteer-strong team. And we work across environment, education, and healthcare. I'm also an international tech speaker, so I like to share whatever I do with the community around me. I get to travel the world, share my knowledge with developers across the world. So you can find me at different conferences, sharing about different projects I'm working on, my learnings, and any and everything that I find interesting in tech I'm also a career coach. So if you scan the QR code, you'll also find a link to my website, which has my calendar. So if you're looking to advance in your career, make a transition from a non-tech

background to a tech background All of those details are listed on my website. Coming to d3. js. So in JavaScript, D3 is a library that allows us to work with a lot of visualizations. And the best part about D3. js is it does not give you a fixed template. You get to code each and everything about your chart. As and when you want it. So, what this gives us is a raw layout, and you can customize each and everything you want about a visualization to match with your expectations. So it is not a single monolith. Rather, it has about 30 discrete libraries that you can include in order to generate charts.

Now these thoughts could be related to a machine learning model that you have created. How would you work along the machine learning model to display data correctly? Or if you're working with real-time data, you would want a very different representation and you would want it to be very interactive for the user in any of the dashboards in Django that you build And it does not have a default presentation of your data. So it is not like a few of the Python libraries that we have that there has to be a set uh number of parameters and it will the display would be uh kind of fixed. You would just modify the values, change colors and things like that. You can actually modify and customize the chart according to your needs. It

also does not include uh by default have a new graphical representation. Instead, with SVG and Canvas, you can integrate it such that everything is customizable. And for any application which uses maximum uh expressiveness for your visualizations, D3 is the way to go. And if you ask me how do you actually have the two worlds of D3 and Django combined together, you simply include it in your script tag. And that is what helps you do it. In a couple of slides, we'll also see an example of working and customizing a chart on our own. To show you the different um capabilities that are possible with

d3. js and Django, consider a plot like this. This is a scatter plot with different shapes. So, this is the basic data set of the iris flower and the different three categories, the three species that are out there. Now instead of representing it as normal dots, I want it to have a more um different shapes to represent different species. So that is something I can customize Or if my application requires me to show what part of the world is it nighttime and daytime. So I have very quick ways to use the 30 inbuilt libraries and build on it, customize it to quickly create visualizations like this. Or when I'm trying to show difference between two

Two quantities. So this is temperature being hot and cold between two cities. So with D3. js, it is very easy to add the logic. You are in charge of any and everything that you want to create You are not restricted by the library's parameters or the set output to have the kind of data that you want. And interestingly, even contours. Contours could be uh contours are something that's like very important when you're working with machine learning projects. Especially when you're dealing with clustering, you would want to have different features showed in the form of like volcano contours. So let's look at an example of how would you actually create this contour So, to come into the code, if you see, you'll uh realize that for the chart, the width, height, all the SVG parameters are possible to be edited.

You can also have what do you want to append it? Are you going to take out the mean of it? Are you going to take out the mean-medium mode? What kind of statistical measure are you going to have such that each of your contours is equally spaced What is the distance you want to show in the black outline of the contour we saw on the previous slide? So, and if you realize, if you think a bit more deeply about um This picture, you will see that there are two major divisions. One is through the colors. So colors could represent different kinds of concentrations. Maybe that is your target cluster that you want to display Or each of the contour lines, what should be the difference? Should it be only based on your data? Would you want to have something more to highlight a particular part of data?

So all of this is possible to be done right here in your code. So everything is editable. It's not a set fixed template that you have to follow. And once you define your chart, then data is wherein uh you add in your data as a JSON format. So you'll simply import it. Um In your chart, and as soon as you do this for the final plot, in order to display the plot, this is how you will add your data dot values and your width, height, fill, stroke So stroke is black, the outer black lines. And with the height width, you can identify the complete plots width height. So, this is helping you to actually go ahead and customize the contour as much as you want.

And once you have all of this created, you just include it in your Django applications, like we saw in the previous slide And that's it, you're good to go. You can include it, use it in your Django application, and this part of your Django application would be powered with D3. js. Now, consider a case wherein you are working with more real-time data. So maybe it's related to sensors. You have a sensor that's continuously giving you a lot of data. You need to have constant real-time um working of these, uh of the data in your application. In that case For modern web browsers, another library is the Bokeh library. So D3. js can be used more

than wherein you have a model already there and you want to have those visualizations to be put up very well. In Boke, whenever you are working more with a web browser-based application, and if you're more comfortable with Python over JavaScript, then you can use this library. The major advantage is that your bokeh library codes along with Django, it's very easy to share. So you can publish it as a web page Or even in your Jupyter notebooks. If you're working in a research-based environment, or if your project is still in the RD phase, You would want to share your project with your colleagues. Now, this could be this is mainly in the form of Jupyter notebooks. So in that case, Bouquet becomes more advantageous

for us. And it is interactive and very powerful. So even inside Bokeh, you can still use a lot of JavaScript capabilities. And these capabilities will help you to power bokeh visualizations much more. So uh this is one of the examples. So if you see on the left side we have different parameters that can be modified. In order to generate different parts of the graph. So anytime you want something that is interactive, you have a number of fields and you're still trying to extract the insight from data. So once we have gathered data, a key aspect of working with data. Is to generate insights from it. And it is not necessary that as soon as you have collected data or you have applied some sort of machine learning algorithm, you will have the insights

There's a lot of back and forth that goes around to generate the actual insight from data to make some meaningful business logic out of it. And that is wherein an interactive plot is very useful. And Bokeh helps you do that in a very easy, efficient way. In order to integrate Boke with Django, um your main HTML file would include uh These two aspects in your head and your uh body. So your script tag and your um bokeh CSS. So your CSS has to be added as well as In your main file, your JavaScript version of Bokeh library has to be imported. So once you have this, having visualizations Like this is very useful. This is used a lot in the industry, especially when a lot of data analysts are trying to understand the information

that you created in your data science or machine learning project. So data analysts tend to work with kind of not the final output, but a raw An output which is not raw but more processed. So using this semi-processed data to generate insights, they would want to play around with the data to see if there are additional insights generated. So anytime any project is targeted towards making the business use case to be more revenue generating, there could be additional insights, and that is where in interactive visualizations play a key role. Now coming to an important aspect that I just talked about. So whenever you have real-time data streaming, how do you take care of that? This could be related to sensors.

It could be related to conferences. So for example, at DjangoCon, we are generating so much of real-time data that could be related to our social media posts. related to the Slack channel, there are a ton of uh real-time data that is being created. So how do you handle this in Django? In Django, we have Django channels. You simply do a pip install for channels and Django channels is ready for use. Now, what is the need of having Django channels? In um at its core, Django is built around the single concept of having requests from the browser. Going to the view function, fetching the response and returning it back to the browser. This is at its core how Django works. But the drawback is that there's no way of keeping uh a connection open.

And it only works when you have. Like a request response kind of a pair. It does not work if you want to keep it open or you want to return back something when you don't have a request. That is wherein Django channels comes for our advantage So uh Channels allows Django to support WebSockets in a way that is very similar to HTTP views. And channels also allow for background tasks to run on the Go and in the same server. The HTTP requests continue to function as You would normally work in a Django application, but also they get rolled out to channels. And once you have the channel layer in Django, this is how the architecture looks like So, it is an additional layer that your Django application works with.

And with using WebSockets, you can still handle continuous streaming of real-time data that is happening in your application. Now, Django channels work across a network. They allow producers as well as consumers to run transparently across machines. And they make use of a concept called task cues. So task cues actually help you send messages through the producers to the consumers and back again. Now All of this makes it very, very usable for Django to handle real-time data. And once we know we can handle real-time data, We are all happy because handling real-time data is the future, and we observe it in any and every task we do.

Along with real-time data comes a lot of time series data. So every data that is generally generated in a fast-moving world, which is continuous real-time, is generally time series data. There is a timestamp associated with every kind of data. And handling time series data is very crucial. It is important for making any kind of informed decisions. Now, these decisions are important to identify anomalies, to generate new trends, analyze different trends that are happening. And all of this is possible with the capability of Django channels. And a key aspect also with handling real-time data is any data science project or any machine learning model that you generate. Having the capability to handle real-time data and also have a feedback loop in your machine learning model to handle the feedback that is generated

While interacting with the real-time data you're getting is the key of achieving uh great results by your machine learning model. So now Django channels is very interesting and it is a bit advanced, so why not we try our hands and let's create a uh Django channels application. So the first thing is you would uh pip install channels, you would configure the ASGI application in the Django project settings. Now the next part is your ASGI Python file. You would import the required libraries And you would set your default settings and uh the nomenclature I'm using across in developing the application is so that If you try this back home, you can simply replace

your project name to what project you're trying to build And in your application, you would have a protocol type router. This is the way you would route your information. So all of your routing information goes inside this block. The next part you have to create is the simple uh WebSocket consumer. So you would have your web, uh you would have your consumers. py file. And you would um mention everything about what happens when it disconnects, how does it connect with it, once it receives a request. Is there a message you want to send? Is there something else that you want to send? All of that logic comes in the receive part And once you have this, you come back to your ASGI file

and you would include the URL router for the for the WebSocket. You would also mention the different endpoints that your Uh, consumer web socket is going to have. Now, these endpoints are important because you or the user would be accessing uh and generating real-time data, generating requests through this endpoint. The next important part is to integrate Django channels with views. So we created the first part of the requests handling and the response handling. Now, how do you combine Django channels with views? So you'll create a views. py file, and this file is what will actually happen. What is the kind of data that you would send back to the customer, to the consumer? So, for example, right now I just said hello from DjangoCon

US 2023. So that is what I want to send back as my message whenever I get some real-time data. You can modify this to also have some visualization. You can include inbuilt libraries or use an integration with D3. js and Boke. It's up to you based upon your application requirement. And once you integrate this, you need to also specify the routing configuration to handle the message type in the consumer. So this is wherein your message type could be different things. Is it just a JSON dump of say text that you want? Is it something else? Also, interestingly, you can Do a quick tweak here. Your JSON dump can actually have insights like statistical measures that you generate as a part of your logic, and you just give it back as a JSON dump instead of doing the processing later on

This way, it is very easy for you in the visualization part. All of the complexity is removed from there. And while returning back the message itself, you are generating everything in the form of a JSON. You just keep appending to the file And this dump is what in what you just give back. And as soon as you give this back, uh you are done. We have uh Django's application ready. And you can just deploy it to any ASGI server such as Hypercom, and that's it. You're done. You have your application ready, which is capable of handling real-time data. Also, an interesting application of real-time data is the chat that consumers have on your web application. So there are a lot of insights that can be generated. Mainly

a chat application has a request response type. So you can handle it without channels as well. But with channels, it is also possible that you're still waiting on a request back, but the c the Person interacting with the chat is like continuously posting things. Or maybe you want to also take care of the click stream data the person is having. So if they are on your application while they are chatting, they're also scrolling. So there's a lot of clickstream data being generated. Maybe they are browsing different pages. So you also would want to capture that to understand what the user behavior is. Even in that case, Django applications make it very, very interesting. Now we know how to handle real-time data. And all of this was a part of bringing the machine learning world into Django in a very, very easy way.

So the most important part of whenever we are dealing with machine learning applications and bringing it to the Django world is to find the correct harmony. Now, when we think about machine learning, it might seem that it is a bit complex, it has to be interactive, and it is ever-changing. So earlier we would say look only towards supervised, unsupervised models. Now we are looking at GANs, at LLMs. So all of this landscape is quickly changing. And A key aspect of keeping our application up to date is to have your machine learning model, but also keep your visualizations to be more interactive and to be ever-changing. So you can directly call machine learning models into your Django application, utilize

D3 or Bokeh in order to have visualizations. So you can have an interactive dashboard which helps the user to understand more about the kind of data they are generating. You can also Have uh your data being uploaded directly and this data to actually be used in order to create the visualizations that the customer wants. So you can create a channels, a Django channels application, have different data being ingested from different sources, handle it as soon as you import it, and then work with different visualization techniques Now, there's a reason why I emphasize so much on visualizations. So when you have tons of data, it is very difficult for you to Find out insights just by looking into data

or by having different uh row modifications, column modifications to understand the insight. Once you have that data visualized, especially for anomaly detection It will be very interesting to see it visually and you'll actually identify some clusters or some data points that don't follow the norm. Now these anomalies could also be something that you had not expected. Maybe your data cleaning was not good. That's why you have a point that is Uh coming outside like has a very different value as compared to the rest of your data. Now, one way would be to actually go back and check back everything in your table in your data modification Or if you have visualized system, it is very easy to identify, oh, this was a mistake during the data cleaning process, which could actually have a very bad ripple effect if it is not handled.

So having visualizations also helps you to correct things which otherwise would go unnoticed. So make use of the great interactive visualizations we have. And Whenever you're dealing with real-time data, try and make use of channels. Channels are a lifesavier. It's very easy to implement. Just like we walked through the application as well, it's a couple of Python files. And you can use this as a boilerplate code. You can add in your actual application logic, change it a bit, and it will work. So, handling real-time data will become very easy. So, towards the end, I would just say: visualize the world and power of Django. Make use of all the capabilities of visualizations not just in the end product, but even in between to see if you are on the right path or not.

That's it from my end. Thank you. And you can find all of my socials on the QR code as well. And happy to chat about things that you're working on or things you are planning to use, D3 visualizations, and Django channels. Um and I'm open to collaborations. Thank you.