Build, deploy and scale Django, GraphQL and SPA

Hello everybody, today we are going to talk about how to build, deploy and scale your Jang 1 single page applications using GraphQL endpoints. So before I begin, let me tell a little bit about myself. I am Dilip Shiva. I'm an optimistic nihilist, democratic socialist, and I quote for fun and profit. I love science, Python, Foss, and Tamil, which is my mother tongue. I'm a dad of two, an environmentalist, storyteller and a gamer. I'm a jock of all traits, but master of none. And I volunteer for a lot of progressive movements around the globe. So before we begin a standard disclaimer, I have no idea what we are talking about. I uh you know play around with a lot of things, front and back and mobile applications, IoT uh you know as long as it's coding I love to you know um

experiment with different things. Uh which is why I'm a jack of all traits and I end up having uh you know a lot of uh breadth about a lot of things but not enough um you know not enough depth. So about my credentials have uh about eleven plus years of experience in uh Python uh built over uh 20 different products using you know just Django alone. I scale more than five products to serve more than five uh you know to serve nearly uh millions of requests per second I built a device farm, iOS and Android device farm, you know, which was used for security testings. And I was involved in building a Bitcoin mining firm with Raspberry Pi and custom ECC chips. It is sort of like an AWS for Bitcoin miners if you will.

And as for my profession, I work as a VP of engineering at Reconcess. and I work as a Hitoshikage at a non-profit called Timis. So let's get this question out of the way. Why GraphQL? It's much more standardized than the fragmented restful implementations. you know everybody have their own flavor of restful implementations and uh and you know it's it's all about their own preferences but that the same thing cannot be done in graphq world um GraphQL has this very specific stretch of format that you'll need to follow. In GraphQL you get you get a live and browsable documentation out of the box. There is no need for you to you know maintain a different uh AP documentation like you would do with uh swagger or or any other thing which is used for REST. There is no back and forth communication between the API developer and the React developer.

So usually every GraphQL endpoints give a uh graph iql explorer where you are free to sort of like uh you know explore all the queries and mutations and subscriptions and types and whatnot And and you can write very precise queries. In Restful Framework, it doesn't matter if you just need like two fields or if you need a bunch of fields or if you need like n less nested levels of fields it'll always return you the same response. But that's not the case with the with GraphQL. You can write like very precise queries you can request for what you want uh if if you want n degree of you know nested uh uh response you you you are free to do that assuming you know graphQL allows you to

uh do such queries. So let's take a look at our very buggy demo application which which which I call you know the Stack Open Flow. It's a very silly clone of Stack Overflow. So let's go and register ourselves a user for now. I'm just going to call them uh test user which is the username and test user which is the password let me register okay it says now it is successfully registered and I'm going to log in as the test user So as you can see there are no questions on our system at the moment. So let's go ahead and add a question.

How to do Join operations in I can to join an array of text in Python. Fine so This is a sample question that we want to ask. As you can see, as soon as the question is added, it says there are zero upwards, zero downwards, and there are no answers So let's go ahead and add a new answer. Let this answer be uh you know uh foo bar and then we just simply

join we just simply do a space dot join of one bar this is going to sorry this is going to do a join operation on RE so let's submit that answer So here's the first answer, right? So the first answer which is given by me, the same test user, is you know join foo and foo. You can add comments So if you see there are no comments here, I'm going to add a new comment saying thank you submit and then and there is the comment fine and you can upvote on a comment or you can you know

down vote on a question Uh right. So that is it. So that is our entire application. Now that we know what our application is all about, let's dive into the text stack that we have used here. So it is on on the back end, it's built with you know Python Django. I've used this library called the Graphene Library which is used for GraphQL implementations. and there is Django GraphQL JWT. It is sort of like a it's a it's a it's a plugin for graphene library if you will where you can have JWT uh token based authentication And we have used Django rules. I just put the Django rules as a as a uh as a hint here, you know, as sort of like a recommendation that you could use that uh

upper object library upper object permission library that you could go for and I've used the Postgres database. On the front end it's uh JavaScript with React and I've used relay and I've used ECPC state management library So graphene have custom fields. The code is synchronous. You can only write synchronous code with graphene. It's more mature. It is the status quo and it has a fantastic integration with Django. On the other hand, using strawberry rocks, uh the schema definition is completely based on data classes. It's it's asynchronous, uses async. io. Uh it's not so mature, uh you know, it's it's being a relatively new library and all. uh m maybe it is the future because it looks really nice.

But at the moment it has a very minimal Jackboard integration so it made sense with to go with the graphene Let's take a look at per object permissions or role level permissions if you will. So choose a rule-based framework rather than choosing a database-based framework. A rule-based framework like Django rules would not hit your database for checking if a user has permissions or not because it's completely based on rules and you can just represent within Python code. But in libraries like Guardian, you will have to set these permissions on the on the database. So for each permission check you make, it makes a database call So which means uh Django when you're using Django rules there are less DB calls, which means less latency.

There's no unnecessary migration due to change in logic. So, uh permissions, right? They change every now and then. You you y you come up with a team strategy and then you decide uh, you know, maybe that's not the right thing to do and then you want to change it to something else or you add new roles. Uh so when you do things like this, right, you'll have to go back and add migrations because all your permissions are laying on your database now. So which is Which is something that you can s totally sidestep if you're using a library like Django rules and it's easier to maintain uh you know compared to uh frameworks that depend on DB2 So let's take a look at our project layer

So this is my backend repository and as you can see I have a SAG stack overflow root root up here. So basically the Stack Overflow open flow is where uh is the one that has you know your uh settings. py and wsj. p so if you have read uh books like two scoop of django they uh recommend you to keep configurations in a separate folder like config Which is not something that I like. It's just a matter of a personal preference. Is it that I like my things being namespaced? All the settings uh you would not access namespace taxes it, you will just import it from Django. conf Um so I'm just saying it's nice to have everything within uh you know, under the same name under the same namespace and that's sort of like the um

uh uh you know base layout that we've been following here at reconses for all our projects uh so that is that And and inside Stack of Up OpenFlow we have uh you know apps like Contrib, Core and QA. say stack overflow graphql uh as you can see everything is namespaced in the stack overflow uh which is something that i like And then typically right when you create a Django app you would find admin. py, apps. py, models. py, test. py, and views. py and everybody knows what these files test or else I have no clue what you're doing at DjangoCon So but in R modules we have these files.

There is admin apps which is uh which is by default that's there and then behaviors is where you keep your behavioral logic like commentable or uh votable uh things that you're able to do as a user right uh so that's that's where it goes into behaviors. py and then choices. py is basically uh uh you know where you keep all your database choices. Inputs. py is like form startpie uh you know when when when you want to get a bunch of input from the client side uh that's where you put your definitions at and everybody knows what models. py is and then mutations. py as uh so if you notice there is no views. py so instead of views. py there is two more files now which is mutations.

py and queries doty So the mutations. py as the name says it has all the mutations and queries have all the queries and then we have types. Types is sort of like your serializers, right? uh it contains all the definition all all the payload definition that uh that graphql uh responds back to the user with So let's walk through the source code. First I would like to walk you guys through the backend code, right? So let's open code. py and in the code. py let's go through you know some of the files. So basically we have this choices. py This choices. py is nothing but uh you know all the choices, all the database choices uh

that the models have. So for example there there could be an admin type of user or a client type of user There might be different types of uploads. You can upload a profile picture or you can upload an Excel report or something like that, right? And then I have an upload status. I have an upload status which says what's what what's the what's the upload status. So basically this choices. py file has all the all the db choices And in the models file you can see there is a base model. So basically we use UAD as the primary key and then the user has a user kind And then there is an upload object. There is an upload model which is used for tracking all the uploads that are done to the system. We have our input startpie, right?

Which is more like form startpy it'll have uh just you know uh it'll say what will the what are all the inputs that the graph ql uh expects So for instance the register input right. So at the time of register we only require username and password. So if you were to look for the register input uh you would find that there is username and password. Feel free to ignore the client ID mutation which is nothing but a relay uh which is just which is nothing but a property that's added by relay so you can save and ignore it And then we have types. type. So every model typically will have a type. file associated with it. It's like gra Django serializes. It's like Django model serializes if you will.

So here here is for example the upload rate. The upload in in this place it can have a pre-signed post URL Generally speaking what people will do is that they'll upload from their browser to their server and then server uploads CS3 and then returns that response back to the browser. So let me tell you why that is very problematic. Say you just have two servers, right? Um and and there are uh there are hundreds of people trying to upload huge files and that's going to be a problem because everybody is going to be uploading to the same file. And let's say you also have to redeploy your code during this time. You know it's not easier to do that because there is active uploads that are going on So that is something what that we should actively try to avoid. That is uh the indirect upload to S3

through uh the servers, we should try to avoid it. So what I usually do is I I create a pre-signed URL I let the browser upload the file directly to S3 and once it's finished uploading it will make a mutation call. So let me just walk you guys through the mutations here So this is the mutations files here. So as you can see there is a register mutation mutation which means it's the mutation that uh you know sort of like registers new user. I create new user objects here as you can see and then there is a create upload and finish upload. So the function of this create upload and finish upload is that when you say create upload it will create a new upload object and it will respond back to the user with an upload object And what we do is when we are getting this upload object, since the upload the return type of upload object is

you know is an upload type The upload type if if you look at the upload type, the upload type will automatically have a pre-signed URL. So when it resolves it's going to generate a pre-signed post URL and it is going to return it to the user So when we get the pre-signed post URL on the browser, we are going to upload it directly to the S3 and then we make another call saying so once it's finished uploading to S3 we make a call saying finished uploading it. So In this finished upload mutation we are just going to send the ID of the upload and it will immediately set it to uploaded. Mm and then it'll just end. Uh and then if there are any uh so this at the moment is not a valid task, it's it's not actually there, it's there just for demo purposes.

So if you want to do some post processing, right? So for example you uploaded a file object and you want to you upload a profile picture and you want to optimize uh you know the size of it using image magic or something that is something that you could do within uh you know within tasks like this So that is all your mutations and then we have all our queries. So queries are nothing but you know all the fetch calls so that's where you fetch your data from and mutations is when you alter something in your database so that's the difference between queries and m uh mutations basically so we uh so for uh So we have like me query which means it will return the currently logged in users input.

We have the uploads query and we have the relevant you know resolve methods for each of these things So these are some these are sort of like you know additional files that we'll be creating when we are using graphene Uh but uh the score app is a bit special. It has even more uh additional things as you can see it has decorators. py which is basically it has all the login required decorators you know if the client require decorators and so on what uh whatever decorators you need you can just keep it over here for f say for instance I'm going to call uh I'm going to query uh all the content types in the system and this content types will have an ID label

app label and model so when I run this I get a bunch of responses back And then we have a storage. py. The storage. py is nothing but it just has a bunch of regular helper functions to sort of like upload uh upload files directly to S3 from browser and then uh generate pre-signed URL keys uh and and so on right so it has all the helper functions to interact with S3 basically so that is it So this is all uh this is all the files that are in the core. So let's look at all the files that are in the Q<unk>A. QA is nothing but you know it represents all the questions and answers that are used in the system uh question answers votes and comments um so let's have a look at model

right so basically we have we have things like vote comment questions and answer Uh the vote as you can see it's a generic foreign key because you can apply vote to any object like answer, question or comment. You can use the same table for all the three uh entities. So that's why it has a the that is why it has a generic foreign key. And the way that you refer it from your comment from your referred object is that you use a generic relation so as you can see here we have used a generic relation and we said uh votes is basically a generic relation to bot fine and then we have questions which is which is basically what questions and then we have our answer entity

which stores all the answers and as you can notice all the models have custom managers Fine which is defined in behavior start by. So if in behavior is if you were to look we have a bunch of things like votable query set which is uh you know any object any object uh uh that that the user can vote on can exhibit a votable behavior. So it has a votable query set. It can have an upward count, downward count and a total vote count And then we have a commentable query set, which means you know these are the these are the models on which the users can comment on So that is it. And then we have a vote country, which is nothing but a case-when

statement to count the number of upvotes and downwards So this vote count is nothing but just a complex query which will count the number of either uh g the the given vote kind which is either you know here as you can see it's either up vote or down vote It will collect all the number of votes it has and and then fight till finally the total vote is going to be the number of upvotes minus the number of downvotes. So as simple as that uh so that is it and and if you were to look at uh the queries dot pi uh this queries dot pi is now going to uh you know uh So since you since you use query set as managers, you get to do things like this with vote count and with comment

count. Right. So this thing will annotate your query set with relevant data so that uh it can be picked off uh on the client end as well. I've seen many people uh you know struggling to represent generic foreign keys in REST or APA calls A neat little trick to take care of it which is like uh resolve comments. So this resolve comments basically will take an object ID and the content uh content type ID and it'll it will get all the all the comments for that specific uh object ID and content type ID. So how do we define this content type ID? If you were to look at the core queries. If you were to look at the core queries, all the content type are listed here.

So let's just play around with the GlafQL Explorer for a little bit before moving on. So as you can see All the queries and mutations that we defined in those files are are here. You know, every queries and mutations that we defined are here. And let's try to make a sample query, right? So this you can forget So we are going to take all the get all the content types first. So when you run this, as you can see, it gets all the content tape button. So it gives like a really nice documentation that you can explore around with and you can also play live with whatever data that you have on your local Uh so this is a really this is a really um handy tool to have. So we also have history here on the on this history button if you just click on the history it will show all the the past 20 or so uh you know

queries and mutations that they have done. Uh so let's look at this input mutation trade. So this is a login mutation. So for examp for example I'm just giving a username and password which is hard coded here and I'm just going to run it and as soon as I run it run this mutation I get a token back and uh you know any further uh calls to the log uh to the views that are protected by login required we send them with uh JWT tokens. with these JWT tokens. So that is all about the client side. That is all about the Django server. Let's take a look at the front end app which is Stack Overflow app on my on my local.

This is where I have it So I'm already inside the source folder. I'm not going to worry about the things that are outside the source folder. We are only interested in things within the source folder. So as you can see, I have a command here. So let me just show you guys that command. So I have uh app export fine have this uh I have this management command called app export so when I run it it's going to It's going to export the choices. js and the schema. json from our repo to the React repo. And in the React repo you get to see what are all the choices. So uh every choice that you declare in choices. py is now available in your choices. js so when it is exported like this you can do things like

uh routes questions So this is a vote component, right? So this basically has a vote button. It has upward button and down vote button. So basically, what I can do is that I can import these choices Fine and I can just say choices. vote kind dot down or choices. vote kind dot up Okay, and then let's now uh you know have a look at uh the non-production deployment which is using Bika. Again, uh the the source code is located right over here. I have forked it. Uh let me just open Uh DJ Con Eo began. Why should

DJ Con E 2021 began? Okay, so I hope the font is visible. So this is nothing but uh uh you know this is nothing but a generic Docker Compose boiler plate. So if you were to open this uh Docker Compose so So Biga is nothing but just a generic Docker Compose boilerplate which I which we use to deploy you know for non non-production workloads. Uh the reason that we use uh Docker Compose is that you know we don't want uh most of the clients that we have uh

they they don't have huge or deep pockets most of the times. you know, they spend money out of their own pockets to sort of build their own product. They don't they have not even raised funding. So we don't want to put a lot of bill for them. I mean we don't want the them to get a lot of built from AWS right. So what we do is just we take a $5 server. We create a $5 server through uh uh you know Docker machine and we deploy it using Docker Compose. So this is like a sample configuration of the Docker Compost. Right. This is not the actual thing like we don't use this these services not so it's just here for a reference reason but yes you can see there is going to be a worker uh instance uh which will uh you know which will build your backend folder

And then we have the configurations to run beat and backend from your worker uh from your worker service. This year we use this thing called traffic, which is a which is a pro which is a reverse proxy The advantage of using traffic is that you know when you're using it within Docker Compose, you can just say where the sock is, where the Docker. sock file is. So once you mount it onto the traffic, uh it'll automatically uh you know load balance within your containers, it'll automatically reroute all the traffic. Uh you don't have to do much of any configurations to uh you know like you would do with Nginx because this is cloud native, it is supported by the CNCF Foundation. If you guys have not already checked up checked it out you should really check it out

uh so this is a project that's worth checking out so all the documentation is is you know it's It's clearly uh it's it's written step by step over here. If you guys have any doubt, be sure to write me an email. I'll be sure to respond to you guys And then AWS Copilot. This copilot is an interesting utility, right? And then we have and then we have AWS copilot for all the protection deployments. So AWS copilot is basically a helper CLI tool which will help deploy your applications on ECS on AWS Fargate. So AWS Fargate is basically a serverless container

You know it's very level and uh and it's very easy to scale because it has like all the scaling configurations built in. You just have to specify the threshold for your CPU and your memory in your manifest files. Once you do that uh as as soon as your uh as soon as your uh you know usage your your resource usage uh meets one of these uh meets one of these as soon as your research resource usage meet uh crosses one of these thresholds it will automatically scale your containers So it's like very straightforward. So it's always software engineering is always about trade-offs, right? So for being vendor dependent, uh you lose uh you gain you gain a lot uh in terms of scalability and reliability

Uh and and this is very useful for teams like us because I've always exclusively worked with small teams, uh which usually is uh doesn't go more than ten people And you know, where uh where we don't have a lot of specialized roles. Uh so this is like really handy for people like us And it's cheaper than Lambda serverless functions because I uh it's not only cheaper, it also has uh low latency compared to Lambda functions. And as far as the and as far as this uh single-page application deployments are concerned, don't let your server uh you know handle static files. Uh it 's an unwanted load on your server.

You know it's it's going to introduce some latency and and it might not be that reliable way to do it is just uh you know build your uh build your single page application push it onto s3 have it uh proxy through a cloud front uh cdn And then attach a ACM certificate to your CloudFriend CTN if you have a custom domain which will give you free SSL certificates. So this way it's much cheaper than alternatives like Netlify and it has lit l less latency compared to when you're trying to serve from your own servers. And and then there are different ways to go about it. If you want uh if you want an automated deployment again this comes from the same big a repo. uh DJCon U2021.

So if you were to visit the Biga repo uh we have a S3 front-end deployment script here which is like manual thing and I have a Pi invoke task. So this is my PyInvoke task for setting up ACM, S3 and CloudFriend And and this is the same thing, only uh you know this is done manually. There are about some 54 steps. So that's it folks. So I guess that is uh all there is to talk about So my handle is Dilipshiva on both uh Twitter and uh link on GitHub. You can reach me uh using the handle. Or if you want to mail to me, it's dilipshiva. pm. me uh or if you want to text me it's it's telly I'm I'm also available as Dilipshiva

on telegram. Thank you so much for your time guys.