Just-in-Time Development with Django and HTMX: Faster, Leaner, and Smarter

Speaker 1: Hello everyone, my name is Thomas, but I'm not here to tell you my name. I am from Belgium and I've been building Django applications uh for over a decade. uh mostly one um long-running application that I started uh 14 years ago. Uh today we're going to talk about uh frontend which is an emotional topic. I get it What I'm sharing today comes from my own journey and my own needs. If your stack works for you, that's great. No judgment. Uh HMX is a front-end library of peace. Um that said, using HMX with Django is according to me the best way to uh create modern-day web applications

Speaker 1: And over the next 30 minutes I'll show you why. Let's maybe start with uh what makes an app feel great to use. So I created a sort of Mental framework um which are called the true the three pillars of great user experience. Um first one is design. So is the interface visually consistent and pleasant. Second one, clarity. Is the app easy to use and to understand? And third one is speed. Is the app uh does the app feel responsive and uh fast? The pillars are not equal. So speed is the most important one. If your app feels slow, uh your users are going to get frustrated, uh

Speaker 1: no matter how pretty or how clear it is. Clarity it comes second. User need to understand what is happening or what they need to do, where to go. If not, they feel lost and they give up. And design is last, is still important, but a beautiful UI uh means nothing when it's slow and confusing. Um here's a little example that I find uh fun to watch at. It's like Windows 98. You see, like it's it's it's kind of clear what you can do, what what what you can click on, what accepts input The design is ugly, although I kind of like it. And then on the right you have Google Calendar, which is um the design is good, it looks good, but like

Speaker 1: what what is clickable, uh what accepts input. It's not very clear. So clarity is more important than design. So the page refresh. This is the villain uh of the story. This is what basically divides front-end developers and basically all web developers Yeah, the front uh the the page refresh is uh our our brain really hates it. Um whenever it happens like the page flickers for a second. It's it's kind of like when you're reading a book. And you read like a phrase or a paragraph and then the book closes, reopens again, and you kind of need to know where you're at again. That's uh that's what it feels like.

Speaker 1: Why do we have the page refresh? Why where does it come from? Who invented the page refresh or or who committed the this this merge request, you know? Well actually it's um kind of a quirk of HTTP and HTML um back in the day when HTML was first generated Uh it was uh you you had documents with links on them and you click on a link and you go to the new document and uh that's how you have the page refresh. It was like that. It was good in the beginning. You know, in the 90s people, you know, you had information like wow, and then you could click and you could go to a new page. But it sort of got annoying and people uh were trying to find solutions for it Why is this still here? Why do why why is why has nobody fixed the page refresh?

Speaker 1: The thing is with HTML is that it's kind of a public domain technology. And if you want to change things in HML, everybody needs to uh agree on this. And um that's that's that's why HML over the years uh has not evolved at all or it The core HTML it looks like it's still from the 90s. Back-end technologies can evolve much faster because the uh the browser doesn't really care about what back-end technology uh you use you can write your own backend code and it will work but front end everybody needs to agree so that's why it's slow. So then people started uh finding workarounds. Um and uh yeah there is different types of workarounds which we will basically look at.

Speaker 1: So in the 1990s you had HTML, which was great. Then in 2005 Ajax came around. It was apparently Google who started with Gmail. So um you had the possibility in Gmail to suddenly like edit an email, send it, delete it without refreshing the page, and it felt like an application. Yeah, people thought it was great and they had to um uh yeah they wanted to continue this. Everybody wanted to continue this. Like nobody likes the uh page refresh of course. Then in 2009 you had uh jQuery come around and these dates are more or less, you know, maybe one year or two years uh

Speaker 1: wrong But more or less during that time jQuery came around and this was kind of a sort of wrapper around the complicated vanilla JavaScript, let's say, and uh Ajax and it was kind of made everything simple to create these interactive uh nice applications. And this is when uh my journey kind of started in 2011. I started building a uh software management uh business uh system And uh yeah of course I want it to be a bit interactive and I started using jQuery for the interactivity. When you click like a save button, you don't want the whole page to refresh, you just want that little part to be updated. and to be very fast. This I I I dug into the code for from from a

Speaker 1: long time ago and um I looked at this. Yeah, who who has written code like that? Who still has code like this in their code base? Who is still actively writing like this? Um I still have this in my code base because um I refactored it but some parts are still like that and um Yeah, so basically what this is doing is on the top you have something happens, an event happens in the in the UI, in the in the client side. Then you prepare the request. With Ajax, then you make the request. So uh you yeah you make the request to the backend and then if it's a success you change something on the screen

Speaker 1: and um if it's a failure you change something else to the screen. screen so this is basically how I uh and and a lot of people back in the day were writing um interactive applications it was great but it felt Very hacky. Every time I was writing this I was like, what is this? Is it it it it worked, but it was very dirty. I don't know. When I was writing this I But it worked. That was the only way basically. And then React came around in 2013 and everybody was talking about it and I had my application that has that I was building for a couple of years with a lot of code in it and I was like, should I move to React? Yes or no? And I was looking at this code and I was like this is so ugly. This is a mess, I cannot work with this anymore. I need to go to React.

Speaker 1: React will clean this all up. Thanks to React, everything will be good. Um so I tried React. I for for one month I did a tutorial of React and I was like okay let's let's do this and I ended up with something like this and I was like this is not easier this is What is this? I I couldn't wrap my head around what the concept was because I was used to jQuery templating. I just thought it was like a sort of framework around jQuery or whatever So I started doing this and then I re- I realized what the impact was for my application to go to this. Uh and then yeah, I decided To go back to Jacobine. So I I I I accepted my fate. I was like came back to

Speaker 1: to the old love, let's say Uh and then I just tried to um make it better, you know, make it cleaner, try to have some helper functions with uh jQuery. But it still felt really hacky. But I continued to do it Until one beautiful day in the summer of 2021, I was looking at a tutorial and suddenly they were talking about HTMX. And the moment I saw what HTMX was, I understood that this was a solution. And really it was like an eye-opening moment. Like as if the sky opened up and the sun shined down on me and God gave me this gift. It was the most beautiful day, one of the most beautiful days of my life

Speaker 1: So I started um refactoring all my code base and uh yeah I'll show you uh that after There is a funny meme uh that says kind of what I just explained. It's kind of I've it's kind of what I had as well. 2004 it's like very simple, let's say, then 2019 you had this complicated uh front end framework uh uh mania I I'm I'm not gonna say that but uh lots of people were using it and then HTMX came out and everything was simple again. Um going back to Ajax, um who knows what it stands for? Asynchronous JavaScript, indeed. It stands for asynchronous JavaScript and XML. For the longest time I thought it meant asynchronous

Speaker 1: JSON and XML or whatever. I thought it was JSON. I thought that Ajax has to be JSON API endpoints, which is not true because of the XML Uh HTML is actually XML. So I think the name should have been Ajar, maybe with a H so that people would have understood, oh you can do HTML with it. And we might have not gone this route, maybe 10 years ago we would have realized you can actually return HTML. And you could, but nobody really realized until Carson Gross came around with HTMX. And basically uh explained to everyone that you can or made it clear to everyone that you can have API endpoints with Ajax.

Speaker 1: And there's this nice phrase, it sometimes takes a genius to point out the obvious, and uh it's really clear uh here that Thanks to what he made with HMX, that everything was clear all of a sudden. There's also a nice meme that explains kind of this whole slide. Which why go from DB to JSON to JS to HTML when you can go from DB to HTML straight away? So the just in time uh philosophy Basically it means to only fetch the data you need when you need it. Forums, models, drop-downs. Also fetch try to fetch the data in an asynchronous way.

Speaker 1: You also need runners, which need to be very fast and lightweight. And I'll explain what runners are after. And you also have lots of small API endpoints that return HTML. So this is this is the application I've been working on. It's a modern application, usually looks like this. You have like a side menu, you have a top menu, and you have dynamically loaded content in the middle. The idea is that you don't want to reload the side menu and the top menu. You only want to load the content because it makes everything faster. And there's a nice way to do this in HTMX. So when you load HTMX, it's library, so you just load in the the the the j uh the JavaScript. And then you can do hxboost true on the body tag And this will make all of the links

Speaker 1: all of a sudden work with Ajax or JavaScript. That's very nice because if you have now a link, you can basically say okay this link what returns from this link from this API endpoint you need to put it in this target which is the main content. So the main content is the thing in the middle. So whenever you click on this, put whatever you have in the main content. And what is nice as well is that inside of your templates What you can do is you can say okay if this is an a request made with HTMX, then we are going to use the base HTMX and not the real base. The base is holds everything like the whole DOM, the JavaScript. the CSS everything, but the base is actually super small.

Speaker 1: The HTMX base, sorry, uh only holds um the content so that way when the backend sends something you can ignore all of the Everything outside of the content so really only send through a small part of HTML, which makes the request super fast because you're not bloating every request with a lot of uh data So what are data fetchers or runners? It's also a sort of mental framework that I created is so that whenever a request happens in the client You have like a runner that runs to the server, gets what's needed, and then runs back. Yeah, it's AI generated, so

Speaker 1: So yeah, and these runners need to be super fast. I was I kind of like this analogy because I was a waiter before and and and I was always running to the back end and and and and bringing things to the front So that's kind of what requests are. So whenever this is a traditional request. So you have the client, somebody uh on the client makes a um request, the runner goes to the backend. And fetches the dashboard and fetches everything. So fetches the complete DOM, the main navbar, the context processors, some the middlewares also. I mean it goes through the middlewares, uh the filters, the graphs, the JavaScript, basically everything. Which makes it which makes him super slow because he has to carry a lot of things. So what you actually want to do is you want to make your runner

Speaker 1: super super fast. And how do you do that? is like uh like I showed with HTMX before is you uh make him run to the backend and you only make him come back with small HTML, it has to go to the context process and the middlewares, which I'll discuss after. But basically what you want is that you have when when the request comes um uh back to the to the to the client you have the smallest amount of HML possible and the smallest amount of drag possible so that way your runner is super fast so you don't load the main nav bar you don't uh load the complete uh dub DOM you don't even load pieces that are on the dashboard

Speaker 1: What you can do is then on in that small initial HTML, you can create you can add some some some HTMX code. To go and fetch a specific HTML partial from the backend. So here we have a partial which is get sales graph. It's kind of slow, partial, because you need to go and fetch from the database all of the sales, create into a nice graph, send it over. What is good about this is that you can have multiple runners at the same time. So when you initiate your dashboard shell, you can make multiple runners and all of them run at the same time to the backend to go and fetch their small um their small HTML. What does that look like in um

Speaker 1: the code? So you have the dashboard shell. This is where you are going to this is what is being sent back initially. So it's almost nothing basically. It's 24 lines of HTML. So very fast, there is something on the screen for the user. And in there we have three divs and they all go to the backend to get the sales, the top clients, and the low stock widget. You have a little loader in there So that way when the user initially comes to the dashboard, they have some loaders and something is going on, and the three runners at the same time they go and get the data in an asynchronous way. This is what the backend looks like, so the view. So you have a very small view where you basically just render a template

Speaker 1: partial you could use, it's just a template. With some context and that way your runner is super fast and um doesn't have a lot. So What is important to note is that middleware and context processors really drag down. So try to keep your middleware as small as possible. And also context processors basically I would say don't use them at all Only use them for because context processors are run at all template um uh design rendering. But they they are really slow because they do it every time, even things you don't need. So if you have context processors only and and the context processors to go to the database, make sure it's cached, make sure that there is as least amount of drag possible for your runners.

Speaker 1: Um well yeah, there is a live demo. So this is the dashboard I just showed with the same partials. And um as you can see when you click on the dashboard It loads in all of these individual. Oh sorry. So like this. Do I have to go? Okay. So let's go let's go to the dashboard. Let's load the dashboard again And as you see, it's really super fast. You just click it and instantly something happens. And this is really nice for the brain, because something is happening. Because if you have to wait People they unconsciously think it's broken. Did I do something wrong? What happens? And then boom, something happens.

Speaker 1: That's not really nice for for for the cock uh for the cognitive brain What you want is have something happening and here all of these little cards they are kind of slow to render, especially if you have a lot of data in the database. They become slow. So you don't want to wait for all of them to be rendered and then render them. You want to have this this asynchronous way of showing Content on the screen. Okay. Um And and this concept of um fetching data in a sort of asynchronous way, we have implemented this kind of everywhere in the code.

Speaker 1: Not only for rendering partials, but only uh also for When the user is requesting something. So here we have a um a button which is the add contact button Usually when you are on a list view you have a add button. What we do is we show a model uh pop-up and uh here When we show this button, there is nothing yet. This is the only thing that has been loaded on the screen. And it's only when the user asks for the form that we go and fetch it and show it to the client And I'm gonna show that. Can I do like this?

Speaker 1: Sorry, I have to do it again like that. Um what also is nice is I mean what I can show you is uh oh sorry it's in So we're seeing quality So this is what what I showed you just before was localhost. This is live. So even in live when you see It's like super fast and these runners they kind of run through internet cables or um below the sea and so it's it's like almost instant and this is really what you want in an application. Um Oh

Speaker 1: yeah, sorry about that, but So what I want to show you now is when you are on the list view, list views in general are kind of slow and heavy and you have like a filter, you have a button, uh add button, you have a lot of things. But LisView 90% of the time people they they might not click on the add contact button. So why would you in the back end look preload this? Because it's not gonna be used anyways It's slow slowing down everything your users gets annoyed. So here what we have what we have done is when you click this button, it will make the request to the server to get the form and then it's being rendered like that. So that's again for speed.

Speaker 1: One thing that is nice as well that I'm gonna show you is We have re- we have kind of recreated the Django admin when you are on a form and you add like a foreign key, it opens like a pop-up window. We have kind of created that as well here. So when you create a product you have the initial form that loads you have like this little plus and then you click on it and you have another form that kind of opens up and where you can add a test category You add it and it gets created here in the form of so that's also something that's super easy to build uh when you use this technique. You basically You you basically have endpoints for specific HTML and um data and you can just add them everywhere in your code and it's it's really

Speaker 1: it's super easy. So as you remember before that uh jQuery example that I showed, this is kind of the same but with HTMX. So something happens, a a click happens You prepare the request, so you show something on the screen, so something is happening. You make the request to the server and you change stuff on the screen. So that's ugly line of jQuery of 30 lines of code suddenly was uh changed into four lines of code and as well on The button, not somewhere else. So that's the beauty of it. You write your code where things happen.

Speaker 1: And that's really one of the most beautiful things about HMX, is that it's it's just right there. Um yeah in the in the UI you also have uh of course a um a filter. Let me just do it like that. You have the filter and these filters sometimes they can be very heavy. You know, you have maybe 5,000 contacts, and when you load the filter and you don't do it properly, you're loading 5,000 selects. You shouldn't do that, but Even then sometimes you have 30, 40 select options and that takes a long time to render. Again, maybe the person is not gonna use it. Most likely is not gonna use it. So why Preload this to s it's it's it's it's slowing down for nothing. So what we have done for the filter

Speaker 1: again it's you have an endpoint for that filter and there the HX trigger is Whenever the page loads, then we are sending the runner to fetch the form because that will make the initial load fast. And whenever the user is ready to click on it, it just clicks on it and it's there. It makes two requests to the server that you need to take into account. But that's normally not an issue. What we also do is we this is a nice nice thing that we have is we have like generic endpoints for forms in the backend. So you just have this URL, get generic form, you give the name and it will return you the proper filter. I'd like to go more into details about that, but that's for another talk.

Speaker 1: So these are the most popular filter um triggers we use. So whenever you click that the request happens. That's that that can be interesting like for when when you click on the create, only then it it it happens. The load happens whenever the page loads, so that means you can um uh yeah hide the slowness basically by making two requests. Um you also have mouse over which is nice so only when the person moves over A button that you load. This is handy for drop-downs, for example. Only when the user moves over it, we're going to load whatever is in the drop-down. There is a lot of techniques and and and and and like little gold nuggets in HTMX. That uh I will not be able to cover today. Um one nice thing as well is if you have these

Speaker 1: um uh these the the the cards for example on the on the dashboard what you can do is i think i can do it like this yeah what what you can do here is when you return something when you return a response from the from the server You can basically put an HX trigger and this will when this uh request arrives to the front end it will basically tell that runner hey go and update that card. So that's that's really beautiful, that's really nice. Um so yeah with HMX um what you see is what's happening on the screen. There is no magic layers, no hidden files controlling the UI, uh no layers of abstraction. Just like Leonardo, painting directly on the canvas, you need to be able to work where you see

Speaker 1: the result, immediately and tangibly, and with without layers in between. And when you're developing like that, you basically move faster with more confidence and your creativity flows and most importantly, you enjoy building again. Thank you for listening and enjoy the rest of Django Com.

Speaker 2: Thank you. Very nice talk. Um I was wondering do you have uh public sources uh resources for diving deeper into the topic that you can rec uh

Speaker 1: the HTMX website itself is really nice. Just go to HMX and there is like a ton of examples, ton of uh I I started doing this technique in 2021 when I mean this way of working in 2021. So I've built over the years kind of like my own custom views in the back end to handle this and not everything was clear in the beginning but over the years I developed like some sort yeah views in the back end like generic views that are really nice that that make it possible to have like Just small pieces of HTML that do things in the back end. I mean to make it super simple actually to to develop. But this is not public. So this is our uh proprietary code base, let's say, um that we don't uh uh share but

Speaker 1: once you start working like this it should be super easy to to to to get going I would say

Speaker 3: Hi, uh thanks for the talk. Uh so how do you manage uh state? Uh what I mean by that is let's say you're sharing uh a particular URL. And you're pointing out that, you know, if you follow this URL, you could reach at this particular state. Like let's say if you are sharing uh Amazon's comment or something, or let's say you have uh you you're reaching out to a point uh with a ID on the uh HTML element. Uh with this dynamically loading stuff it it becomes more like an app, right? You cannot actually statically link to any of the state in the page.

Speaker 1: But the the the thing is there is no state with this way of working. The state comes from the back end and the state actually is the HTML itself. So you can start working with HTML itself, which is actually in XML, which is a sort of JSON format, which is the state And when you start working like that, it's kind of a mind shift that you have to have, but basically everything happens from the back end. The front end is kind of dumb and everything is happening in the back end which which means you have only one state which is the whole beauty of this which means that if you want to do something you can just do it in in in a matter of minutes like instead of with Again, I'm not very familiar with front-end frameworks, but I heard that when you want to implement a new endpoint, you need to agree with the front end to do this. And here with this way, you basically do everything and you are in control and everything is just right there.

Speaker 1: super easy. So yeah, that's my answer.

Speaker 3: Just to follow up that so let's say if you if you are pointing out that you want the person, uh client customer to reach uh to one particular point in the page But that ID is not loaded unless and until you particularly do a kind of action. You know what I mean? So if you want the page to scroll down at the bottom Yeah, so

Speaker 1: I'm I'm not sure exactly what you mean because I'm really not in the same mindset as as as working with like two fr front ends. For me this is Only thing I know basically how it works. So I know it's just the backend that sends everything to the front end. And then in the front end there is not really a state. We I we kind of use Alpine. js for some tiny state management, but Almost nothing, so I'm not sure I can I'm the best person to answer that that that question.

Speaker 4: Hello. I have a value pack of questions. uh three for the cost of one. Um I have a small a big app uh in production with uh HTMX and we had struggles on security because at some point we decided to add a CSP. Have you had this problem too?

Speaker 1: A CSP? What what do you mean with that?

Speaker 4: A content security policy

Speaker 1: Uh but what was the issue?

Speaker 4: Um every HX uh attribute on HTML was uh um treated like inline JavaScript. So it was uh

Speaker 1: forbidden by the browser or by the

Speaker 4: by the browser because you had the CSP. So uh you didn't meet that problem in real life?

Speaker 1: No, I don't think there is uh any security issues with uh with this because this is actually HMX is just JavaScript. So

Speaker 4: Yes, but it's just in like JavaScript in reality, so we had uh to extract a lot of code uh from the HTML to put it back in JS files So it was a bit disappointing, but we could manage to make it work.

Speaker 1: But there is no need for that. I can

Speaker 4: show you.

Speaker 1: Yeah, I'd love to see it

Speaker 4: And uh my colleague is a bit disappointed because we have a lot of very small views and small woods who do which do uh only one thing I like it, but he doesn't.

Speaker 1: It's not how they were taught or it's not how they do things, which is normal, you know. It's like there is a million different ways to build a house. Uh some people don't like this way and do it that way. Uh it's Whatever you prefer and and for my use case and for how I work, this really works uh wonders.

Speaker 5: Well we don't have enough time to take more questions, but you can find the speaker after. Or at lunch?

Speaker 1: Yeah, indeed.

Speaker 5: Thank you again.