Paris Kasidiaris - DjangoMaxxing: Serving Thousands of OTLP Requests per Sec...

Speaker 1: Okay, so good morning everyone. Uh thank you very much for coming. Today I uh I mean this is the presentation I've been most excited about for uh lots and lots and lots of years. So we're gonna talk about uh Django maxing. I had this um you know uh th this term came to my mind, you know, many people talk about, you know, token maxing, uh Euro maxing, all those things. So I was saying Okay, uh wha what is Django maxing? And it's doing the most possible things with Django and utilizing it to the to the greatest extent. because usually people say hey you can use Django to build a web app but you know when

Speaker 1: things get serious you you know you get it out of the way so you can do some real work. So what we did is we Django maxed it to see what are the limits of Django and we built a single process OpenTelemetry server with Tursum. So I'm pretty sure this is something that no one understands here because it's Greek. Is anyone by any chance? No. I said so. So I will translate this for everyone. But this is an eye message from my business partner Antonis who told me I think we have to remove Sentry from PR environments because it's eating all of our quotas. So um you know maybe Sentry is an observability tool um that you can use to track exceptions.

Speaker 1: application performance metrics and all those nice stuff. And this was sent to me on 13th of October, maybe of August, maybe three days before the deadline for the talk. So mm I thought okay, um he he has a point, maybe we can self host something. But um everything seemed so complex um so I mean even to self-host a sensory it wasn't an easy task. It needed a few gigabytes of RAM and I thought okay Let's find out. What can we build that can get us through the PR development environments, uh observability, exceptions, logs, metrics, all that stuff, and maybe production.

Speaker 1: So We came up with this name which is called Fakos, it's a Greek word, it's cringe, I don't like using ones, but I was looking for an easy domain to buy to buy. Some people say me this might sound vulgar in English, so I told them this is going to be the uh the crash test. You're gonna tell me afterwards if this is true. And now getting to the point. Uh actually before I get to the point, um there's So many things I can talk about in this talk, but um I try to laser focus it on one topic that we will see in a bit. So What is FACOS? It's um an open telemetry server. It's an observability tool that works in one process One Async IO loop, one Python thread, there's no thread pool,

Speaker 1: there's no database server. It's the simplest thing you can run. And it can serve thousands of open telemetry requests per second This means that you can send more than one thousand logs, metrics, traces uh combined per second and it's going to process them successfully. Now uh a primer in what is open telemetry because um I've said this I don't know maybe eleven times so m so far and some people might not know it, uh they don't have to. It's an open standard for telemetry, again, logs, metrics, traces, exceptions, all this nice stuff. And um Profiles also you know you take a snapshot of a process and you see how much uh is spent in in this function and this function and there and how much

Speaker 1: uh RAM and stuff that it's coming as well. And it's transported over HTTP or GRPC. We don't support GRPC so I struck it off with a format called protobufs. Now, what does Facos, the application that we build, do? It ingests open telemetry requests. So it accepts and stores them into a SQL table. It projects these um these requests into relational tables This is important. I'm gonna explain more about this uh in a minute and then you can query all those um all those nice data But we have before we go into more details we have to talk about the constraints.

Speaker 1: And this is Django 6. 1, it's the latest one. We have Python 3. 14 Although I'm waiting for Python 3. 15 in the next few weeks that we have uh that will have a few nice features that will make the code um quite beautiful. Um we're using only Django ORM. This was another thing that I wanted us to push towards to, not use any RAW SQL and stuff like that. Just Django RM, it's it's incredible. I don't want to write any SQL. I like it. We use um a SQLite database via an a new engine called Turso, which is a Rust rewrite of SQLite. So the fact that it's Rust it's um irrelevant to us but it gives us some nice features that we're gonna talk about and that we needed

Speaker 1: And the other very important thing is that when you send logs, metrics and stuff and we say, hey, 200 All good. This means that this is durable disk. So if this is on a physical server, let's say that you know you pull the plug this time It means that the data are there. So because telemetry can be critical to troubleshooting things, we want uh you know status 200 to be um reliable and this is the performance gate that we had in mind more than one thousand requests per second by the way this is a lot this is this is a lot having one thousand requests uh per second and um this is the the floor that we set to say hey um

Speaker 1: we're gonna go And before we get into the implementation, because hopefully I will manage to deliver the whole presentation, I want to give you the numbers up front. So so you know what's the status we are right now. So we were at one thousand one hundred requests per second for a mixed um for a mixed load, meaning all logs of metrics traces for five minutes straight. We accepted and stored all of them so this was quite good. So a as we try to tweak um how many because in one request you can have ten log log records, let's say, or more. So we were doing some tests to see depending on how many records we could have in one request

Speaker 1: we could increase or decrease it. So we reached up to 2395 requests per second uh when we were saying when we were sending uh logs only but um one log re log record per request. Um in this case where we send ten records per request We had 1536 requests per second sustained. But because we had ten records per request, you can think that we we stored ten times that in open telemetry logs, which is let's say four times um this one. Um looking into doing a search into 3. 2 million logs um takes about 25

Speaker 1: milliseconds And bringing a trace, I'm not sure if you know what a trace is, but it's um let's say you can imagine an HTTP request like In Eastern Europe you know the babushka that's the one inside the other and or f you know um envelopes that's one inside the other. And you say I have this function that calls this function, that calls this function, it's a tree-like structure So if we uh bring to view a trace detail with 161,000 spans, this is like a lot. Um it took 43 milliseconds. This is fast. Yeah, so this is I want to get your attention about what what we're talking about, and now we're going to see uh more the details and our approach.

Speaker 1: So the first thing was to minimize a stack. Remove everything. Remove the database server, remove uh threads, remove everything, one process and the file system. So all bottlenecks are now exposed And every time we remove the biggest bottleneck in front of us. But we wanna we wanna keep so I said user experience because I was thinking about myself, but I I I needed to say developer experience by the way. Um so we wanted to keep great developer experience so it's easier for us in the future to build features. Now The elephant in the room that I wanna talk about is that async. io is not a speed up in general. It's it's lower. I if if you take the architecture, it's by design.

Speaker 1: slower the CPU is still blocking um nothing becomes magically faster when you use a sync. io and actually this is the the soccer that I had that m in my mind when I was reading uh is that actually Django is not asynchronous. It has an asynchronous interface, but it's not actually asynchronous. It has a a thread pool uh underneath and anyway So therefore we to get the results we wanted we we had to change a few things And the first one is that we needed to build a native Async Io database driver, which was a bit more difficult because

Speaker 1: Python does not have built-in a sync. io support for file system operations, which is um what you do when you have SQLite. And the other problem is that it was really really difficult and not worth it to make SQLite to work asynchronously because The way the virtual file system of SQLite works is completely synchronous. There's an abandoned branch about this, but this is how Turso, the SQL write, re the SQL Rust rewrite came into play. So we had to do some patches there to get it work nice with IOU ring. In case you do not know IOURINK, it's an incredible Linux kernel feature that allows for truly asynchronous file system

Speaker 1: I. O. operations. And we had to fix a few bugs. We have a vendored version right now. The second is to make it truly asynchronous Django RM So we had to so there's this thing called sync to async that is used in Jaguar Django and you know puts things in thread pools and stuff like that. We got this out the out of the way We implemented async managers, query sets, connections, cursors. There's no joints and select related for now, but this was okay. This was something we could do. And something else is that we had to inspect the the stack and ensure that we have one thread everywhere. So we had to do some changes in the OthStack, the sessions, and the middleware

Speaker 1: to ensure that no other threads are being um spawned and um and such. And why would we do this? Because I'm trying I'm trying to think how I I would say this in English, but it's b because we said so. I mean I mean that there's no there's no obvious reason. We said okay let 's reduce everything. Let let's pull everything out No conduct switches, uh no threads, nothing. Let's see how far this is going. So started pretty much as an as an exploration. And now we'll talk about the architecture, which is really really interesting. So we have the OpenTelemetry client. This is uh let's say maybe an SDK, there's an OpenTelemetry SDK with automatic Django instrumentation. Let's say that sends information here

Speaker 1: or is a collector that uh gets your system logs. Uh by the way you can configure your AI agents To send open telemetry logs to a URL and you can see you know how much um you spend in tool calling and stuff like that. Um anything that can send open telemetry requests reaches to Ubicorn, which is the web server we're you we are using. And the interesting part is that we have three ASGI apps In case you don't know about it, ASGI is um I'm not sure it's an official Python framework like WISGI for web applications, but it's what Django uses in channels and for its asynchronous capabilities to support um asynchronous HTTP operations.

Speaker 1: So we have one root ASGI app Which inspects whether we are ingesting right now we are accepting data for open telemetry, someone is sending us logs, metrics and stuff, or it's a regular let's say uh Django request which could be a dashboard view. Now in case we are ingesting data We we don't wanna mess with Django and we will see in a bit. And I mean with the Django stack, the ORM and all the nice things they're still there And now we have a queue then that we say okay you you have to write those things in disk. Then we have one coroutine that writes these things in disk, and then we write them to the SQLite database through Turso. And this is I mean this is

Speaker 1: I like this code, I have to say this. I like mad statements by the way. I I was so excited with Python 3. 10 when uh they They appeared. So this is the root ASGI app. I have removed a few things to make it a bit more readable here. So when you get the scope, this is a standard ASGI object that always exists. So you get the type and you say if it's a lifespan event, you know return this if it's an HTTP event and it's an ingestion event which means that the path is slash v1 slash logs metrics or traces then return the re await the result of the ingest app Or if it if it's not

Speaker 1: but it's still HTTP, you know, you can send it to Django. Now, why do we have ingestion outside the Django request stack? Because We don't want uh the middlewares, the sessions, nothing. It's um it's write only endpoints. You just write logs and all this nice stuff. So Um you don't want to check about CSRF sessions all the time and these things cost CPU and as we said before CPU is still blocking and we don't want it here. So we got it out of the way. And the way this works pretty much is we we decode the body, the protobuffs, and then We we create a future. This is how this is called in Async Io

Speaker 1: means this is going to complete at a later point. And we enqueue, we put this into the queue, write this to the disk. And then we wait. If something goes wrong, we s we respond with 503, which means the client should retry. And This is what we're doing in NQ we check if the queue is full or closed, then we don't do anything, which means we return false, and this is what would give the 503 that we said before, which would mean retry to the client later. But if not We're putting this in the waiting list to be to be written and we wake the writer, the coroutine, to write this. Now, let's talk about the writer. By the way, what I said before is that we will focus only on write performance in this talk

Speaker 1: because I really do not have much time available. So it's one core routine Same event loop. Um this is it. I said to you, I will move over forward to save some time. Uh important we never wait for more work to fill in the queue so every time. What's inside the Q and fits inside our bounds, we take it, we thrite it, we write it, and we're done. Now let's talk about storage a bit. We wrote a custom database backend Django Turso. We're using I. O. U. Ring, this incredible Linux kernel feature for a synchronous file I. O. We did something very interesting that outside of this presentation, but I'm gonna send it to DjangoCon Europe and in case it passes we're gonna see it there.

Speaker 1: We're storing parquet blobs in the for metrics to query them uh quickly in the SQLite database. Now this is how it looks nothing special about it it's just it's just an engine. um a few nice queries. For example, I wanna I wanna see a trace which is the uh the outside most babuska or ev envelope whatever and I want to see where it starts and then and ends based on the on the spans the nested envelopes it has I can do A aggregate, asynchronous aggregate requests, real nice ORM. Um again I can do some annotations

Speaker 1: um in all my metric points here and Then I can do asynchronous uh you know iterations on them later. Um this is something else that I really liked. Because we're using SQLite we can do globe searches, you know, with the stars real nice. So I say hey bring me a service that the name key underscore underscore globe we're doing a globe lookup looks like this and then an asynchronous iterator real nice Um and then things will go wrong and we'll see how how we handle this. And it's First of all, when something's go wrong, 503 as quickly as possible. Let's move forward to the next. We're not killing commits.

Speaker 1: We're just flagging that there's low right now and if we kill this mid load, if we have returned 200 to anything, this means it's stored on disk. Now we're gonna see what got us to one thousand requests because this uh was not easy uh from the get-go. I remember I was with my wife at some point and told her, hey, I have this presentation and I have promised 1000 requests and I'm stuck at 350. So we're like But uh yeah after after a while there was one thing the most important thing that um got it working So um first of all um we had to reduce the CPU work as much as possible. This is very important because CPU is

Speaker 1: is blocking the whole Async IO thing. I mean the the whole uh you know the whole process First of all this next we store what you sent us now but we will make these queryable tables and records later it's okay where later means a few milliseconds by the way And my favorite one, the one that really made the cut and made this presentation possible, was commit grouping and we're gonna see about it. So uh how did we reduce um CPU per request? We used some caching um in batches, so if we resolve something one uh once we can reuse it in the other requests are coming in within a bound. Um we're casting uh the the queries

Speaker 1: so we just fill in the values. every time and we don't have to rebuild them. And something really really important is we are not using init for our model objects. Because um we could really save some time so we created a new class method We call it FastInit and does some nice exotic stuff and saves us some CPU when we are storing our data. Store now, think later what I said, give me the open telemetry data, store them to disk, and I will make these queryable in you know smaller slices. Why smaller? Because when I finish one slice I can resume accepting more ingestion data. And this is the best thing is commit grouping, which means we could accept multiple HTTP requests with

Speaker 1: with uh you know with records and we could write all of these together at the disk. So this is I mean for 80 concurrent client it over tripled our um throughput And this is how it works. Um let's say we have uh four requests, they come at the same time, and they are they are being put in a queue, this which is what we said at the beginning, and then the writer says, Hey Give me what you have in the queue and I'm gonna write them all together and it batses all those nice records in the disc. This is how it looks I have simplified things a bit. We get a batch and we create a new list raw requests. We fast init them to save CPU time.

Speaker 1: with the data and then we do an asynchronous a bulk create and we store everything in the database The win with Async. io is that we have zero um we we have zero contact switches in Python. By the way there are other sorts of contact switches that we will talk about later for everyone interested. Um one thread for ingestion and dashboard, this is the real win, nothing is blocking it. Um but one catch is that you have to use bounds. This means that you have to limit everything in time, in size, in everything, otherwise Everything can become blocking and now I wanna do some

Speaker 1: demo to you. Let's hope it works So this is it pretty much. This is our our development cra cluster actually. It's what it's being reported right now, which means the last 15 minutes I can go to the services here and I can go and look for Crix which is our proxy and I will open up this and I can see my logs here. Let's go. I can see the logs live, which is another interesting feature that we build with server-send events and Django. Let's see if I can see some sort of metrics. I have

Speaker 1: do I have two more minutes? Okay, yeah. Um so I can see graphs. These are SVG server rendered, so there's no uh JavaScript here for no reason. Anyway, I don't want to do the whole you know um f feature you know tour right now. Um I'd like to flex a couple of things that All of this happen with sixteen to one hundred and sixty-six megabytes of RAM on your server. Um also there's nothing else you have to to look at. There's no Redis or Postgres or anything. I mean, this is it, the real thing. And from almost zero CPU to not even one percent. Um takeaways

Speaker 1: Is nothing is is magic. Really, if you start and get into the code and now I think with you know the AI agents and all that stuff, it's easier than ever to explore existing code bases and see what are the actual bottlenecks and what you can work around them. Developer experience is really important because this is how you make something uh you keep something really maintainable. Django is extremely capable. I mean it can get you this is a new phrase I have in my mind. It can get you from garage to IP or really it's um It's amazing. Thank you to the people that built it. Now, a few things that I hope to have open source by today, but I was doing this presentation until 3 a. m. yesterday, so uh I'm gonna do them in the coffee break

Speaker 1: It's um we're gonna send some upstream fixes to Turso, the database. We're gonna publish a super package, it's DJ11 that we will include things that we could hope can Go into upstream Django which is the asynchronous ORM, the globe lookups and server cent events There's a Django Turso database driver. We built a really interesting tool about load testing, which is Gita where it can compare work trees and see how one performs with the other. And the core, uh what you already saw about this um observability platform, Facos, it's gonna be open source as well. Uh next we have to take care of retention. What I didn't say is the

Speaker 1: what I was roaming into was about 15 gigabyte of data, millions of log records, um metrics, everything. Um we're gonna create synthetic metrics like P99 you know staff latencies, something like that. Make it Native ORM queries within the parquet blobs alerts, and we want to pass the 10,000 requests per second threshold This is it in case you wanna check it out. It's facos telia dev f a k o s dot dev. You can scan the QR code if you want. The early preview is gonna be twelfth October October and that was it. Thank you very much for your time. I'd love to hear your questions if you have any. I hope I wasn't speaking too fast.

Speaker 2: Woo!

Speaker 1: Thank you.

Speaker 2: Thank you. Well, we have time for questions. Um and uh Vidya has the microphone and there's a hand there, there's a hand there.

Speaker 3: Okay, thank you for the talk. So one question, is torso the same single writer approach as SQL light or other multiple writers and if it's so then uh have you tried doing some tricks like uh I don't know keeping several write requests

Speaker 1: Yeah. So um great question great question. No multiple writers for now. Um or we're gonna try it in the future because when we vendored um tourso it wasn't I mean there was a few things that were not that sturdy with the multiple writers. Now they got a new version a couple of days ago. We're gonna give it a go. And yes, we are batching so as we accept requests we queue them and then we group commit them. Now they also build group commits for multiple rights a couple of days ago. I'm so excited to see it, but it's one writer for now. So we tried writing to two different databases, but it um really did not help with uh with performance

Speaker 4: You said that the CPU was still blocking and then showed us a chart where the CPU utilization was under one percent. Can you explain that?

Speaker 1: Uh yeah. Uh what when I'm saying blocking it wasn't that it was blocking the whole process. It was that this was the bottleneck. I mean when I have one process I mean I'm gonna use whatever the scheduler is gonna give me. And what I was saying is that I have the timeline of a request, the most time was spent on CPU

Speaker 5: Hi. Thank you for the talk. Regarding the use case. It's a very interesting use case. But could you uh uh see this being used in in totally different contexts like games or chats or any other kind of high speed uh writing.

Speaker 1: I'll say I'll say four of them because I'm trying the three of them right now. So One of them where it started is our development cluster where we have things. We're gonna put it in production as well, by the way, because it looks It looks good so far, but I'm also uh using it to track my AI coding agent um data since they can export open telemetry and I can show them to you later it's in my local host. And the other one that I'm you know baking right now is I have a NAS at home, a Synology device, it's like a server. And I'm gonna put it there and I'm gonna track you know my smart bulbs and stuff like that. I'm gonna intercept the TCP protocol and play. Yeah. So this is the other thing, IoT, let's say

Speaker 1: Because it it's it's it's really easy to set up, it's just a container. It's just a process. You just need Linux and IOU ring, and it's a single process.

Speaker 2: Any more questions? Anything from the internet? Alright. Let's give us a big round of applause.

Speaker 1: Thank you.

Speaker 2: Um bye. You're next.