Advanced, free, open-source application performance monitoring for your Python apps

Speaker 1: But uh hello, hello, uh welcome to my talk on um application performance monitoring. So uh today we'll we'll do a little bit of a dive into uh kind of the history and context of uh APM and why we want it, why it matters to us at least and why it maybe matters to other people. And uh we're going to look at some alternatives for actually using APM and integrating it into your applications. So my name is Emmanuel. Emanuel Tolev. I'm a community engineer for Elastic. uh the the makers of Elasticsearch, Kibana

Speaker 1: and so on, monitoring tooling that you have likely heard of and also uh the open source search engine. Um the so I do work for a vendor, so uh Elastic does make a whole APM solution as well as various monitoring. solutions but I have tried to touch on uh other projects and on other things and kind of the wider context of this but I will later show you some stuff in Kibana and then APM. However, Restsured all of it is completely free to use, including in commercial context, so I'm not here to uh to sell you anything So uh let's get started um with kind of what IPM is.

Speaker 1: Uh how many of you have an idea of what I'm talking about when I say application performance monitoring Okay, that's reasonable, there's still some uh who who don't, but that's the point of of these talks, right? So um APM very quickly then uh APM at its core is search. It's a solution to a cognitive problem. Uh when you Want to know what your system is really doing specifically at the edge, what your application is really doing. So not simply monitoring the percentage of CPU or the percentage of RAM or the you know the disk IO that's being used, but uh really getting in there and looking perhaps at individual lines of code, uh bottlenecks in your application, what is slow, what is breaking, errors, etc.

Speaker 1: So that's application performance monitoring and uh sort of a short story about why it matters to me and kind of how I got interested in this whole area as uh uh of course like probably many of you I have also been on call uh for nearly five years and that was the uh kind of it's pretty hardcore it was the beginning of my career actually in in computing. So straight out of uni and I became a freelancer. And I joined this small uh hipster collective of freelancers, this agency And uh soon enough I I was in charge somehow of determining the operational strategy for clients. for a couple of smaller clients and later for some bigger ones.

Speaker 1: And that's where kind of My initial enthusiasm started to grow out of bounds and to hit a snag, right? Because um what what happened was that we would build all of these features And then in the middle of the night I would get a page and I would have to get up and fix uh fix the website. that had fallen down. Unfortunately, I was in a 24-7 contract and I was the only person on call for a few years on that project. Um and then the but at some point uh you know you you do this, you know, I'm I was like what at twenty-two, twenty-three, so you you do this and you don't really mind it, you know. It's it's uh it's for the project, for the client.

Speaker 1: But then um a few years ago, what happened was I w I got woken up at three and then again at six and then again at seven, same day. And then at eleven and then I spent the next uh week trying to desperately make this website work. And what was happening was we were getting access to I think legitimately from China, but it was just not a use case that we had predicted. And um it got really out of hand. And that's where uh I got kind of really interested in how do I prevent this from happening to me or other people ever again and uh really understand it

Speaker 1: Is that okay? Can you hear me? Can you hear me? Alright, okay. I'm just gonna just gonna hold this then. Um So yeah, so that's where uh I I kinda got really interested into how to solve this problem. And uh so as far as I could tell, I mean at least in my career the evolution was I went from looking at log files and htop for like the first couple of years uh when I was sort of helping out small companies uh to then trying to collect Fuck it. Uh to then trying to collect server information. Uh uh uh

Speaker 1: so federated data collection. There we go. And uh then finally APM. So we moved past the servers, we moved off the servers, off the virtual machines, and into the code itself, uh, which was kind of mind-blowing for me So this is a little slide from applicationperformance. com and I came in towards the end of that, so we're about thereabouts here when you start seeing the emergence of uh kind of more complex software as a service providers and we'll talk about what can they offer in a second. Um but we're sort of now obviously past this stage, so we're looking at what's the what's the next step. Um at the time

Speaker 1: when uh that uh when at the end of that slide uh towards the bottom right, uh competition was Very difficult in the APM space. It was very labor intensive to build new solutions, including open source ones. Um instead competitors went for a narrow focus. And so this is actually a Copenhagen company. So this is Upbeat. How many of you have heard of Upbeat? Ah wow, geez, okay. Right, I'm not gonna spend a lot of time here then. So it specialized in JS and Python, including Django, and uh later on it uh in 2018 Uh it 2017-2018 it got acquired by Elastic, the company I now work for.

Speaker 1: And uh Elastic APM kind of fits within the wider infrastructure uh of the Elastic Stack. So Kibana, the ping thing, is used to visualize uh the data that you put into Elasticsearch and everything else is used to put data into Elasticsearch. Uh beats and so beats collect metrics So numbers with labels, uh logstash puts logs in there, so text, and APM puts application level metrics like the particular lines of code and transactions that might be slow. Um so let's see if if the Wi-Fi works. Uh so the the landscape has evolved significantly since then. Yes. Yes, excellent. And this is an excellent website.

Speaker 1: It's called openapm. io and I love it. Um it really showcases where like how crazy things have gotten. Uh so there's the elastic stuff, so if you click that it doesn't really interoperate with much else, so it's just the elastic stack. And you can use Grafana also to visualize it. And uh you can use certain things uh to to do alerting on it, but um it it's pretty much designed to work as a as a as all the pieces together. And people sometimes add elastic beats to it, which allows you to get other metrics like you know your node metrics. So how is uh how are your actual servers performing underneath your containers? Your application code.

Speaker 1: However, since we'll talk about that later, I want to talk a little bit about a few other solutions that you might use. So another possible one is Prometheus. And that's an open source metrics store and time series database and the whole toolkit around that. It's very cool. It's uh based on Google Borgmon, or it it was based on Google Borgmon. Now it's its own a very mature project. They released version two. So what you do with Prometheus is you have like uh with all of these things, you have an exporter, so you have an agent, something that hooks deeply within your application. And uh extracts all the useful information out there, like what transactions are

Speaker 1: being um processed. Then you have a collector, so where the agents that sent all of that information, and that's the Prometheus server. And then you have storage. In this case, uniquely it is also the Prometheus server because as I mentioned it's a very capable time series database. And to visualize it, so yeah, the Prometheus itself can do visualizations. They're sort of individual and kind of basic. So what people really tend to use with Prometheus is Grafana. And uh Grafana is a very good tool. Uh it's very flexible, it allows you to embed visualizations, and it um it actually plays well with uh uh Elasticsearch as well. And so this is typically what uh happens here.

Speaker 1: So here you have again we're talking about monitoring your servers. So the operating system level metrics, orchestrator metrics, container metrics, as well as your application metrics. Right. So uh so Prometheus, uh so there's so what what sort of because you know the I guess the the point of this is to I mean n not really compare because I I can't really do that since I work for a company that makes this, but to talk about the the different possibilities here of what you can do and maybe you walk away with an idea of what what you can use to enhance your current stack or when starting a new project. So Prometheus is a great alternative.

Speaker 1: You can Uh so the the main thing with it that that that I like that really makes it stand out I guess is uh its pool data collection model. So uh with Prometheus you don't have you don't run like this risk of overloading your monitoring infrastructure because you configure at the monitoring infrastructure how often it should pull and you configure you monitor that for when it's overloaded. So there is very little chance that it'll fall over. It's pretty stable It's got a very powerful query language and it it serves does all of this stuff in a single tool. So it collects metrics and uh does the viz and supports the querying. It's all in Prometheus server, uh which is pretty cool.

Speaker 1: Uh let's see what 's the time? Right. Uh I have a video about this actually, but I'm not gonna risk it because there's also time and And so on. But I do have uh um some extracts from there to show you like roughly what that looks like. And here is where we get into a more interesting topic. So um this uh you should totally if if you're interested in using it, totally watch that video. It's an incredibly clean, like clear talk Um so this is how you generally use Prometheus. You define uh that they have counters, histograms, gauges, like various types of objects. So in Python you would define what you want to monitor and this could be anything.

Speaker 1: So this could be application level metrics, which we'll see in a second, um, or it could be business metrics. So it could be how many sales are we making? How you know how as well as how slow the website is, it's like how happy are our users, uh how many units of stock have we shifted, uh how slow is the website specifically for our uh administrators and so on. So it's very ex extremely flexible and powerful in what you can define but um the the Other side of that coin is that you have to define that. So here and what what's happening here is we're using uh the definitions from the previous slide

Speaker 1: so we're using the counters, the gauges, etc. to actually monitor something in our application with Prometheus or well to send it to Prometheus. So um Over up there we can see the entire route essentially being monitored. That's like at request time. time. And here we see a particular piece of the code being monitored. It's the context, so with analyze time And so that stuff is going to record obviously how long that piece takes and send it to Prometheus. So that's pretty cool because at the same time, you know, we could enhance that information in any way we want. We can tie like essentially how long a particular piece of code takes to how much money we're making. So that's excellent, right? The

Speaker 1: the thing about it is that uh it's metrics that you have to define explicitly. And the other thing about it is that it's only metrics. So metrics in the sense of you have some number and you put some label to it It's only that. So you don't get logs with it. You can't do log monitoring in Prometheus and you can't correlate with log monitoring. Uh with uh uh this you you will need to do quite a lot of work to achieve the same a level of sophistication and ease of setup as you would get from a commercial software as a service provider. So

Speaker 1: that's a bit of an issue, and people have used this very uh successfully, and there are even community projects that help you get over that initial step. So, for example, there is a Prometheus Django um kind of module that you can use. That does a bunch of this work for you, it provides you with middleware and it instruments your app. However It like the support and um well how often these are maintained and updated and how well they might fit your specific application. So how much customization you'll have to do uh varies quite a lot because you know that that that in particular is just a community project. So how whoever's got time, you know, it's got like two, three hundred commits. Um This is how you actually install it to give you a quick idea.

Speaker 1: So you just got you put it Django Prometheus if you're using that and uh installed in Yeah, in installed apps, and you pop the middleware in there and this is gonna be very similar for what I'll show you later And of course and you also do its URLs because it's a pool, what happens is it exposes an endpoint called slash metrics. So that that's what this does And from then on the Prometheus server that you configure separately can pull from here. So it's really easy, really simple to install. It's basically settings and URLs, a couple of lines, and you're done. And that's what uh the built-in dashboard looks like. And this is what Grafana looks like.

Speaker 1: uh with it. So it's pretty reasonable. You know, you got your responses, uh response codes, and uh kind of breakdown of requests and there are more there is more like there's uh the 95th percentile, 99th percentile and so on. that you can do with this. So that's pretty good. Again, the only thing with it is that it doesn't do logging and uh Yeah, you you may need to do more customization with it, I think at least. So uh here we get on to Elastic APM. So so far what we had from that Slide that uh from this slide is you have the software as a service providers uh down at the bottom right where you literally just install something, you you put in an agent.

Speaker 1: into your app and then it automatically sends all the information and you know everything is cool. You go and log into the dashboards and you understand everything about your application. Or you know that's the idea at least. Uh then we have Prometheus, which is very powerful, very flexible, but you might need to do some more setup uh to fit it to your specific app So that's where ElasticAPM for me came in. And actually that's one of the reasons I joined the company was because I was excited about this project. Because what happened is they bought Upbeat and then with uh rather you know a lot of flair uh they uh open sourced almost all of the code of this commercial SaaS provider that had been in development for five years

Speaker 1: Uh what they didn't open source but is still free to use is the user interface, which you will see shortly. Uh however they did nonetheless provide an in for the first time I guess or at least as far as far as I'm aware for the first time provide an entirely free solution that's like commercial grade has been developed for several years And you can just use it. So the advantages of this is that you have more consistent support and you also get the logs in the same place. And I guess it's uh pace of development when it comes to specific integrations is probably greater. Like you know, it has uh. NET and Ruby and lots of stuff that might otherwise um

Speaker 1: fall behind a little bit or not not just not continue development at the same pace. So uh Let's have a have a look at at some uh actual integrating. So this is the app that we're instrumenting. It's a relatively simple app. It has products, it has some orders, and it has some customers. We're gonna look at the products page. Um here.

Speaker 1: Ah, lovely. That's what happens when you uh All right, number one. Uh let's go with this. So that's what APM looks like. And actually, um, so this is because we're talking about APM, I'll start from here, but then I'll show you what the um the rest of the whole thing

Speaker 1: looks like so you kind of get an impression of what the entire monitoring solution uh actually is all right so This is not what I originally intended to show you, but unfortunately I have uh uh put aside the link somewhere to the actual demo and I have closed the tab with it Uh so we're going to look at this one. Uh so what this is, is you're probably familiar with this, this is a similar screen to what SaaS uh vendors will offer you. You have your transaction, you have the full details about your transaction, like the particular request, the response, the uh uh no the operating system information

Speaker 1: as well as being able to add custom tags. So here you can do stuff like customer tier. This is a you know very high value customer. Uh or you and user you can have things like the specific email address or user ID for the user. And then the interesting part, of course, is the actual tracing. So here in this case, this is an Elasticsearch request. So we can see the body of the search as well as the specific stack trace. So we can see all the way through the libraries and I do have a Django demo, but this is a Flask one. You can see the entire stack down through the libraries, down through Green Unicorn.

Speaker 1: And you can see exactly which part is taking how much. It also supports salary, so it supports asynchronous workers and it they will just turn up as soon as you instrument it And it supports SQL as well. So it will break out your SQL queries in the same way that we have this Elasticsearch bit here. It will break out the SQL too. In fact, let's have Quick look and let's see. This is not uh not Python, but Let's see if I can show you that. Ah yeah, okay. So that that one is a bit more interesting. So you can see uh what's going on here and I assume a lot of you would have

Speaker 1: seen seen similar stuff in in your applications. Um so the the thing that I like about this besides that that it's uh that it's free is that it combines uh performance monitoring with all the rest of the stack. So uh we start from something like this. So this is Just your regular unstructured logging. So that's basically uh file beat, which is a log shipper application, so that's sitting on every server or you know everywhere where it needs to sit and it's just tailing specific files. It has a lot of defaults, uh so it's pretty easy to configure.

Speaker 1: Like you don't have to do a lot of configuration to tell it. get these files and these files and these files. It tends to find Apache and Nginx and um all of your standard um applications. Uh so this is searchable as well. And uh yeah, that's you know, Elasticsearch, it's pretty good at that. So Then from unstructured logging uh you we can go to a more structured logging or we could if it was the demo if I prepared originally at least

Speaker 1: So as you can see it's uh pretty easy to uh uh to set up a visualization in Kibana. Like the discover. Oh, that's a link. Bless you.

Speaker 1: I guess this is what happens when you lose your uh Plan demo. Okay. The thing is this definite so this is the official like uh like the kind of core last search demo that's uh Kibana demo that's used by um you know the entire company of over a thousand people so I think what's happened is that something has actually happened to it.

Speaker 1: It's uh Not just me, and this is always full of data and it uh processes about well it processes thousand thousands of requests per second. So uh There is definitely a problem here, but uh very possibly one that I will not be able to fix on my own on stage. So uh Let's have a let's trash that. Let's just try that again.

Speaker 1: So you generally simply select the file beat index and then you tell it what times what field it should use. in order to kind of separate the records in the index and to be able to get structured logs out of the unstructured information. And then ideally when you click discover , Uh I uh wait a second, it could be simply the no Aha, right, okay. Yeah, so this should definitely be full of data. So something has gone wrong with our official demo. So there we go. Why not? On stage, you know, that's that's totally normal. Right.

Speaker 1: So that's the structured logging view. Woo. Right, cheers. Thanks for the support. Okay, so the um the probably uh yeah something has happened to our demo. I just got a message. Excellent. Very very very timely. Right. So never mind. Um so uh we the cool thing about this is that It allows you to do a whole bunch of interesting things with this information. So for example, we can see uh what we are running.

Speaker 1: So what the container name is And uh you you'll see in a second what this little table button is doing. Okay. So this is pretty cool. So I would have really liked to have this when I was just looking at logs and you know like trolling through log files. What this does is it takes one field out of the entire record and then it shows you the same field in a summary view. So as the records are collapsed, you can kind of go through and see So, you know, this is the uh the the first thing is the name of the container. So this is the actual web application and you know this is an Nginx instance and these here are different pods

Speaker 1: So in this this in this case this is a Kubernetes setup, not not what I was intending. Um but so This is super cool because you know you can do things like log levels so you can easily uh scroll through and find all the warnings or errors. Uh you can filter this The additionally you can sort by pulling out specific fields. So purely as like a log navigation tool, it's uh very powerful. And uh yeah, it's it's uh pretty cool. So you get unstructured logging in in logs, you get structured logging in uh here in discover and you get application metrics in APM.

Speaker 1: You can do a lot of other things with this as well. So in Visualize In visualize, you can define visualizations. Uh and there should be about 300 here. So you can kind of build your own dashboards. So here is the bit where you can combine business metrics with your application performance metrics. So for example, you know, here is where you could plot. like sales on on the one axis and 95th percentile performance you know under one second or however long it takes on the other axis. So which is pretty important, right? Because uh obviously that's how uh we relate what we do to the rest of the business that we are operating in.

Speaker 1: Uh so um so so far You've seen uh Elastic APM, you've seen the rest of Kibana, so you've seen logging uh kind of combined with that And you've also seen Prometheus and how to how to integrate that. So for Elastic APM, the way you integrate it is pretty similar. to Prometheus. Interest of time I'm not going to try that demo just 'cause yeah too much. Um but

Speaker 1: What uh the it's it's basically extremely similar to Prometheus. What you need to do is you run Elasticsearch, you run Kibana , you run the APM server. uh which are pretty much all Prometheus in the case of Prometheus and Grafana. So you run Elasticsearch, you run Kibana and you run the APM server, and then you instrument your apps with the APM client or the APM agent, which is simply pip install Elastic APM. And extremely similar to Prometheus , you simply show them you simply uh shove it into installed apps and you install some middleware And that's about it.

Speaker 1: The extra thing that this does is um that I guess you you can't absolutely do with Prometheus, you just need to pull in more components and configure them more. Uh the extra thing that this does is that it also allows you to in the same manner instrument uh RUM, so real user monitoring. So sort of uh this is what I showed you so far, is all on the server side But there is also monitoring you can do on the client side. So when you yeah, so you kind of know when the JavaScript has loaded and at what points it has loaded, how long it took, how long certain um parts of the page took to really load. And how long did the did it take for the user to get anywhere useful? So this is included as it is in most

Speaker 1: software as a service vendors. So additionally the the kind of the other feature that's worth mentioning uh that I would have f like to show you, but there we go. Um is distributed tracing. So if you have microservice architecture uh or you simply have an architecture that has enough components that they start calling each other in some manner Uh APM uh supports this. Actually, I wonder whether I still can show you a picture because it's you know it's worth a thousand words. This is like really basic mode here.

Speaker 1: Yes, the different colors are different services. So uh the the blue is a Ruby service calling a Python service in green Which calls a Java service in purple, which calls a Go service in red. And this is supported automatically. So um There's like there's a complicated bit of inference that the the APM server does when you instrument all four apps. with the Elastic APM agent and you just instruct them to send information to one server, it will infer that they're connected together. Uh which that's a pretty cool piece, I think. And Yeah, distributed tracing I should say

Speaker 1: is uh is pretty standard uh nowadays. You would expect it from software as a service vendors. Uh but hey you know now you can get it for free. Now I do keep saying for free, uh and it is, but of course uh it's only for free in the sense that now you have to put in the effort to maintain it and it's like you know, it's the old uh the old problem of uh who monitors the monitoring and who watches the watchers. So uh as as most like I guess most product companies if you're you know like a startup or uh small to medium sized project, uh you don't really want to deal necessarily with the configuration and hosting of this stuff. So what you do is you go and get it somewhere.

Speaker 1: And there are multiple providers, Elastic sells this, but for Prometheus there are multiple cloud providers as well. So you can totally do this. Uh and I expect not sure but I expect that the pricing of this will remain low because the actual underlying code is free to use and uh the majority of it is open source. So that puts some pressure on on my company, you know, not to uh not rush up the prices too much. Um whereas obviously and proprietary environment that pressure doesn't exist. You just get charged for the value right that you get out of it. So yeah, so

Speaker 1: I think it's a pretty cool project. Um it's a pretty uh like advanced one and I do really like that they went out there and they published something uh that was of kind of this sort of commercial great where it's just you install it and stuff just happens automatically And uh but the I I do still feel that um there is a fair bit missing from this space and like you know Uh we've been talking about um logging, uh monitoring and observability and this sort of logging and metrics in particular now for several years and

Speaker 1: In terms of actually innovative thought in this space, I haven't seen it progress uh that much, although there are some uh exciting new projects. uh that may progress this so we we'll see how that turns out but more importantly um what I would like to see a bit more of is more collaboration between different open source projects. So you know Elastic imports Prometheus data and you can use Grafana with Elasticsearch. Uh but ultimately I've seen the kinds of stacks people have in the wild and they're they're you know they're pulled together from all sorts of components. You know, you have Fluent D and Kafka for processing your messages and like

Speaker 1: It j just all sorts of stuff. And it's um you know open source is supposed to solve this, right? It's supposed to not be this pain in the ass where uh if you want some component like Elastic APM you now suddenly need to migrate to the whole Elastic stack but you're currently using Prometheus So I would like to see more collaboration between these things and that can really only happen uh if there is pressure for it and if people actually collaborate on metrics. If you care about metrics, if you care about monitoring. And you know, you maybe don't that much. Maybe you just want something to use that's easy and relatively low cost, who knows. But if you do, um

Speaker 1: please write to me and uh Also please tell me what your setups are because like the more data there is on what the setups are and I do plan to republish this information when I get enough of it. The more data there is on what the setups are, then the more obvious it's going to be to potential uh you know open source maintainers. that there is demand for this thing. And I think that there is demand for this because even in like in a small size project you may start with one stack, but soon enough uh even a medium-sized project is a couple of years old, you move to a jumble of different technologies and you know uh um initiatives like open metrics which came out of Prometheus uh kind of you know give you this you know

Speaker 1: base level of cooperation and there's also the Elastic Commons schema which is also an open thing. But it's this is not quite there yet. I think we could do a lot more on open source collaboration. So thank you very much for for listening to me and I'll just take some questions if we have time.

Speaker 2: Awesome.

Speaker 3: We do have a little time for questions. Um again you can we can ask questions on Django Con um QA on Twitter or you can do DjangoCon on the IRC. It is always awesome. The last two talks are open source in the wild. It's so cool. So ask some cool questions about it.

Speaker 1: Go for it.

Speaker 2: Okay. Hey, uh thank you for the nice talk and for your perseverance against adversity. Um I was wondering if uh the APM server had an API that could be queried from the outside, for example Just random example, a Raspberry Pi with one of those green, uh, yellow and red lights. And if my uh server load, for example, goes over five the light uh on my desk turns orange because it's getting information from that APM server.

Speaker 1: Uh yeah, totally. Yeah. So that that's exactly how it works. And I mean what you saw there, Kibana is reading often APM The APM server is entirely open source and an entirely separate component. So yeah, you can totally do that.

Speaker 2: Thank you

Speaker 4: The the internet may defy your demos, but the internet has questions as well. Can you compare the resource requirements for APM and the ELK stack behind it to other options like Prometheus and Sentry? ELK has has somewhat of a reputation of being resource hungry. Does that uh how much re-ho how big a setup do you need to have to be able to log a a small or medium-sized site?

Speaker 1: Yeah, sure, yeah. So that's true. Um the uh for a APM is a kind of very different project from the core Elasticsearch. I probably not gonna really go into saying well this is faster than the other because I do work for one of the companies. And I you know, I mean like I I'm a web developer, but like I don't want to uh to necessarily say it's faster because it also depends on the situation. Right and like what load you put it in and what per the particulars are. However, uh the APM team takes the benchmarking pretty seriously. I mean that's one of the good things about it being an open source component that has the backing of a commercial company, right? Because they can spend weeks uh benchmarking in very specific situations, right?

Speaker 1: And they do. So as far as I know, they're I know they're very proud of like how small the percentage overhead is. So as far as I'm aware, it's uh It's in a good class of percentage overhead. So it should be pretty fast and it's uh it's very different from the Elasticsearch. uh piece of software which yeah uh you can like if you're a small project uh what I would probably do personally, right? Um this Yeah, it involves giving money to Elastic, so take a grain of salt, obviously. But um I I would probably go and just go on uh Elastic Cloud and the like cheapest tier that we offer and that offers APM

Speaker 1: and it's all fully set up. So you just send it stuff. You just instrument and send. So that's what I do personally.