Introduction to OpenTelemetry with Django with Jessica Garson

Carbon

Speaker 1: Hi, my name is Jessica, and I'm here to give you an introduction to Open Telemetry with Jane Gow. And I can be found on most places on the internet at Jessica Garson. I'm also on Mastodon at Jessica Garson. at macaw. social and I am a senior developer advocate at Elastic. All of the code and the slides for today. um can be found on my GitHub, github. com slash Jessica Garson introduction to open telemetry with Django. You could find some other resources, all the code I'll be showing, everything like that can be available, is available on my GitHub. So I had attempted

Speaker 1: to find a short primer on getting started with OpenTelemetry, but I couldn't find much more than a very simple demo application. Um, I found the Python one on the open telemetry docs and I found one for Django, but couldn't find anything that was like a little bit more robust, and I was trying to kind of figure out what was going on, but couldn't quite find what I was looking for. So I asked my friend Chat GPT, maybe your friend who, to explain open telemetry to me like I was five. And I think it fixated a little too much on the five-year-old part, but it told me that OpenTelemetry will watch me color and draw. And if a crayon breaks, it will monitor that for me. Um, and

Speaker 1: it will tell me exactly why my crayon broke and why. And so I could fix it next time. So my picture turns out great. But this was a little bit confusing. And in my experience, this isn't quite what open telemetry is, but it's a similar metaphor, but it was a little bit confusing and hard to get started. So I decided to make this talk because I'm hoping that you will walk away from this talk with a strong understanding of what OpenTelemetry is and how to get started integrating it into your applications. So before we dive too deep into open telemetry, I actually wanted to kind of do some level setting and talk about some observability fundamentals. Some of this was actually a little bit new for me as well. But observability in general helps you determine what's really happening inside of your system by analyzing the data that it outputs.

Speaker 1: And it is the understanding of what is really happening to your system. And it's about the why, not just the state you're in. And I find this to be really fascinating. But it also reminded me of being a cat mom. Um, so what can y'all infer when you look at this picture of a cat? What what's happening to this cat? I think that it is either yawning or having a belly rum, but this is basically the way that I think about open telemetry in general. It is similar to how my cat will tell me when she needs water by meowing at the water fountain. It is the external outputs of a system, such as error messages, system logs, performance metrics, they act as hints that offer insights.

Speaker 1: into the internal state of a system. And by analyzing these outputs, you can gauge the health of your system. And therefore you can identify any discrepancies that may indicate underlying problem. So it's really at the end of the day, what isn't happening as expected. So one key aspect of observability is the ability to quickly determine Like what's not working, what's not going right, what with what 's going wrong, and abnormalities and performance metrics, errors, things like that, they let you know what's happening. And it goes beyond detecting problems. Um it's also about understanding who is impacted by the issues that you see. And this involves understanding how failures

Speaker 1: affect end users and other dependent services within a system. And knowing the impact really helps you prioritize fixes, helps you kind of see the spikes and be like, okay, that's what was happening there. I know what's going on and it helps you kind of see the potential impact that things may have. It also provides necessary insights to help you fix issues and determine what might be going wrong or well with the system. There's another word that gets floated around a lot when it comes to observability, and that is what is an agent. And an observability agent is a software component or tool that processes, collects, and sends telemetry data from a system to an application.

Speaker 1: or an observability platform for analysis. So I think that that's something to kind of think about as well. And you might be wondering at this point, like what is OpenTelemetry? So in early 2019 The OpenTelemetry project was announced and this was a merger of two existing open source projects, Open Tracing and Open Census. And these were two competing protocols and OpenTelemetry took a little bit from each, as well as also some ideas that were floating around around tracing headers and a W3 wire. protocol or W3C trace context wire protocol. So yeah, it kind of took a little bit from

Speaker 1: some projects that were already happening and really kind of added to it. And the way that I heard Tom Eastman describe it in his talk at KiwiPyCon is it is the language that everyone is using. And I really like that because It really is an observability framework that is open source and vendor neutral. It is designed to work with any back-end system And it provides a set of standardized APIs, libraries, and tools to collect telemetry data such as metrics, logs, and traces. And it standardizes data collection, making integrating with various tools and platforms just easier to do. And so why why use it?

Speaker 1: In my experience, it actually once you get the hang of it, it's actually really easy to start using it That's one reason that for me like really stands out. But beyond that, the main reason is actually that logs alone sometimes just aren't enough. And you might be like, um, what? Um and so I this is a real log from an application that I have created. And if I had to look at this without knowing what was going on, I'd be like, wait, what is happening? What is this? And I've had the experience of going into a new copies and only looking at the logs and being like, I have no idea what's happening. what is going on? Somebody explained this to me. And I think, you know, being able to have observability features and things like that, you're able to kind of

Speaker 1: figure out what's going on, what what the logs might be catching, what what they might not be catching, and really kind of work from there to create a more robust way of seeing what's going on with your system. And the other thing that I think a lot about is that, you know, in modern development, we work with lots of complex interconnected systems. We work with a lot of microservices. And with distributed systems, finding the root cause of issues can be painful. It's hard to optimize and open telemetry enables a comprehensive view across the entire system, not just isolated parts. And so this allows you to understand the state of your system as a whole, monitoring everything from individual components to complete services.

Speaker 1: And it's also designed to be future-proof. I remember for me in 2011, I was using fog bugs, and I thought that that would be how I would always track my bugs. Turns out that that is not the case. And um traditionally vendor monitoring tools were vendor specific. So it limited their use to certain environments or technologies. However, the trend is shifting more towards open vendor agnostic tools that offer greater flexibility and compatibility across different systems. And so the idea is as you shift towards new and different tools, you'll be able to monitor them in a comprehensive and robust manner. And so for this talk, I'm going to be talking about the classic Django

Speaker 1: example Which is the to-do list application. This is near and dear to my heart. This is one of the first things I ever built with Django. And We're going to be talking about that as a frame. So I talked a little bit about longs earlier, but I figured I'd actually go in more in depth and define it. So longs are records of events in a system. They document operations, errors, and activities to aid in troubleshooting and monitoring. They're basically at the end of the day, things that you print out. A log for most of us was our first program. And it's really anything that can have a timestamp. So if I were to add a print statement to my

Speaker 1: adding a to-do list item. um function, that would be a log. That would be one thing. Metrics, they're like aggregations and numbers. they're a quantitative measurement that tracks the performance and health of the system. So, you know, the percentage of uptime, things like that, that's metrics. how often my uh to-do list application is working. Um and then traces are a little bit more complicated. They're events that have a start and finish, they tell you where something is happening, and they track the path of interactions of a request through a system. So you can actually see all the microservices that are happening under the hood when you press a button to create a new to-do

Speaker 1: list item. And so one of the things that I found a little bit tricky, especially when working with open telemetry, was the idea of spans within traces. And so one way that I've heard this put before is that a trace is a family and a span is one person in the family. It really represents a single operation within a a trace and so each trace contains a root span which typically describes the entire operation and optionally one or more subspans for its sub-operation. So the way that I like to think about this, going back to cats Is every night my kitty will uh meow and then she'll run as fast as she can to her toy and then she'll run up and wake me up in the middle of the night every night without fail.

Speaker 1: Um and the first thing she does in that operation is that she meows. That's the root. And so spans are the building blocks of traces. They include the following information. They have a name and a parent ID, but this is empty for root spans, so for my cat to meow to indicate that she's going to wake me up in the middle of the night. She would not have a parent span ID. Cool. So this took me a minute to get, and it wasn't until I actually saw this printed out And I was like, oh, okay, cool. That's what's happening here. Having the ad item view and being able to actually see that like This is what's happening and what piece of code is being hit.

Speaker 1: I was like, oh, I get it. Okay, cool. Like I see the trace ID. I see the span ID. Cool. I get it. This makes a lot of sense to me. Okay, cool. And so it took me actually seeing it printed out to be like, okay, understand what's happening. And so The reason why I was able to see this is through a process called instrumenting. And instrumenting refers to the process of adding observability features to your application. And you might be wondering, like, how did you go and get that to show in your console? And so the first thing that I would want to do is to install the required packages. So I would install Django. And then I would install Django and Byron, which so you don't see my secrets in this process. I'm using the Elastic Open Telemetry package, which lets me work with OpenTelemetry.

Speaker 1: And then also work with, since I'm working with Django, I'm also using the OpenTelemetry instrumentation for Django. And then from there, I can actually do OpenTelemetry Bootstrap and install all of the required packages I need to monitor my application. And then from there, it's really simple. It's actually just kind of going in and taking a look at my manage. py file from here I can actually see that there is Some changes that I made to the main function. I um exported some settings on the Django settings module as my to-do list project dot settings. And then from here

Speaker 1: I've actually added some like a service name so that's what will appear in my observability backend and I can put a version. I can also put the development environment um and then from there I can have like my trace provider and then I have this console exporter that lets it print to the console and that's all I really need. And so yeah um From here, let's talk a little bit about automatic instrumentation. So this is the process by which an agent modifies the bytecode of my application. uh classes to insert monitoring code. And so this means that I can automatically get monitoring code without doing much to it. So The first thing to do is to set my environment variables, and I could do that inside of a.

Speaker 1: emv file. So you can see here that I have my hotel exporter headers and then from there and that's where I pass in my authentication information and then I have my host endpoint for me that's Elastic and that's my Elastic Cloud endpoint. And then from there, I can actually update the manage. pmy file. And as you can see, it's to-do list app and I have kind of everything kind of Up here and you can actually see that I have like my trace provider. But up here you could see this one line of code, this Django instrumenter dot instrument. And this is where I'm like, hey, Django, will you automatically insert everything I need? Um, and then I have these other kind of That just lets me know that I'm going to be providing traces and then I have that I'm exporting it to Elastic.

Speaker 1: And then from there I can actually go in and have that spam processor and start to kind of get everything kind of up and running in there. And then I can just pass in my environment variables from my. emv file into my settings PY. And then I can show you what this looks like. So let's actually go and take a look at the code.

Speaker 2: Thing that we're going to want to do in our local environment is we are going to want to run our Django application just as we usually do. So we can do Python managed py run server and from here it will let us know that everything kind of worked well and that We can view our local development server up at localhost 8000. Now that we have this running in local host 8000, we can click where it says add new item to add a new to-do list item. Let's call this new item and call this Yay a new task exclamation point. And now we can actually add a second new item.

Speaker 2: Let's call it second list item Cool. Giving a talk. And we could press add. So from here we have two items that are up and running. And if I wanted to delete the second list item I can do that there. And it's just a pretty simple to-do list application. And now you can see what it looks like up and running. So now inside of Elastic, if we click underwear to services, we can actually start to see our to-do list application using automatic instrumentation um reflected back here. So if we click here we can actually see the latency, we can see

Speaker 2: all of the requests that we just made, we can see kind of everything up and running. We could see the throughput. We could see any failed transactions that may have been made. Um and really anything like that we can kind of get everything kind of up and running and we could kind of see that reflected inside of Elastic with all of the connections that we have set.

Speaker 1: Cool. Thanks for taking a look at that. So now let's talk a little bit about manual instrumentation. So manual instrumentation requires incorporating particular code segments into your application to collect and transmit telemetry data. So you might be wondering like, okay, cool, why would I do this? When would I do this? And so A couple of ways that you can do this is with an older code base. It's great to kind of like go and get started and kind of use automatic instrumentation to kind of get started. But also if you are more of a developer or you want custom tracing or custom information or maybe custom metrics, things like that, you might want to use manual instrumentation.

Speaker 1: Also for op like SRE um auto instrumenting might be really helpful because you can just kind of get something up and running without having to touch The code. I often have been using it for my demo applications, so you can actually see what's going on. And like I find it very helpful to kind of just get something, especially for smaller applications, just up and running super quickly. And so if you wanted to add manual instrumentation, the first thing that you want to do is you want to delete this Django instrumenter. instrument line of code that um inserts the automatic instrumentation, delete that. Then you want to update your views. pny. I'm adding in a metric in here in this piece of code to kind of let me know kind of what's going on.

Speaker 1: And then from there I can actually start adding traces to my to-do list application which is kind of cool manually and then from there I can add it to the add and the delete and it's pretty much like a pretty seamless process. I can actually kind of go in there and start to kind of like configure it as needed. So that's something that's really fun. And then from there, I just need to update my models. py file. And then yeah, should be able to get everything up and running. Let me show you what this looks like. So

Speaker 2: to run our manual instrumentation, we're going to do exactly what we did before. So we could do Python. Manage PY run server and this starts our server up locally Now we can see our server running and we see our to-do list page. We can actually go in and delete. We can add a new item in here so we can call this new item adding a new item for demo exclamation point let's press add there And then if we go into services, we can actually start to see everything.

Speaker 2: So if we look here, we refresh the page, we see this to-do list app manual. And keep in mind it could take up to five minutes for things to show. But we can see our manual index view span here. And we can actually also if we run everything, we should actually see a little bit more if we refresh the page. Yep. So from here we can see the index span, the add item span, the delete view span. This is just what we did. We can also see our throughput, any failed transactions. Um any dependencies, errors, things like that. We can actually see kind of our instance and pretty much everything kind of up and running. um as we can kind of take a look at the servers. So yeah, everything is sending and working correctly.

Speaker 1: Cool. Um there's also another option as well. Um open there is an open telemetry collector, so this is a solution for receiving and processing and exploring telemetry data. I actually have an example up in my GitHub if you want to take a look at that. But it eliminates the need to manage and maintain multiple collect agents or collectors. And you could do that all in one YAML file. And you can actually see an example up on MegitHub if that's something that you are interested in. So you might be wondering, like, how does Elastic fit into all of this? So you can actually use a collector. We actually have like a built-in collector for infrastructure for metrics and logs. But for also application metrics, logs, and traces, you can actually

Speaker 1: use the language SDKs. And I'm using the Python one and the From there, that goes into the Elastic Observability backend. And the Elastic Stack natively supports the OpenSilemetry protocol, which means that trace data and metrics collected from your applications can be sent directly to the ElasticSec. And this is a work in progress. It's a very young project at the end of the day. And as one of the talks I was Looking at before giving this talk, um, somebody said that it's getting better all the time, that they like had tried to use an older version and it was harder than It is now and I saw some, you know, older documentation that looked a little bit more complicated.

Speaker 1: So I found that like It was actually pretty easy once I got the hang of it to actually kind of start instrumenting my code. I thought that automatic instrumentation was like super helpful. And then also If I wanted something more customized, I can very easily use manual instrumentation and get everything up and running for a small application really quickly. And lots of features are experimental. Not all tools at the moment support all the features of OpenTelemetry. It's a very big project, so there's lots of different things and it's also Still a work in progress. You can actually see the status here. So it actually, if we take a look at Python, we can actually see that it's stable and then logs are still in development. My understanding is that's because of the way that

Speaker 1: kind of open telemetry got started was that um logs came last in terms of the development of it. So I have some closing thoughts. OpenTelemetry is an open source vendor-neutral observability framework. It is designed to integrate with any backend system. It's designed to be feature-proof. It's really cool. It's really easy to start instrumenting your code. It's really, really easy to kind of get kind of the hang of it. It's highly configurable and extensible. So you can probably make it work for whatever you're looking to do. There are some ways that you can kind of like extend it, including like adding your own custom open telemetry. receiver to your open telemetry collector, you can

Speaker 1: load custom interpretation libraries into an SDK. You can create a distribution of the SDK tailored to your specific use case. There's lots of things you can do. And it really can be configured to meet your own use cases. It scales very well from large-scale applications to small-scale applications. I showed you a small-scale application, but it's pretty much the same process. for instrumenting a large scale application as well. And yeah, there's some next steps. If you're interested in taking a look at my code and taking a look at any of the resources I recommend, anything like that. Feel free to do so at GitHub, github. com slash Jessica Garson introduction to open telemetry with Django. And let me know if this talk inspires you.

Speaker 1: to build anything i'm jessica garsen on most platforms thanks again um also always want to speak at jjango con and really happy to that i had the chance to talk to you today thanks have a great day Then I got number back then. And then we do the one with the hand on them. Do them the women.