Easy App Metrics by Frank Wiles

Speaker 1: Thanks everybody. They they said to make your intro fun, right? So I thought I'd I'd do that. Um so As many of you know, Django came from a journalism background, and we've had some talks today that relate Django in like literary terms, and so I'm going to take it back to the journalism roots and do a little bit of the who, what, where, when, how of app metrics. But I'm going to screw up the order a little bit. So first, who? Me. I'm the founder of Revolution Systems and have been working with Drango for a really long time. For this talk, the most important part of my open source world is a project called Django Appmetrics that helps make some doing some of this a little easier, and we'll get to that towards the end. But then we're going to talk about most importantly the why. When I first proposed this talk, I thought, you know, I want the shorter time frame

Speaker 1: because covering how to do metrics I can't fill 45 minutes with. Right, it would just get really boring and repetitive. But then when I went to actually do it, the landscape has moved and things have gotten a little easier. And so I can't even fill 25 minutes with it on how to do metrics. Uh the you know, the code part. That part has become really, really easy. The hard part is what should I be tracking and why should I be tracking it? So let 's cover some of those things. Um I come from Kansas, I had mentioned Django's from Lawrence, Kansas. My state has some trouble with uh believing facts And so but I, and I don't know about you guys, I am driven by facts. If you can show me some numbers and I believe that you Somewhat accurately collected those numbers, I'm much more likely to take action than

Speaker 1: you know what? I feel like that button should be a little bigger, or I feel like we should make this feature more prominent. So this helps give you ammunition with your coworkers and your bosses for things that you want to get done or what you don't want to have to do. So obviously we collect a lot of metrics and have pretty graphs for the the kinds of things you expect ops. You know, what's the load, how much disk space, those kinds of things. So we need it we need it for for that. Um but we can also use metrics to decide on development prioritization. If you have and also to be able to tell customers no. You'll get requests from your coworkers or customers saying, you know, it'd be really great if you could make it do this

Speaker 1: I know you do, you handle CSV, but really want an actual Excel download, and you're not too keen on it. Well you can go look and you you know what? You're the only customer that uses that feature at all. Right? Maybe we shouldn't prioritize that because it's just not something that's being heavily used. We can also use metrics to make design decisions. All designers are gonna hate me for this slide. I did like everything wrong, right? Like uh which is funny because this is the slide that took me the longest to actually design, right? So I guess that's good for designers. It's gotten so easy to do it right that you have to work hard to do it wrong. Um I'm pretty sure my Chrome doesn't even allow you to use Comic Sans

Speaker 1: anymore. Um so but You you there's a lot of things in on our sites, on on web apps, where we we do them kind of out of of history. We keep we put the controls at the top, and we put the controls again at the bottom. Well, if you track where they're actually used, it may be that the nature of your app is the user always scrolls to the bottom of the page because that's where the newest data is. And so they're only using the controls at the bottom. And so get rid of them at the top. You can get some screen real estate back. Maybe other controls should go up there. And if you track the difference, not just I used feature X. but I used feature X from this spot on the page, that can become an important inf piece of information.

Speaker 1: We can use stats against our data with with internal things, internal politics. Who should get more funding? Who should get a new staff member Should we bother to have those meetings? We talk a lot in our world about getting rid of the walls between dev and ops. But we can use stats to get rid of some walls between devs and ops and sales and marketing. Because if I can give the sales guys some numbers that they didn't know about or the marketing guys some insight into what they're wanting, now they're my friend. And they're going to be a little more understanding when I tell them, hey, this is going to take six weeks to build. Are you sure we really want to do this? I can't do that right now or whatever else. So this can be a way to kind of break down some walls inside your organization.

Speaker 1: So we're gonna grab lots of crazy metrics, but we have to remember that, you know Correlation is not causation necessarily. So if you see a big spike in your graph, that doesn't necessarily mean that the other thing that happened at that same time point is what caused it Right? So it's usually just a signal of I need to dig deeper into this problem. So what kinds of things should we be tracking specifically? So most of you can probably come up with the, oh, before I cover that, I need to cover costs. When I talk about costs on metrics, there's a couple of things to consider. There's the actual storage cost, right? I'm going to collect all this data. How much is it going to cost me to store?

Speaker 1: There's the time it takes me to implement it. And then there's also the what this does to the runtime, right? I don't want to slow my app down considerably by tracking nine million things on every page or every action, so I need to balance them. But Keep in mind that metrics are pretty darn cheap. If you set your retention policies right with graphite or with hosted services, we'll get into some of that. You can store a metric for like 20 megs of of s of of space for a 10-year period. So don't be too afraid of of collecting too much data. But do keep in mind that they're you know they're not absolutely free. So we we know to track the default things, right?

Speaker 1: Load disk space. memory usage, how many WISGI processes we have going, how many worker processes are running. Most everybody knows to track that stuff. And it's a good place to start, right? It's better than nothing. But there's other things that you might not think about that are good to be able to overlay and look at along with feature rollouts and bugs that can help you Solve real problems. So table sizes, right? Not just how much disk space does the database use. What happened? I did this deploy and all of a sudden this table's blown up in size. Was that Does that make sense? Or is that maybe something really bizarre and a bug that we just haven't detected yet? How many deploys are we doing? How long do they take? When did we have outages?

Speaker 1: Uh how many times is somebody SSHing into the server? That can be a really good metric of we're not doing enough ops automation. And it's something that you can easily script up and and keep track of. But more importantly than just keeping track of some raw numbers, we need to make sure that we segregate some of this data. So like I use this bottom example here of support tickets. We don't want just tickets created in resolve, but we want, you know, is this created by somebody internal versus external? What department? You know, we want to have a little bit more dimension to some of these these data points so that we can see are these support tickets all generated by customers or are they all generated by that one guy in that one department who's really just kind of you know a pain. Right? Um so we have basic app metrics, and this is kind of like what marketing and sales wants to see and growth hackers.

Speaker 1: Right? You want to see like signups and free paid and upgrades, downgrades. We all know that kind of stuff. Uh you know how many people like you on Instagram or whatever. Uh but you know there's some other things like What emails are we sending to customers by type? I mean we know how many we sent, probably, if you use Mail Gun or a service, like they give you an invoice and say you said 4,812 last year or last week. But By what type? Is though are those all password resets? We don't know. Um, you know, how many times are people logging in? Logging does anyone ever log out? You know, that can be a design decision of we don't spend any time on what our logout page looks like because one person does it every five days. Um so you know most people do the cookie just times out. You don't actually physically log out.

Speaker 1: So Having that data helps us to see how things are being used in the app and where we should be focusing our time, right? In bug fixes, in new features, in performance. You know, a lot of people track how many items are in the queue, but they don't necessarily track how long individual types of jobs take in that particular queue. So, you know, how long does it take from I click password reset to I've told Mailgun to send the email? If that's a second, cool. If that's 15 minutes. Not cool, right? Because I'm I'm a user and I think something's wrong and now I'm hitting it five or six times and I'm plugging up your queue because I kind of expect to get an email within a minute or so.

Speaker 1: You know, I don't necessarily expect it to be instantaneous, but I expect it to be a minute. Um what's our cache hit rates? You know, we're using caching, but uh, you know, are we using it ineffectively? And on like API usage, is this all unauthed public API access? Is this all, you know, auth or just our normal customer? our customers, what endpoints are being hit the most. Sometimes you'll find that a feature is used very heavily from an API and not used at all from your desktop app. Or use the lot desktop and API, but never on mobile. And so you can de-emphasize that in design and in terms of bug fixes. And we should track random internal stuff. When did we have meetings?

Speaker 1: How many staff members did we have then? Because we might be able to correlate bugs fixed to how many people were working. Right? How many people were on vacation that week? We might see that when one particular staff member leaves, that Bugs go up or go down. Maybe that person was more important than we think, or that job was more important than we think. How many times your coworkers are having to work more than 40 hours? Are we generating problems because of burnout? We can track these things, right? But we can't correlate them without some sort of visualization. And you know, we should track chat messages and emojis used and you know but we we can have some fun with it, right?

Speaker 1: I I I have this coffee example here of did you do it in the office or did you go to Starbucks and Was it grande or vente or whatever? To to not so much that I want you to track your coffee consumption at that level, but to think about that on tur in terms of metrics. It's not just how many celery tasks did we do. It's which of them and why? And you know, on what kind of spectrum were we doing this? Was this how many free users used that feature versus paid users? So in the when, well you should be doing it now. You should have done it yesterday. Um But we can talk a little bit about uh time travel here in a little bit uh on how to do this, but you need it before you need to have some metrics in place before you really need them, right?

Speaker 1: So start collecting them early, even if you don't have a good plan to visualize them So how do we go about doing this, right? There's the easy stuff. You know, Google Analytics. Everybody's got that on their site, right? And then there's some stuff that helps with ops. You can use opbeat, who's one of the sponsors here. You can use New Relic, Datadog. Those are pretty easy to set up and they give you a lot of insight into your app. Um I kinda hate some of these services because before they existed everybody had to come to RevSys for performance help and now they have these services and they can see, oh, it's this one query that's making my site slow and and uh so but then there's Other services like Librato and MixPanel and Keen. io

Speaker 1: that let you kind of create your own metrics and create ways to graph them. And it's the kinds of things that you would do if you were hosting your own metrics uh without having to do any of the ops work or set up yourself. Those do they have different pricing and and some different features, so check those out. But if you really kind of want to get to the the top level, right, you got it you've got to do it yourself. So how do you do it yourself? The best way to visualize is Grafana. There are other dashboard systems out there, but if you don't have one picked out that you really love, you should look at Grafana. Um it is beautiful. You can show it to bosses and clients and everybody likes looking at it. Nobody's gonna say this looks like mapplotlib or something that you know we

Speaker 1: we did for fun. But you know so pretty is important. If people don't like looking at it, they're not going to look at it on a regular basis. And Grafana's actually besides being beautiful, it's easy to use. It can target graphite and influx dB for its its sources and more importantly it can target multiple of them so you can kind of shard your metrics as you need to to grow. So you can have some metrics pulling from cluster one and some from cluster two all on the same dashboard. So you need to I the one of the mistakes I see people make is they have one big dashboard with 9,000 things on it and nobody's gonna look at that uh and nobody's gonna find that useful. So create special purpose dashboards for the kinds of things that you're looking at. at what things do we care about in this release?

Speaker 1: What things does sales care about? What things do marketing care about? But Once you've got them set up, you're gonna look at them a lot, maybe stick them on a big screen in the lobby and have them rotate and you're gonna kind of stop looking at them. But you need to be able to uh dig in and look for new and interesting facts. So spend some time every now and again just overlaying different metrics to see what see what falls out. I think the main reason people don't do metrics is because graphite is a pain in the butt to install. Who here has wasted a day trying to get that graphite set up? Okay. So most people maybe I'm wrong. Maybe people don't use it because it's hard. So there's an easier way, and that's graphite API. It gets rid of all the hard parts of the setup and gives you just the parts you need to work with Grafana. So check out Graphite API.

Speaker 1: And then InfluxDB is the new kit on the block. It's written in Go and it's the hot new thing. Um its main difference is you can have individual metric points and then attach arbitrary tags to them. So you instead of having in Graphite, you would have CPU load for US East uh you know uh box five. With InfluxDB, you can just have CPU load and have your availability zone and instance type be just tags against that. And so you can do more aggregations and querying more easily So how do you transport these metrics around? Um you use statsD. Stats D is a really easy service to run. The biggest mistake people make is they have a stats D system. and then they have like 20 web servers all talking to it.

Speaker 1: And that actually gets slow and it can slow your app down. So run one on every spot you're going to collect metrics and use it to shuttle them to wherever they end up getting stored. Another thing that people don't tend to see notice is that if you're using Logstash, you guys are taking logs, right, and you're doing stuff with logs and shoving them off into a service. You can use Logstash to generate metrics off of logs based on regular expressions. So you don't necessarily have to retool your app to put this stuff in. You can generate log metrics off of logs. And then if you want to do front-end metrics, then all you need is a little API that you can just Suck these in from your JavaScript, right? And shove them into the rest of your your back end. So how do you do time travel?

Speaker 1: Most of these systems you can pun you can push into with a a specific time point. So when did this happen? So you can go back and replay all those logs you've been storing in S3 to come up with what the data that you're looking for to show trends over time. So how do we do this with Python? The Stats D app is very simple. You get a Stats D client and then you have a counter that you can increment. You can increment and decrement. You can increment by more than one. These are just simple examples. You can have gauges. So right now my stress level is at 25 % out of 100. I'm just picking arbitrary numbers here. And then uh you can do timing. How long did I sleep last night? Well I got about eight hours. Uh you just need to store it in milliseconds.

Speaker 1: So you can have timings of how long did this particular thing take. And then influx db, um this just uh is very similar. But uh they take a specific format that's the measurement name, whatever arbitrary tags you want to attach to that, and then the time value, and then what the value of the field was. So Giving Django a 10 out of 10, and I'm attaching that it was in Austin and that barbecue is the predominant food. So, to the Django bit. Django Admetrics is uh a little utility that is kind of like uh haystack in that it abstracts out where you're storing this.

Speaker 1: So you can use Django Admetrics and target mixed panel, Labrato, Stats D, store them in your database, store them in Redis. You know, so if you have a personal home page you want to collect some metrics on, you don't want to set all that stuff up, just store them in the database and you You can do some simple aggregations and roll-ups and uh or in Redis. So you can put this in when your app is small and change where you're storing them over time without having to retool everything. So you can have a simple metric like uh code examples, so that just increments the code example metric, or we can time things with a context, or we can set a gauge If the NSA knows more about what your users are doing with your application than you do, you're doing something wrong.

Speaker 1: If all you're using is Google Analytics, they probably know more about what is going on in your app than you do. So embrace your inner NSA agent and get in there and be a little Snoopy. Anybody got any questions?

Speaker 2: Hey Thanks for the nice overview. So uh my question would be uh how do you decide how to generate the data if you're gonna do it directly from your application or you're gonna log it and then have something parse it out? Like how do you how do you make that call and uh what comes into the decision?

Speaker 1: I mean I think that the biggest is do I already have the code written or not? Like if I've got a big app that's got a lot of things going on, I'm probably going to try to collect the metrics from the outside. So I'm gonna I'm gonna try and pull it off of logs. Uh the other reason that I would might use uh off of logs instead of inside the app is because of performance. If I don't need the answers in real time or near real time, but trying to collect that metric any other way would slow the app down. I'm going to push it into Elasticsearch and Kibana and pull or use Logstash and pull those out and maybe re-transmit that into something like Graphite just so that I can have it overlaid with the other the other bits.

Speaker 2: Thanks.

Speaker 3: Do you have support for the InfluxDB tagging kind of stuff in Appmetrics?

Speaker 1: I don't yet. I had hoped to have time to add that and uh for for today, but I haven't done the Influx DB back in yet. Okay. If anybody wants to help with that, that'd be awesome.

Speaker 3: Sprints.

Speaker 1: Yeah.

Speaker 4: Hey, I was just wondering if you had any tools or recommendations for more like more of those esoteric uh metrics you were suggesting. Like the the culture y type ones like pull requests or

Speaker 1: yeah I mean well pull requests and things you can so GitHub will do webhooks, right? So you can listen for those and then just shove metrics and you can go and backfill that pretty easy. easily by hitting the API. Things like do we have a meeting? You know, like I you know, do you maybe off your calendar, you look for the string meeting and automatically pull them in. Maybe a very small little web app that you can just hit a button and say, hey, I'm I'm in a meeting. Or, you know, maybe you want to track good meeting versus bad meeting. Like You know, um I know that when I was able to you know show that all the requests were coming when I was an IT manager, all the most of like 70% of my requests that were eating up most of my time was coming from one department. When that department complained about how long things were taking, I said, well, you know, almost all my requests are coming from you, and uh

Speaker 1: y you know, how about you give me some of your budget? So I can hire more people and we can both flow better. And he's like, absolutely, I didn't realize that you're doing all of our work, right? But if I didn't have numbers like that, I'd just be You know, he said she said kind of arguments of no it's your fault, no it's your fault, it's Steve's fault, you know.

Speaker 5: Hi, uh I'm from Upbeat. Thanks for the mention. Um I have a sort of almost cultural question. Um Do you have any tips on making your coworkers or co-developers uh care more about performance

Speaker 1: so I think like anything you know um If you measure it, people care about it, right? If you can show them that your work over the last week slowed everything down by 10%. They're gonna start to at least check that. It may be valid, like it there's that there was no way to make it any faster and the feature was necessary and so you you can't get You know, you can't get too hard on them for it taking longer with more functionality. But, you know, most of the time nobody's noticing, no one's looking. You know, and you can't if if you're not seeing it on a change by change kind of basis, you you can't figure out it's harder to figure out the root cause if you only look at your performance numbers every six months. are only when the boss starts to complain that the homepage has taken too long to load. Like, what

Speaker 1: what was it? I don't well go look through the last 8,000 commits and and we'll try and figure out what where the problems were or who the problem is, right? It could be that 90% of your developers are kind of performance savvy and there's the 10% that aren't and they just need some mentoring. Uh but if you're actually looking at this stuff on a regular basis. You know, if somebody gets an email, uh it's like with uh testing. You know, I have projects where if testing falls below eighty percent coverage it doesn't pass CI and then can't be deployed. So boom, everybody's just gotta keep that up. And sure, sometimes people just game the numbers, but like if it's something maybe you need performance testing like the guy was talking about in the last talk and and have that be something that's

Speaker 1: you know stopping the build.

Speaker 5: Thanks.

Speaker 6: So this is kind of a tooling question on Grafana. You can get data from a lot of different places. and you can put it in a lot of different places, but knowing what data to put together to actually get meaningful insights is fairly Kind of a pain.

Speaker 1: Yeah.

Speaker 6: And uh is that something where Grafana makes it kind of easy to take these disparate uh pieces of data and Try to match them together in a goo.

Speaker 1: It makes it easier. Makes it easier. Okay. It makes it easier. I it's not easy. And that's why I say like I think you should go play, you know, a little bit. I mean you nobody has all week to sit around and just you know, graph numbers on top of each other and see what sticks. But you know, try and you know tet try and look at everything every now and again and see does any of this match up with anything else? Does it, you know Does overworked mean more bugs, you know?

Speaker 7: So kind of touching on that last question, you touched on earlier in your talk about how these aren't for free, but I was kind of curious if there were any like good guidelines or any hard and fast rules in terms of monitoring every little thing in the office or in your project or whatever.

Speaker 1: I mean I think that you know I think we could all agree that if you needed to hire a person to track shoe size at your company, probably not a metric worth collecting, right? So I mean like there's time costs, actual costs, you know. So I it It's a balancing act, but if you have a time tracking system already that you're using for other reasons, then pulling data out of that and shoving it into something that you can graph against other things, you know, that's probably worth the 30 minutes it takes to do that If you've got a calendar or you know all meetings happen in the conference room and you have uh an app that tracks when's the conference room in use, it may be worth collecting meetings. You know, um it may be worth collecting employee happiness if you have other reasons that are doing that.

Speaker 1: If HR has got a system that surveys everybody every three months, it might be worth collecting that and shoving you know, min-max average and mean into into Grafana and see if anything shakes out. Cool.

Speaker 7: Thank you.

Speaker 3: All right, that's our time. Thank you, Frank.