How to Enjoy Debugging in Production

Speaker 1: Hi there, I am Karen Tracy and I thank you all for coming and I thank the organizers and everyone who in is involved in setting this up. have very much enjoyed this conference so far and hope I'll enjoy it more once I'm done with this talk. I am a Django developer for something like 17 years. I've been working at Cactus. for something like almost 15 years. Cactus is a small web application development company in North Carolina, United States. We've always been a pretty small, we've gotten a little bit bigger, smaller over time. We've always been pretty small so that over that 15 years of working for Cactus

Speaker 1: as a developer, I've had a lot of opportunity. Most developers have the opportunity to support their code in production once it runs in production. So I've had some experience there, which I wanted to share. I do enjoy debugging in production, so I wanted to share that. I am aware that some people don't really look positively on the idea of debugging and production. Who here enjoys debugging in production? Yay, there are others like me who really want to have nothing to do with it. Yeah. Um so depending if you're a developer you may look on on launch day as a as a you know, new beginning of figuring out how well your your development has met the expectations of whatever project you're working on.

Speaker 1: Or you may see it as an end, like I'm done with the project. I can get it off my machine and I can move on to the next project, and I don't really want to deal with production issues. And I think the reason for that is uh There are definitely some downsides to debugging in production. Primarily it can be very, very high pressure. You've got um clients or higher ups in the organization who are perhaps afraid that the problems are causing a loss of money, a loss of data which is going to turn into a loss of money and they want things fixed as quickly as possible. And simultaneously with that high amount of pressure that's getting applied to developers. You're working with a system that has a lot less information that's provided to you than you're used to in your development environment.

Speaker 1: You have no ability to just like jump into the debugger and see what's going on. You don't even have a debug page. If it's 500 error, you may get a traceback email, but you don't have all the information you could gather from the debug page. There's another con con kind of limited information you get when you learn that your real users aren't gonna provide the same kind of information you had when you were developing locally or your co-working testers knew to provide, you know, how did you get into this situation? What do you mean it doesn't work? You know? You're gonna get a lot of it doesn't work and if it's not a 500 error, you may have to dig and figure out. Um What exactly is going on? What's the nature of it doesn't work?

Speaker 1: I think both of these can be addressed by uh Things I'll talk about a little bit later. You can't really get rid of all of them. Like there's always gonna be a little bit more pressure debugging something on production than on in staging or in your development environment. And you're not going to be able to get all the information you could get from your development environment on a deployed environment. But I think You can make these things better and you can alleviate and make production debugging more fun. Before I get into those though, I want to talk a little bit about well why don't we just make it not necessary? Like why don't we just make it right the first time and not put bugs in production? And there was some talk of things like that.

Speaker 1: I was interested in Mia's talk this morning. She said, you're going to be surprised by what production data looks like Thank you. Um, So unexpected data is I'd say one of the most common things you're going to run into. You can try as hard, and as I said, I've been doing this for years. You can try really hard. Now you need to adjust the mic. Okay. You can try really hard to have tests that test all the unexpected data you can think of and there's gonna be something else that gets thrown at you Um

Speaker 1: so I'll go I I'm gonna try and save some time and not go into examples. Um When you're working on a project, another this is a one class of reasons for production issues is specifications that are in conflict with reality. And often when you're working on a project it's it's involves some like multi-step process and you're gonna say that, okay, we're gonna do this in phase one and then this data is fit into phase two and in phase two you're not gonna change that data. It's gonna be fixed. And that's a red flag. You're gonna get into a situation on production where you need to change that data. As an example, a survey application. You're gonna say, okay, you're gonna upload the survey, it's gonna be very you know reviewed, checked out. And you're gonna open the survey, people are gonna start taking it.

Speaker 1: You're not allowed to change the survey then, because you've already got people answered the questions. You don't want to change that, that you're gonna invalidate your survey. Um it's gonna happen. A translation is gonna be left out. And you're gonna be left with the option of, okay Spanish speakers don't get this question really because they get it in English, or you allow somehow to update that data and insert the correct Spanish translation. And then you got some people, you may you track of when you did it maybe so that you can know what r what responses are associated with which question, but you're going to need to be able to update the data. So any kind of thing like that where you've got um A process that says you can't make changes after, just like keep it in your mind that says, mmm, probably I'm going to need to change it later, and think about what the implications of that might be.

Speaker 1: A related thing is a handling should not happen cases. Clients and higher-ups often want to focus on the common case and say, okay, well Deal with the happy path. Don't really worry about the unhappy path, you know. If you're supposed to be locating a user locating the company associated with a user that's being pulled from some other system based on email address. And you said, well, a user is only going to be associated with one company or none, but they should never be associated with more than one. But that system doesn't enforce that. You're gonna run into situations where you find a user who has multiple companies. What are you supposed to do? And you might get a lot of pushback.

Speaker 1: Say, well, you know , we're just not gonna ha have that happen. It's gonna happen. Um so it's helpful to at least get enough feedback in you should you fail it should you like fail ingestion of this data and say this data's invalid, I can't have a user associated with multiple companies? Should you let it happen and just kind of Pick a company when you're asked which company is this user associated with, if you could get some guidance and leave some logging trails or something so that when something happens in production you can say, oh well that's because this user has this case that's not supposed to happen and we need to fix that and then everything will work properly So these are some of the reasons that you're gonna hit hit this is one class of reasons for your

Speaker 1: hitting errors, issues in production. The second class is Related to the difference between development and production. Your development environment is not your production environment. There are differences and they are gonna crop up. Um one of the big ones is single versus multiprocess. Running run server on your machine is a single process. It's all got all those It's multi-threaded, so you can do multiple requests at a time, but they're all sharing the same memory space. And it might happen that you do something that involves setting a module level global variable that as a kind of cache maybe that's not going to work on production because in production you're generally going to have multiple processes no matter what.

Speaker 1: So something to keep in mind. Uh CPU resources is another CPU disk like you the I was kind of surprised when I started working on web development. I kind of thought web servers out in the cloud, they're gonna be big beefy servers. They're gonna be way more capable than my development machine. They're not. It turns out I could do stuff on my development machine and it worked fast and worked just fine and then it got deployed out to a staging environment and that environment was not as capable and it took longer and we needed to be a little more efficient in the coding we did. So um It's just interesting that interesting thing about the diff uh an unexpected difference that I learned when I first started working in web development.

Speaker 1: Another issue is traffic volume and the data volume and the contents of the data. Those are all generally going to be bigger in production than you're going to be able to do on development. You can write scripts that generate data You can use tools that generate load, but it's going to be very hard to really mirror what you get on production with a development environment. And then timing and concurrency is another thing. You're gonna hit bugs in production that that are related to concurrency that You may have tried your best to think about it and make sure you use select for update or get and create and everywhere you need to do that and you're gonna just realize When you hit a bug in production, oh, we forgot to lock something and make sure that we did it in an atomic way.

Speaker 1: So these are reasons I think that you're always gonna have some things that you're gonna have to solve in production Uh so how do you deal with that? First I'd say build on a solid foundation. You want to understand the problem you're solving and you want to understand the solution you're you're providing you want to do all the good things and make sure that the product you're building is really solid and Choose good tools. Hopefully you have the choice of choosing tools. If you have a depending on your interactivity requirements, you may have more Interactivity requirements that may influence your choice of front-end framework. Of course, I'm going to say you probably want to choose Django for your web framework.

Speaker 1: But Choose tools that match the problem you're solving. Do all the general good things for a good code development. Code review, I think, is very important. If you're in a team that's so small that you don't really have the ability for someone else to review your code, I and I even do this in teams when I have a someone who can review my code. I like to like let my code sit and look at it with fresh eyes before I send it off to the next person. I like to to review it with fresh eyes myself before sending it off. And do all the unit tests, edge cases. You know, test as much as you can and try to build a really solid foundation so that and what this does is help reduce the number of times you have problems in production because you have built something that is

Speaker 1: Solid and that can help reduce the pressure that gets applied because if you have very rare cases of problems in production, then higher-ups and clients are tend to be likely to give you a little more grace in solving them versus if they're happening all the time that can that can really amp up the pressure. So building on a solid foundation helps to make it easier to debug in production. Next step I say is have robust testing on a staging environment that is as close as you can make it a replica of production. You want to be using the same software, you want to be configuring things in the same way. The very specifics of configuration may differ. You may want to lock down

Speaker 1: staging so that people can't generally access it. So you may want to put basic auth on the web server. You may use different API keys for different services. You may want to lock down the ability to send email. But you want something that is very much like production , particularly in the software and its configuration. then um if possible and if you can easily move a database from production to staging that's a great way to get production data on stag. You may have data that you don't really want to replicate on staging so you may have to include in that some sort of modification of changing emails or changing names and stuff so that you don't duplicate your potentially sensitive data from production to staging.

Speaker 1: But getting at least the volume of data you have on production can be useful. And then I like to encourage wide use of it. I like to use staging server myself to see replicate problems. Of course test uses production. I also like clients to be using production and I like clients to be using production for their own tests of the stuff that they want to be doing. If they're opening a new something or other, working on more content encourage them to use production and see it as a way of staging rather and see it as a way to test things out and as a way They might hit problems with what they're doing or they may notice new features. And it's kind of like a soft launch. It's kind of like it's a place where you can learn what issues you might have in finding errors in the deployed environment and how to overcome those limits.

Speaker 1: What do you need to add to make it possible and easier to debug on a deployed environment? And that goes So next step is when you have production, you want to know what normal looks like. And this is this is more related to if you're going to have traffic issues that are or issues that are related to the volume of traffic you're getting, perhaps, or the content of the data. on your deployed environments, you're going to want to know where the logs are, how you search the logs, how you can look at logs, and you want to know what's normally logged. Some of the services that are generally included with a deployment are very chatty. They log a lot of stuff. And you may look at that and say, this doesn't look good.

Speaker 1: But if it's normal, then you don't want to be distracted by that when you're looking for a production problem. Or examining the logs may alert you to something that's not quite right in production and that you could fix. So I'd encourage looking at the logs. the system logs, database logs, uh whatever logging to make sure that things are working okay in your normal production environment. And I didn't get real specific here, but you want some kind of monitoring that shows you what kind of traffic you're getting, what kind of response time you're seeing, what the CPU, the disk, the network, the throughput utilization so that you have an idea of okay this is normal traffic and then if you've got periods of very high traffic you know that's not normal and you can see how close your normal traffic is to meeting

Speaker 1: to reaching potentially limits that are gonna cause it to fall over with ex excess traffic. There's all sorts of monitors you could use. I've used things from as basic as something called MUNIN. New Relic, uh Data Dog mentioned Net Data is another one that I've seen uh recently that I really like. So you want to have some ability to to kind of see into your production and deployed servers and see what what things look like normally so that you know what's different when things start to go south. Um Beyond just monitoring, you also want to anticipate that there are going to be problems and you want to have uh services in place that it will alert you to when there are problems.

Speaker 1: And the most basic would be uptime. So basic services that fetch a page and verify that you get a good response code. A lot of them will do it from around the world so you can know if you're actually accessible from different places around the world Error alerting is another sort of essential service if you have a site that does have a fair amount of traffic and something goes wrong and it's going to generate um thousands upon thousands of error emails if someone you know some something happens, uh an error alerting service that consolidates all that down into one one alert to you and then aggregates the information that so that you can kind of pour through it at more at leisure is being valuable

Speaker 1: because having your email inbox bombarded by thousands of errors at a time adds to stress um if you look at your email. Another thing to think about is if you've got things that are supposed to be happening periodically. And so you've set up Celery, it's working, and it runs every day, and everything's fine But for some reason it stops working. You may not notice, you may not notice the side effect of whatever it was doing hasn't happened yet. So we've taken a way of making sure that if something is expected to happen, we've coded it in a way that if it doesn't happen, we get an alert. And I'll go down at the bottom there. The

Speaker 1: Health Checks I. O. service is really great for this. You can configure it to say this is something that should happen. every day, give it a grace period of an hour and if it doesn't it doesn't get alert if it doesn't get a notification that it's happened then start alerting me and you can configure how do you get alerted. And what it gives you is a URL and then in your code what you do is you get to the end of whatever task it is that you want it to have done and make sure that it happens and you ping that URL. So you send a request to that URL So the health checks keep track and it says, okay, I heard from you today, all's good. And then if it doesn't hear it from twenty-four hours, twenty-five hours say it would send you an alert and then you can start anticipating you can start investigating what's gone wrong and potentially fix it before your clients or your users have even noticed that there's a problem

Speaker 1: Now I'll go back up. Some of the essential services that we also always use. We use a lot of AWS infrastructure and there's a lot of alerts you that you might be setting in your monitoring and that may be an easier place to set alerts to let you know when things are getting close to limits. Um for AWS though. They have a and maybe other providers have this as well, but AWS has some instance types that have uh are a little bit more affordable by having a lower CPU throughput, a lower disk throughput for some disks. And the thing is it they allow you to burst higher and burst higher usage. And

Speaker 1: they what they do is they have something called a credit balance or a burst balance. And y all's good if your burst balance is positive. But if you start to get traffic that continuously depletes your burst balance for CPU or disk. Everything's gonna be fine until it gets down to zero and then you may be working on a machine that just doesn't have the capability of supporting whatever traffic is coming in and everything falls over and you're like, what? What's going on? Like nothing broke So it's really it's really useful to have a mo an alarm in AWS to say, hey, this burst balance has gone down. Maybe you need to up the disk capacity, not the not the capacity, but the throughput. you can upgrade to a different kind of disk and you can do that on the fly and then you can see your burst balance go back up and you can um

Speaker 1: handle the traffic that you're getting. So that's AWS alarms that I find particularly useful are those burst balance because they are useful to have to save costs, but um it and they can be they can be difficult when you get down to zero And Sentry I. O. is another one that I'll mention as the error alerting, uh the consolidating the errors so that you Don't have to get a thousand emails, you can just one email and it says, okay, this problem is affecting this many people, and it can organize the errors for you. So in conclusion, I'll say I like to celebrate launches as a beginning of learning how well we've done. I do think it's important to avoid unrealistic expectations of no issues.

Speaker 1: You want to minimize issues in production, but I think it's healthy to realize that you're not going to be able to solve everything before you launch. It is very helpful to be prepared, to have an idea. Okay, well this particular issue constraint that we coded might be a problem, um, and to have thought through how to potentially solve that in production, like hitting a should not occur case. It's always also good to be prepared to know what normal is, to know what the normal characteristics of the system are. So if you get hit with traffic that's unexpected, you know what limits you're hitting. Um and it's also uh helpful to do this for a while. Um practice helps. So hitting issues in production and figuring out for an anything that does happen.

Speaker 1: We always try to say, okay, well why do we hit this problem and how do we make sure we don't hit this problem in the future? Is there a way that we could enhance our development process not to hit that. And on the bright side of production issues, uh for me, I like to I like it to know that my code is being used. I like to, that gives me a warm fuzzy feeling. So any production problems usually indicate that actually the code we wrote is being used, so that's a good thing. And I have no idea how much time I have left, but if we have time for questions, I'm happy to talk

Speaker 2: Thank you for the talk. Um do you feel like because uh bugs in production are inevitable, uh that means that you can uh Perhaps only test the happy path or not optimize too early or something like this and then see what happens? Or are you stay saying It's more that you should uh be aware that you're still going to encounter it even with the best efforts.

Speaker 1: I don't I I am I would say you should test as much as you possibly can. We have some great testers that uh know to test the limits of input fields and put weird characters in And we try to do that in unit testing. So I would say test as much as possible, try to get it as solid as possible, but be aware that you're probably it's still gonna hit issues in production and have the tools ready so that you know how to start addressing those issues. Yeah Hi

Speaker 3: Karen, thanks for the talk. I I loved it. Um one thing when I saw your talk coming up, I one thing I imagined you'd be talking about is SHH'ing in and YOLO editing things to see what you if you can fix things, but you didn't mention that.

Speaker 1: I didn't mention that. Um

Speaker 3: what 's your thought?

Speaker 1: Usually try to avoid that. I can't say I I can't say we do it all the time. Um sometimes it is necessary to SSH in and fix things on the fly. Uh I would like I've done that for changing data. Like this data, we did not offer a way to change it in production. So, but it needed to be updated and in some cases we could use the admin to do it, but in some cases it was safer to actually do it via shell on the on the production site. You need to be very careful doing that. But yeah, having the ability to SSH in or get into a pod or whatever is useful.

Speaker 1: But it's a last resort. In my mind.

Speaker 4: Uh sorry, but we don't have time for any more questions. I just want thank you, Karen, for the speech.

Speaker 1: Thank you.