How to Enjoy Debugging in Production with Karen Tracey

Speaker 1: Hello, can you hear me? Can't necessarily see me unless I stand to the side. Uh hi, as uh As just mentioned, I am Karen Tracy. I'm here to talk about how to get closer to the mic, how to get how to enjoy debugging in production. Who here is works in production and helps solve production problems as part of their daily and who enjoys it? That's good. I actually do enjoy debugging in production and I've found over time that there's a lot of people who don't enjoy it so much. Okay, so I actually do enjoy debugging in production,

Speaker 1: and I hope to share some of the things that I think make it a little bit more pleasant than sometimes is the case. Okay, I can reach that. Uh I think part of the reason that I do enjoy debugging in production has to do with a perspective I have on launch day that I've always had, which has to do with like that's the beginning of something. It's a beginning of a new phase of a project when we start to learn how well what we've been building on, what we've been working. isn't going to match the reality of what users are looking to do with their their use of our product and starting to get real feedback and I find that exciting. I do know that some people, and if particularly if you come from a large company background, you may have more of a perspective of

Speaker 1: uh launch day being sort of the end of a project. If you're a developer, you work on it in development and then you sort of hand it off to the customer service and the the change team people who deal with it in production. And some people I I one thing I really liked about moving from a bigger company to a smaller company is I started to get the ability, the opportunity to work on things in production. I like working with c I like getting that feedback from production and and learning how what I've what we've worked on, what we've built is getting used and I enjoy that. Um where was I? Uh I don't know. Um but some people are more used to That being the end of the project

Speaker 1: and they move on to the next project they're going to develop and someone else supports it in production. And that they're not really interested in delving into supporting in production. And there are good reasons for that, so I'll start with that. When you have issues in production that you have to solve, often there's a lot of pressure and it can be very high pressure situation where The client or the man upper management in your company is seeing issues and they're seeing a loss of dollars and they're seeing a loss of data, potentially a loss of clients or customers. And there's going to be potentially a lot of pressure to get things fixed very quickly. And that can that can cause a lot of

Speaker 1: uh high pressure to get placed on the people who are responsible for uh achieving that. Simultaneously you may realize that, oh, in production , You don't have the same tools and abilities you had in development. You may have far less information about what went wrong, far less information about what exactly is happening and how to fix it. You may just get be getting feedback that it's not working. I was like, what do you it's not work what isn't working? What are you expecting? What so it can be it can be a very high pressure situation. So that can be something that that often makes people try to shy away from not wanting to get involved in production. But I think a lot of that can get alleviated by

Speaker 1: thinking ahead and sort of building with an eye-to-eye to making it easier and more pleasant to debug in production. I wanted to start out with talking through some of the reasons why we start hitting errors or issues in production. And part of this is there's two Two com there's two slides I have here about common causes of issues hit in production. And um Understanding these and sort of seeing them can help alleviate the issues. Like if you see them in development and see them coming, you can sort of steer away from them or at least mitigate what might happen. Um so I want to talk through those before I start getting into other things to make debugging and production more enjoyable. Um so the first

Speaker 1: A very common issue I see is an issue where you've been building in sort of an ideal environment of what you expect the users to want to do, the data that they're going to supply, and um the reality of what they're expecting or what they're going to do, then isn't doesn't doesn't mesh with what you've been building. So specifications at odds with the reality of what you have to build and what the users want. is often a cause of issues that arise in production. The three things I have listed up here, I'll just talk briefly through them. One is unexpected data. You have an idea of what kind of data you're going to be getting. Maybe you're getting it directly from the users and you expect that you're going to be able to clean it and it's going to be fine and you're going to ensure it's valid.

Speaker 1: Even in cases where you're in control of cleaning the data, you may find that users provide things that you didn't expect and that you can't deal with. I had not I had two different projects totally unrelated. Both had to create XML at some point with user input. Both of them hit an issue where users were somehow entering ASCII control characters in the form fields And we weren't catching it because it was fine. As far as we were concerned, that was fine. But the XML generator was like, this isn't valid XML. So we had to figure out how to strip that stuff out of the user-supplied data. So just be aware that you're going to get things you don't expect. And if you're ingesting legacy data from other systems, it's going to be far less clean than you would like.

Speaker 1: One of the things you can do ahead of time is try and get that data from your client, you know, get access to it so that you can make sure that it's going to be ingested correctly and you can deal with whatever weirdness exists in the existing data Another common cause of issues that arise in production I've seen in um Is when there's like a process that's going to be followed. And maybe phase one of the process, you're building something and it's going to create some data, and that's going to be fed into phase two. And the client and the upfire up say, okay, after phase one, this data's fixed, this data can't change. And you can rely on it not changing. It's going to be the same. Invariably, I think I've seen cases where that does

Speaker 1: it doesn't happen. Someone makes a mistake. Um, and you're gonna have to be able to say, okay, we're not supposed to change this data, but in fact we need to change this data. For example, if you're opening up a survey, you're creating a survey situation, a survey thing, and thousands upon thousands of people are going to be taking the survey, you wanna Make that you want to not change the survey once it's open because you don't want to be asking some questions to one set of users and different questions to a different set. So you're gonna want it I mean that's perfectly reasonable to say in in once the survey is opened you can't change the data. Um but if a mistake happens And a survey opens and your client starts getting feedback that, hey, this question doesn't make any sense or it's missing a word or wasn't translated.

Speaker 1: Maybe it would be better to actually be able to fix that data in the survey when it's open and allow for the fact that okay, some number of people saw the wrong version. figure out how to deal with that rather than have to say, okay, we need to close that down, start it all over, tell everyone who took it already they need to take it again. So just thinking ahead and say, okay, well maybe yeah, you we're not gonna provide a front end for changing this after we're not going to allow the front end app to change this data after the surveys opened. But maybe we have a way outside of directly touching the database. Maybe we make it accessible in the Django admin so you could have sufficiently privileged people who can make little tweaks like that to the data. So maybe that helps alleviate issues like that.

Speaker 1: Another common case is where the focus is entirely on the should happen case and Everyone wants to ignore the uncommon case or the rare case of data or situations that don't quite match what you want to do. And it's really helpful, I think, to At least acknowledge these things might happen. As a concrete example of this, we had um we were ingesting membership data from a third party. And in this case, the website we were developing was using that membership data to decide. If if you were a member, or rather really if you were employed by a member company, then perhaps you had access to other things and you could operate on behalf of that member company. Whereas if you weren't a member company, you wouldn't have access to that.

Speaker 1: And the simplifying assumption they really wanted to make on the website was you are either associated with one member company or no member companies. You were never associated with more than one member company. So we always spoke about it in terms of, you know, a member company. And then but that wasn't actually enforced by the third party that was managing the membership. So the data we ingested could indicate that this user with this email address had multiple company associations, member company associations. So when you ask the question, well, what member company should we say when we're SSOing into another site? Your answer was one of these two. Which one should we use? So you need to kind of at least acknowledge these things might happen and ask about what

Speaker 1: should we do? Should we fail the entire synchronization of membership data if something invalid like that happens? Probably not. Should we Make this user look like they don't have any member company association. So then it's gonna badly impact that user, but then they can figure out how to fix that. So you you just thinking through what kind of situations you might hit and And figuring out how might best to deal with it might help so alleviate these common causes of specifications versus reality, causing really big issues in production. The second common cause I see of issues that hit in production, I do not live in the nirvana of development being the same as fraud.

Speaker 1: Um I I my development environment is actually significantly different than a deployed production environment I work with. And one big difference there is I'm I'm generally going to be using a single process run server to run my development server. And a deployed server is going to have multiple processes. Things that can be easily shared unintentionally in shared memory space on my development server may end up being in different processes on the deployed server. And that can cause some confusion and surprise to developers first encountering that. Something surprising when I first started working in web development was that my CPU on my development server was usually more capable than the web servers out in the cloud.

Speaker 1: You scale up websites to support lots and lots of traffic by having multiple lots and lots of web servers. They don't have big BT servers usually. So I was surprised to find that I could do something on my on my development machine, like ingest a bunch of data and do some modifications on it. Provided a response to my local, you know, user quick enough that it wasn't too laggy, but that was too slow for the web. server. So I would need you need to be aware that sometimes you need to be more efficient maybe than you realize if something runs even a little bit slow on your dev server. The traffic volume that you get on the deployed server is going to be higher than you generally are going to have on your

Speaker 1: local development, um just the volume of data usually on a production server is gonna be far higher. Um and the contents, like I said, the data is gonna be somewhat different. There's gonna be more exotic data, there's gonna be conditions that you hadn't considered. in the data. And finally, timing and concurrency. Generally when you're testing on your local development, you're going to be doing one thing at a time. You're not going to be banging on four different web servers at the same time with the same user trying to do the same thing. Um and Django offers all sorts of um Utilities and ways to make sure that you do things atomically, like get or create, update or create. There are utilities to make sure that you don't uh you know read data here, hang on to it, update it, and then save it all while some other thread did the same thing over here, and you end up

Speaker 1: stomping on each other's changes. But you kind of need to know to use it. You kind of need to sort of integrate that fact that multiple threads might be running through this code simultaneously and make sure that you code things in a way that That stomping on data doesn't happen. So these are the two broad cases I see of I've encountered a lot of how How production issues tend to arise. And now I'm going to sort of switch over into how to approach a project and build a project to make debugging in production easier. The first recommendation I have is to build on a solid foundation.

Speaker 1: Understand what it is you're building. understand the solution you're providing. You really don't want to jump into debugging something on production and not really be understanding what it is that it's supposed to be doing. So helping helping getting to a point where you really understand what you're building I think is very important. You want to choose good tools. Hopefully you have the opportunity to choose tools that you like to work with. You want to do all the good production development things. You want to do code review, you want to do unit tests, you want to do quality assurance. And all of this will make it so that hopefully needing to debug in production is less, less. It happens less. In general, your code runs well. And needing to solve issues in production would then be a rarity

Speaker 1: and that kind of thing can also help with your clients or your higher ups that if things are usually working When there is a problem in production, they might give you a little more grace to give you some time to fix it and figure it out rather than if it's breaking all the time in production, they may get a little antsy about fix it now. Second, I like to say uh I like to have an ability to have a a staging environment that is as much like production as possible, that has the same web server configuration, that has the same As much as possible the same data as production, the same configuration of the software, as much as possible like production. And that is sort of your your test bed for how things are how hard things are going to be to debug in production.

Speaker 1: In addition to making it as much like production as possible. I encourage clients to use it for their own testing. Our own testing uses it. And that gives you an opportunity to learn how hard it is going to be to debug issues that arise in production. So that's second. The third thing I like to do is understand once something is running in production, what does it look like when it's running normally in production? So having some sort of monitoring service, there's all sorts of services out there to do monitoring that will um Let you see what the normal traffic patterns are, how many requests per second you generally get, how much of the disk is being used, how much of the CPU is being used.

Speaker 1: Looking at the logs and seeing what generally gets logged. You can look at that, those logs, and say, hey, there's a lot of stuff in here that looks alarming. And it may turn out that that's just normal. That's the way this thing logs Or there may be the problem that you can find even when things are running normally and fix. But getting in a sense of what normal looks like really helps when something starts to go wrong, particularly if it's traffic related, um, to understand like what happened and what How maybe to fix it. If you jump into a new site and everything's not working, um you may it may be hard to figure out like uh how abnormal a certain situation is if you don't know what normal looks like. Next I would

Speaker 1: say anticipate problems. There's alerting services to let you know when things are going wrong, like when you can't access your server. There are services like Sentry that let you consolidate error reports. At a minimum you want to be getting like the Django traceback error emails. You may not want them going directly to your inbox. You may want to send it to a server that could consolidate them and say, okay, 18 people are hitting this issue versus you getting 18 different email alerts. Another thing to think about is keeping track of things that you know ought to be happening and making sure that they are happening. And a common thing there is you've got a celery task that's supposed to be running periodically.

Speaker 1: And a common characteristic of those failing to run for some reason is you're not going to notice right away necessarily. And it may be not until a while later that start people start saying, hey, this data looks out of date. Why isn't this, hasn't this been updated? And then you realize the celery task hadn't been running for a while. So coding the celery task to make sure that it's alerting someone that it ran, and then having that alert you if it doesn't see the alert is a good practice. Some of the ones that are essential that we use a lot, I've listed up there. AWS alarms, there's all sorts of things you can set up. One that I found to be really useful is if you're using any of the AWS services that have a baseline

Speaker 1: baseline performance, but allow bursting. So you can use this much CPU, but if it gets really busy for a while you can burst higher. Um the disks have the same kind of thing. You can use this much I. O. throughput, but if it gets higher, you can burst higher for a certain amount of time. Setting up alerts for those so that if it goes below some threshold. If you looking at what normal is and saying, okay, normally we run at 90%, like we never go below. You know, we get some bursty traffic and that might go to 90% of what we normally have, but we never really go down. To let you know if it goes below 50% of what you expect, then you can have usually some time because we've encountered a number of cases where excess traffic has caused these bursty these not bursty, but these resource types

Speaker 1: to over a course of like sixteen hours or so go from a hundred percent to zero. And then at zero, you can't even handle your normal traffic patterns. Like you don't have enough CPU to generate to handle your normal traffic. And it's it's a very confusing problem to kind of debug because everything's working. It's just a lot of things are timing out. So you can't, it's hard to figure out. And then you look at that graph in AWS and see, oh, we've got this traffic that's been coming in for 16 hours that is reducing our capabilities So that's an interesting kind of alert. Sentry is another one. I mentioned it for error alerts. that we use. And health checks is the one we use usually for that, remembering that something we're keeping track of things that should be happening,

Speaker 1: but that might not be happening. You can configure it to say this endpoint, you know, the here's this thing. I want you should be get pinged on this endpoint every four hours If you don't, and a grace period of an hour has passed, please alert us on Slack or email or whatever. So that's a I found that to be a really useful service. In conclusion, I like to celebrate launch days as beginnings of exciting new phases of a project. I do think it's important for everyone to avoid unrealistic expectations of perfection, like you're gonna hit problems in production. I do think it's also very useful to be prepared for those, to sort of um As I mentioned with the beginning areas of um

Speaker 1: kind of being aware of things that might cause issues, like in your in your um Focusing on the common case and how should you deal with unusual cases. Practice does help. I've been doing this now for over 15 years. It's probably a little more stressful in the beginning. And I was there's somebody something good about having production issues in that the code that I wrote and worked on with the team is actually in use and people are using it. I like to have my code be useful. So I see that as a bright side. And I think I might have like one minute for questions if there are any.

Speaker 2: Yeah, and we just have a break afterwards, so we can take a couple more if you want to

Speaker 3: Thanks. Um as far as uh anticipating problems, how do you think about uh I guess like being pragmatic, like not over-engineering and being too anticipatory, but also being realistic about that there won't just be no issues.

Speaker 1: Yeah, there's a trade-off there. Um I mean And I an area where I see it a lot with clients is when you try to talk them through the unusual cases, they're often just like they don't want you, that's never gonna happen. It's gonna happen. And if they're really unwilling to talk about it, then I usually try to structure the code so that it does something fairly innocuous. Like maybe it doesn't work for that user who's in that use in that odd case, but it doesn't break the whole system. So I just and similarly with trying different kinds of data, you know, I kind of add to my list of weird data over time and include that in unit tests and stuff, but I don't try to proactively figure out what the next new data, weird data I might find is. It's a balancing act of trying to make it perfect versus making it good enough.

Speaker 1: And you just kind of have to Try to find the mean middle path there somewhere. And usually driven by deadlines and stuff to get it done enough.

Speaker 2: All right, last question.

Speaker 4: How do you um handle the stress of debugging and production, especially if like the site goes down or something really Just you're lost in the sites failing.

Speaker 1: Take a deep breath. Uh drawn my team if I have a team to help me. Uh uh usually I mean on the bright side I'm not generally working on things that are life critical. So if users are hitting issues if the site is down, it's probably not the end of the world. It can life can continue after we fix it. Um so usually like just trying to Um sh shield my mind, like turn that off in my mind and try to focus on, okay, this is the information I have. This is what's not working. Why would it not be working? What could have caused this and what would

Speaker 1: If it's try often it's like try to get a sense if it is a sight-down situation, which thankfully hasn't happened a lot, you know, try to figure out Is there something we can like if this is not going to be fixable really quickly, maybe there's a quick, you know, page we could put up that is like We're having issues, you know, come back later kind of thing, and then kind of give me breathing space, give your team breathing space for actually creating a solution that's gonna work.

Speaker 4: Thank you.

Speaker 2: All right, everybody a round of applause for Karen. Thanks, Karen.