Entomology 101: Effective Bug Hunting by Frank Wiles

Speaker 1: Come on, no.

Speaker 2: So the deadline for putting in a talk was coming up and I was actually kind of stressing about it. I wasn't coming up with any good ideas and work had been super busy. And I get home and my youngest daughter is like, let's go look for bugs in the yard. And it was hot and I was tired and work had been stressful and I really wasn't feeling it, but we went out there and you know, how do you say no to it to that cute face? So I go out and and uh while we're out there was like you know daddy hunts bugs all day at work she's like really I was like well computer bugs and Apparently I'm not very good at finding real bugs, because we didn't find any bugs that day, which might have been because it was like a hundred degrees outside, right?

Speaker 2: Um but we are all bug hunters, right? We just do it in comfy chairs And with air conditioning and not in the Kansas heat or the Kansas humidity. But one of the things when we're going on a bug hunt, you've got to get the proper equipment, right? So we need to get we need to dress for the part. And we need to get our net and get ready to go on a bug hunt. So as a developer, what's one of the first things you reach for for equipment? When you're debugging the the print statement. Everybody does that. Who everybody here? We've all used the print statement. You shouldn't use the print statement.

Speaker 2: And you may be thinking it's a trick question. Maybe you should be using this print statement, right? Because we should all be using Python 3, right? Right? You shouldn't use that print statement either. There's better tools. Depending on what kind of bug you're hunting, we're looking for common bugs, regular bugs. We can use something a little easier that helps in more general ways specifically for Django, Django debug toolbar. Who here uses Django debug toolbar? Okay, good. Most everybody, but some people haven't. I'm always surprised the people who have never heard of it and never looked at it and think it's hard to use or that it's something difficult. So we're gonna take a quick look At Django Debug Toolmark.

Speaker 2: So I've got a silly little app that I'm working on, and if you look over here There's this little highlighted thing, and if I click this, it shows me all sorts of interesting information about my site. It shows me, oops. It shows me which versions of things are installed, how long different things took to how much time do I spend in the request DOM content shows settings. All the different settings. If I'm confused, I don't need to go look at my settings file. What settings are being used right now? What type of headers? And then the request itself. All my cookies, everything I'd want to know about your average Django

Speaker 2: request. Now this is one that I think people don't use enough, which shows all the SQL queries. This is where your app gets slow. We can see the whole statement and then exactly where it was generated. We can see a query plan from Postgres, see if we're missing an index. If something's not showing up, In your template, you can print out statements and see that it's set in the context, or you can just click into here and see exactly what is in your template context. You can also see the signals that are fired, any times you called cache, any logging statements you

Speaker 2: used. I apologize if my voice is a little hoarse. I'm a little bit under the weather. But sometimes there's harder bugs. Bugs that you just can't fix with debug toolbar. Scary bugs. So maybe you need to reach for something like PDB, the Python debugger. Who here uses the Python debugger? I'm surprised it's almost the same number of people. I I never ever use debuggers. I don't know why. I I don't it's just never been my thing. I'm never confused about the program

Speaker 2: 's flow. I'm always just confused about I'm just wrong. about what data is set where I think it's set, right? I think I've got a list, it's actually a tuple. I think I've got a dictionary filled with data and it's actually a none, right? Like that that those are my confusions. Those are where my bugs come from. They don't come from I don't understand why this if isn't firing. It's not firing because the data's not set, right? So you can PDB, if you're not familiar with Python debuggers, you can run your program through and if you set trace at any particular spot. Your program will stop, and then you can slowly iterate through. Go to the next line, go to the next function, show me the value of this variable here. And you step through your program line by line.

Speaker 2: Like I said, I don't particularly find that useful, but maybe you will. This is really helpful if you are very c if you have a very, very complicated program and you're very confused as to why you end up over here when you think you should be over here But in my experience, if your program is that complicated, you should probably work on just simplifying your program rather than getting into more advanced debugging. So yeah, not a huge fan. I am, however, a huge fan of this. You can embed a Python shell, an iPython shell, anywhere you want. It's, in my opinion, basically the same thing as using a debugger. So let me show you a quick example

Speaker 2: So, this is the the view for this dashboard page we're looking at. And if I do this. So from my Python import embed and then call embed. I use it so much I have a a hotkey for it in in Sublime. So if I go back here and I refresh the page, you'll see it's just spinning. It's just waiting. But if I go back over to my run server, it's now dropped me into an iPython shell Right in that, so I can say self-request user. I'm right in there. What's in the context? I can add stuff to the context. And then when I exit, it continues on normally, and we go and we've refreshed the page now.

Speaker 2: And if I look in here. In my context, we've added foobar there into the context. So I'm really just plugged right in at that line. But I can navigate with iPython, which is something I use all the time, and not stepping through with a debugger and having to remember all those commands. Of course, it jumped me. But I think the real secret

Speaker 2: to an effective bug hunt is to use login. There's really two kinds of bugs. There's the bugs that I've just generated locally in my local development. And then there's the bugs that happen after we've deployed to staging or production. Logging gives me the information for both. So I've got good logging and I'm logging locally, I get my answers about what my data is set to, and then when I go to production, I have that same data. Right? So I'm killing two birds with one stone. Two bugs with one stone. So Obviously, you can just use the standard Python logging library. And it's really it's really hard to use.

Speaker 2: Um this is an example of Using Python login. You just set up, this streams it to standard out. You're using containers in a good 12-factor app. This is really all you need to do for a script. You can set different levels so you can turn off. So right now we're logging the debug level. So we see everything from the debug level and above. And so we would see both testing and debug in our logs when we ran this. If we set that to just info, we would only see the testing. So we can turn off the really verbose data that we don't want to see. There's a slightly better version that Armin Renauscher

Speaker 2: did called logbook. It's quite a bit faster, so if you really get into logging and you do a lot of logging and you're worried about eking out that last little bit of performance to your app, having a whole lot of logging statements is actually fairly slow The logbook is quite a bit faster. I think it's about twice as fast. Um and it's just about as easy to use. Here we just set up slightly differently, and again, this will just log to standard out. We'll log. Here's some info. Finding bugs is fun. There's also a Python library called struct log. Um its example does not fit well on a slide. It's much more verbose.

Speaker 2: But what it allows you to do is as you're going through your code, you can say, ah, bind this variable here. to this name in my structured log. And then when I get to a point where I want to emit a log, I just say emit this log and all the variables that I have bound, their values end up in the structured log And you can decide to output them as JSON or as CSV or however you want to use them downstream. As I mentioned, my daughter doesn't think I'm very good at bug hunting. Because when we went out there, you know, I kept turning over rocks, and we weren't finding any bugs So one of the one of the techniques for finding where is your bug is to divide and conquer.

Speaker 2: First I lift up this rock, nope, no bug. I go over here and lift up this bug, this rock, no bug. So it's gotta be here. It's gotta be under this middle boat, or this middle rock. And so I can dig in there. Once I've verified it's not over here and it's not over here, it must be somewhere in here. I can dig in and get into the meat of things Another thing that I've found people don't do is they keep attacking the problem from the kind of the same state. And often if you start with a clean slate, Fresh data, clear out your database, get rid of all your testing data, recreate whatever you need. Fresh perspective. Go grab lunch. Go walk, go grab a coffee. Um when I used to be a smoker, I can't tell you how many times I'd be so frustrated with a bug and I'd get up to go outside for a smoke and I wouldn't even make it outside before I go, oh, that's where my problem is

Speaker 2: Just getting your mind out of uh out of the problem. And then a fresh set of eyes. Some people uh have talked about this is called rubber ducky debugging, you know, talk to s Just say it out loud. Get your friend to come over, take a look at it with you. Sometimes just explaining the problem to them will help you find it. And then if you've got this complicated mess, whatever you do, only change one thing at a time. Because you're never going to be able to figure out where this is coming from, where your problems lie, if you're pulling out four wires at a time. Does anybody have any questions? Happy to

Speaker 3: We actually might want to get you to put the hat back on and grab the net for a few But yeah, we actually do have plenty of time for questions, so please don't be shy. Um just for recording purposes, do Come up to the mics on either side and let's chat.

Speaker 4: Hey how you doing? Um you just basically described the story of my life. And

Speaker 2: it's all of ours, right?

Speaker 4: This is all I do. This is all I do is is find the story And I guess I could make a lot of comments, but I'm not gonna do that. I'll ask a question. Um after you've done all this stuff, um talk I would love to see you talk about about the role of regular expressions and funny things like that, right? You know, to find all the instances.

Speaker 2: Of this particular bug, yeah. Um well I mean that's really more of a problem of being dry, of not repeating yourself. Like, you know , You should really only have a bug and that same bug in one spot, right? Like if you're having to find it copied in many, many places, that's probably a just bad design. Um but yeah, you know, using Using regular expressions to find common patterns happens. I do a lot of refactoring for people, and so I'll see that they're using something, a bad pattern. Um that's more of a a maintenance nightmare than really a bug. It does what it needs to do. It's just, you know, verbose or slower or something like that. And so I'll have to search through a code base. Um there's a really great tool called ACK.

Speaker 2: ACK that's uh will it's basically grep for programmers. So you can say search all, you can say ac Python. search and then a a search string and it will search from the current directory down only Python files for that string. Or you can say acss and it'll only look for CSS, but also is smart and looks for SAS and and less and and those kinds of things. So um that that saves me so much time. And it gives you some context lines around each match so you can kind of see where in the code. Here I'm I'll show you an example. Oh , yeah. So you can see it picked up a bunch of HTML docs and stuff where I'm mentioning

Speaker 2: And so if I just do ac Python, it'll only find it in the Python files.

Speaker 4: Yeah, that's good stuff. to know because some of us we go into strange code bases, large strange code bases. So rewriting it is is not an option, you know, refactoring it is not an option. You know, but fixing bugs is a requirement.

Speaker 2: Yeah, yeah. No, um another one I really uh I'm I use Sublime Text and it's got a really great search replace feature so that you can step through and be like, ah I should replace this, I shouldn't replace that, I should be replacing this one, and it kind of walks you through and you You can do multiple files with the same. So that might be a something to use.

Speaker 5: Hi, great talk. Quick question. So you mentioned Uh PDB and how you're not too fond of it and then you mentioned um IPython and how you love it. What are your thoughts on IPDB, which is PDB with an IPython

Speaker 2: shell? Um it's so the thing that I that never clicked for me uh until recently is that with uh so IPDB is better, like it's got the better tab completion type interface and everything. everything and the history. The thing that never clicked for me was that with PDB, you could you can issue the interact command and then it basically gives you an IPython shell at that point. But for me, I that was the first thing I was always typing, right? And so I just switched over to going straight to the shell because I didn't need to step through anything. I just I don't know I I'm very rarely confused as to why I'm in this function or why you know how this flowed. I just need to see the data here and then maybe over here, but I don't typically need to do that in one session.

Speaker 5: Alright, so it was more of a habit thing. I think

Speaker 2: Yeah, it could be.

Speaker 5: I think you'll like the new IP DV with iPython five though.

Speaker 2: I'll I'll check it out.

Speaker 6: Hi. This is a little bit more of a vaguer uh open-ended question, but you know, I find that w as you become more experienced uh programming, you sort of develop But when working with a novice programmer or someone who's learning, um a lot of times they sort of tend to shoot in the dark because because they they haven't really learned how to properly debug yet. And you know, I always say the first step is look at the error message, right? But do you have any experience or Just comments on when you're working with someone who's new to programming, how to sort of guide them through the debugging process?

Speaker 2: Um so What are uh it it depends on the bug. If it's really like if if I walk up and I see the error and I know exactly what they've done, right, um that's different than if I have to dig a little bit. If I'm digging a little bit, I very much tell them exactly why I'm doing the thing I'm doing. I'm gonna look over here because I think the problem is in the template and it's just that you're using the wrong template variable name And it's not that the data's wrong, it's just you're using it wrong here. Oh no, I'm wrong. Look, you've you've used it right, so it's that the data's not getting set. So we now need to go to the view and look and see how the data's being set. Right? And instead of just going, uh, oh, it's right here. Because I might jump through three or four files real quick and confuse them.

Speaker 2: Yeah, I don't know. It's debugging is hard, right? Like it's it is something you get better at as you get hopefully also you get better at not end bugging and not putting them in in the first place. But uh You know, but debugging is is what we do. This is why we s you know uh the I the one thing I do when I when I teach training classes is I tell people I was like, you know, just last week I spent two hours going What the fuck? Why is this not working? And you know, it ended up being something very small that I just overlooked three or four times, right? And so to give them that encouragement that it still happens to me, it still happens to people better than me, right? Like you 're never gonna get rid of this, it's always just a a matter of

Speaker 2: of you know, working through each one. You you meant uh you mentioned regular expressions uh uh in the last one and I just uh realized that for me the hardest thing in the world to debug is is a problem with the regular expressions in URLs. py. And I'm really good with regular expressions. I came from the Perl world. And so like I've read mastering regular expressions five or six times. I'm really good with regular expressions, but for whatever reason that is the hardest thing for I can I don't see them, the see the problem in the URLs.

Speaker 3: All right. We have uh I think we have enough time to get these two and that one, so we'll let you go ahead since you are next.

Speaker 7: Uh yeah, I was just wondering if you had any specific tools that you know of or strategies as far as like debugging microservices, if you have like a Python microservices architecture, if you're familiar with Because I think like PDB and those sorts of tools work really well if you have something monolithic, but like if you're trying to trace something across a bunch of different even if they're all Django REST framework APIs. You

Speaker 2: 've got to use logging, right? And you need to log them into one central thing so that you you can look at them and and then you can see, okay. issuing this request and I got back this data and then it went to this service and it issued this request and it asked for this data and it got this data back in each one and that's really the only way you can do it right. Because otherwise like you said you're gonna have to put a set trace in here and step through stuff and set trace over here and step through stuff and that that doesn't work very well. Okay.

Speaker 7: Yeah, um Django toolbar is really great. But what if you're someone who's doing a single page app and using something like uh React JS and everything's in Ajax and suddenly all I'm doing is using the browser's console to see what the network track Do you have something to recommend for folks in that situation?

Speaker 2: So if you're using React and Redux, there's really good uh browser plugins for Redux that can show you your the Redux dev tools um is really great. But also one thing that people miss, and I'm gonna have to look this up real quick because I don't remember in this project what uh one of my APIs are here. But if you um Okay. If you hit the Django Rest framework endpoint , you can use the first thing. The debug toolbar still works. And it

Speaker 2: like so, for example, if you want to see the queries or you want to see anything about So if you hit the browser interface, that's supposed to be the human consumable part, if you have any questions about how that data is getting in there on the Django side, you can still use the debug toolbar. But in the entirely in the JavaScript world, you've got to rely on the JavaScript tools really to help you there. Okay, cool. Thank you.

Speaker 7: I've got one quick question. As a big fan of Spectrum, and if there are people in this room who don't know Spectrum, check it out. Twitter. com slash dev spectrum if there are any new or cool features on the way.

Speaker 2: Um so right now I'm so Spectrum is a product that we put out that that makes logging useful for the developer. Uh right now I'm just adding support for other languages. So like we've got a Chrome plugin and so you can get console. log and I'm slowly adding in Ruby and Go and JavaScript logging libraries for Spectrum. Okay. Well I think we're at time, so thanks a lot, everybody.