Lazy Looping: The Next Iteration by Trey Hunner

Speaker 1: Uh so my name is Trey, and if you write Python code for a living, you might want to check out a service that I run called Python Morsels. Sign up, choose your skill level, novice, intermediate, advanced, and you will learn something new about Python every week. And if you don't, send me an email because I want to know. So we are going to quickly start with a problem. This code finds lines in a log file that contain errors. But it prints out not just the error line, but the line just before it and the line just after it. You don't need to understand this code. We are going to look at it later, so for now, just forget about it. When we do refactor that code later, we're going to use lazy looping, which is all about iterabs and iterators. So we need to define what iterabs and iterators are.

Speaker 1: An iterable is anything that you can loop over. Lists, tuples, files, lots of things in Python are iterables. If you can loop over something If you can write a for loop to iterate over anything in Python, that thing is by definition an iterable. You're able to iterate over it. So we kind of already know what iterabs are, even if you don't know this term. But what are iterators? So an iterator is a bit more complicated than iterables. There are two definitions of these things. First one is that iterators are the object that powers all iteration under the hood in Python. Second, they're kind of a lazy iterable, which is consumed as you loop over it. We're going to completely ignore that first definition.

Speaker 1: It's outside of the scope of this talk. We're going to focus entirely on using iterators as lazy iterables. If you are interested in that first definition though, you can watch another talk that I've given in the past called Loop Better. In it I talk about how for loops under the hood in Python are powered by iterators. We're, again, not going to talk about that though. We're talking about how iterators are lazy iterables. So this is one example of an iterator in Python. File objects in Python are iterators. You've probably worked with files before in Python, but you might not have known that they are iterators. I'm going to use this object here to show you what the two things you can do with iterators are. So first thing we can do is pass an iterator to the next function that is built into Python.

Speaker 1: This will give us the next item in the iterator, which in the case of files gives us the next line in the file. The second thing we can do is loop over them, which is the same thing we can do with any other iterable in Python. But unlike other iterables, iterators are consumed as you loop over them. So if you start getting items from an iterator and then you stop, the next time you start again, you'll be right where you left off before. And if you loop all the way through an iterator and then you try to loop over it again, the second time it will be empty because this iterator at this point has been exhausted. We've consumed all the items from within it. So the benefit of iterators is that they are lazy. Instead of doing a whole bunch of work up front, they do little bits of work as we loop over them.

Speaker 1: They usually don't know what items they've given you already or what items they might give you next or in the future. They only know how to give you their next item, how to compute that next item and give it to you Which means that they're often pretty memory efficient. If we loop over a file in Python, whether it's one megabyte or one gigabyte, it will store the same amount of memory as we are looping over it. So, because we consume items from iterators, as we loop over them, you can think of iterators as like Hello Kitty PES dispensers. When you take a PES out, it is gone. You've consumed it. And once that dispenser is empty, it's useless. There's no way to reload this iterator. Uh and it has no memory of what used to be in it, and it has no idea how many things might be in it.

Speaker 1: Iterators are lazy single-use iterables. So lazy looping is all about creating iterators and looping over iterators. We've already seen how to loop over an iterator. You just write a for loop. How do you create an iterator? So this function is uh very similar to the built-in enumerate function in Python. It's just called denumerate. Instead of counting upwards, it counts downwards. If we take an iterable, say a list of strings, and we pass it to this denumerate function and then loop over the result, the thing we'll get back are tuples that have two things in them. uh numbers counting downward and the actual items that were in that uh iterable in this case a list. Uh

Speaker 1: So when we call this thing, it does all the work up front. It's building up a list of tuples, and then it's going to return that list of tuples to us all at once Which might not be the most memory-efficient thing, depending on how big the thing that we're looping over is. Instead of creating all of the tuples right away, as this function does, we could make this function return a lazy iterator to us, one which would only make the tuples just before we need them. It would delay evaluation, only make them just before we need them. We can actually do this by removing code from this function. The first thing that we would remove, uh somewhat strangely, is that return statement. We're not going to return anything anymore.

Speaker 1: Then we would also remove the values equals an empty list. And then this values. append, which doesn't actually make any sense anymore, we are going to turn this into a yield statement. So this thing that we've just made, it is called a generator function. Generator functions return iterators when you call them. So when we call this denumerate function, this new one that we've made. We can take the thing that we get back from it and just like a file object, we can call next on it. Or we can loop over it. But if we loop over it a second time, just like the file, it will be exhausted, it'll be empty. And this is all because of that yield statement. That turned what was a regular function in Python into a generator function. And generator functions are really different from other functions in Python. If you have a regular function

Speaker 1: Python and you call it, it will execute that function. And if we look at the thing that it gave us back, we'll see it gave us its return value. But if you take a generator function and you call it Nothing will seem to happen. The function doesn't actually execute at all. And if we look at the thing that it gave us back, we will see it gave us a generator object. functions and generator functions are completely different animals, all because of that yield statement. So we're going to walk through every step of looping over a generator object. This is going to be quick, so if you've been asleep up till now, wake up for a minute or so. I'm going to be bolding the code as we go here. So when we loop over a generator object, it is going to start executing our function.

Speaker 1: So we will see those first couple print calls happen. And then the generator gets to that yield. statement and it will put itself on pause and yield control back to our loop, passing that tuple back to us. Then we execute the body of our loop, which will print something out, and then we will ask the generator for its next item, which will start it up right where we left off, right after that yield. So it prints again, and then it gets to another iteration of its loop, which prints again, and then it will yield again, which puts itself on pause again, gives us another tuple back in our loop. So we execute the body of our loop again, which will print something out.

Speaker 1: And then again we ask the generator for another item. It again starts looping right where we left off, right after that yield, which we'll print. And then when it asks for another thing from its loop, it'll see it's actually empty. There's no more items left. So it'll get to the end of that generator function, uh which will actually exhaust the iterator. So when this returns, this iterator, this generator is now exhausted. There are no more items within it. So we are done looping. Generators and all iterators compute their items as you loop over them. When you ask a generator for its next item, it will execute some code to figure out what that item is. It'll yield that item to you, to whoever it is who's looping over it there, and it will put itself on pause until you ask it to generate another item for you.

Speaker 1: So lazy looping is all about not doing any work at all until just before the moment where you need that next item. Generator functions return generators to us. They return us generator objects, and generators are iterators. This is a generator function. It's yielding, just like our denumerate generator function did, but it does something a little bit different here. If we call this generator function, the thing that we get back is a generator object. But this is not the only way to make generator objects in Python. There's another way to make generator objects. This is a generator expression. It looks like a list comprehension if you've ever seen these things before, except that list comprehensions give us new lists back when we execute them.

Speaker 1: Generator objects give us new generators. So this object that we just got back from this generator expression, it does the same thing as this other generator object that we got back from this generator function. So instead of making a whole new function, uh sometimes you can just write a single expression that does the same thing, makes a generator for you all in one line of code. So to make an iterator in Python, you can either make a generator function or make a generator expression. And calling a generator function, it makes a generator object. Evaluating a generator expression makes a generator object as well. And generators are iterators, the same way that files are iterators. They are lazy iterables. Technically, there is another way to make your own iterator

Speaker 1: , but you don't need to know about that. We're not going to talk about that, and it probably will never be important for you and your code. Now once you've got an iterator, whether you made a generator function or a generator expression, what do we do with it? So the obvious thing to do with it, which we've already been doing, is loop over it. We can take an iterator of numbers here and loop over that iterator and maybe sum up all of those numbers. Or we could pass that iterator to a function that'll do the work of looping for us This second way of doing things, it's actually so common in Python that the Python core developers have made a special rule just for generator expressions. If we pass a generator expression into a function as its only argument.

Speaker 1: we can actually drop those extra set of parentheses, kind of merging the function call syntax and the generator expression syntax into one. Regardless of how you're looping over iterators, whether manually you're passing them off to a function Looping is really the very last thing that we will do with them because once we've looped over them, they are exhausted. There's really no use for these things anymore. But before we do loop over them, we could take this iterator and wrap it in another iterator. So as an example of this, we're gonna write some code that adds up travel expenses. We will read a CSV file that has expenses within it. We'll use Python's CSV file to help us parse it, or CSV module to help us parse it. So we have here a file object, and we already talked about this, that file objects in Python are

Speaker 1: iterators, we're passing it to csv. reader, which is actually going to return to us another iterator. We're going to take that iterator and wrap a generator expression around it. And that generated expression is going to unpack the columns from each row, grab only the travel expenses from it, and then take the costs from those travel expenses and convert them to floating point numbers. We're then going to take that generator expression, which again is an iterator, and pass it to the sum function. So we have now an iterator wrapped in another iterator, which is wrapped in another iterator. And no looping is actually going to happen until we pass that final iterator to the sum function. It will start looping over that outermost one and then we actually start the chain of work happening inside each of those iterators.

Speaker 1: And again, because we're passing a uh generator expression as the single argument to a function, we can actually drop those extra set of parentheses. So that analogy of Hello Kitty Pez dispensers, it doesn't really reflect the fact that iterators can be wrapped in other iterators, which is kind of unfortunate. So if you'd like to, you could instead think of iterators as kind of like xenomorphs, where there's one head that's wrapped inside of another head, or another head's wrapped around it. Except that this also doesn't really model the world properly because you can wrap iterators as many levels deep as you like. So you can have heads inside of heads inside of heads And also that PES analogy still holds because iterators are consumed as you loop over them.

Speaker 1: So maybe iterators are really like recursive Hello Kitty PES dispenser xenomorphs. When you loop over an iterator, you wrap an iterator rather around another iterator, the outermost one has to delegate to the inner one all the way down until you get to the innermost one, and then you have to work all the way back up until you get the thing that you're actually looking for. If this analogy doesn't work for you, I'm sorry, but I you're gonna have to come up with your own analogy. I don't know of a better one. So there are two general things that you can do with an iterator. You can wrap it in another iterator as many times as you'd like. And the last thing that you'll always do eventually is loop over the outermost iterator, but only once, because once you've looped over it, it's exhausted. There are three ways to wrap around an iterator. The first way is to pass it to a generator function, or you can pass it to a generator expression.

Speaker 1: Or you could pass it to some other function that will return an iterator back to us, like csv. reader did. There are also a couple ways to loop over an iterator. You could write a for loop, or you could write a list comprehension, or again you could pass it to some other function that will do the looping for us, kind of like the sum function Python did. Alright, so we're going to finally revisit that problem code that we saw earlier. We were trying to find errors in a log file, but we also wanted to print out the line just before the error and the line just after the error. This is what we started with, which is a kind of a lot of code. It's at least dense code here. A lot of this logic is around getting the line just before the current line and the line just after the current line.

Speaker 1: So we could refactor our code. to look like this. And we're done. Except not really because we're actually passing our log file to two functions that don't exist. We're kind of this is sort of aspirational programming here. One of these In theory, we'll lazily strip new lines from the ends of our lines, kind of like we were doing before. The other one should give us the line just before the current line, the line just after the current line If these functions did exist, if we're you know we're practicing aspirational programming, if these did exist, uh and they gave us lazy iterables, they gave us iterators back. then our code would be just as efficient as before, but it a lot more readable, at least in my opinion. So if we can figure out how to write these functions, our code would be more readable.

Speaker 1: Uh fortunately you don't have to figure out how to write these because I wrote them for you. This is strip new lines, and this is a round, which is a big and scary function, but it's already been written, so you don't have to care about it. So this code, it's a lot longer than what we started with. If you take the original for loop, it is shorter than this code here. But I find it more readable. Mostly because the four lines that we ended up with in that loop, they hide the details of what we're doing. The details that we don't need to care about if we're just glancing at this loop, trying to figure out what it even does. So we've broken up our loop into little bits of work that the reader of our code doesn't have to understand unless they decide they want to dig into a specific detail.

Speaker 1: Now I do want to make an important note. While you are writing generator functions, keep in mind that the best code is code that you don't even need to write. Python has a bunch of iterator returning functions built into it, a bunch of lazy looping helpers. An example is the enumerate function, which is what that denumerate thing was based on. It's built into Python. There's also zip and reversed and any and all and a whole module in the standard library called IderTools that's really just full of lazy looping helpers. I'm not going to explain what these do, but you can look them up on your own. And if you don't find what you're looking for built into Python, uh you might find it hiding in a third-party library. There's one called More Iter Tools, uh which is

Speaker 1: Like the iter tools module, there's just more of it. And there's another called Boltons, which has a whole section that's just for Eater Tools like stuff. For example, this around generator function, that big scary function that I wrote, I could have actually written this. Like this. The Boltons module and the More Age Tools module actually include a helper in them called windowed. So here I am using this windowed thing which returns an iterator and I'm wrapping around it. And in fact it is also wrapping around chain which wraps around some other things. So we're relying on third-party code that does a lot of our work for us. which is kind of nice.

Speaker 1: So I have some bad news. The way I used the word generator throughout this whole talk is not a universal usage of that word. So the way I defined these terms is that an iterator is a lazy single-use iterable that computes its next item and gives it back to you as you loop over it. Does work as you loop over it A generator function is a special syntax that we can use to make a function that returns one of these iterators. A generator expression is also a special syntax that looks like a list comprehension. and when you evaluate it, it gives you an iterator back. The type of iterator that you get back from these things is called a generator object

Speaker 1: or for short just a generator. When you call a generator function or evaluate a generator expression, you will get a generator object back. And generator objects are a type of iterator. They're basically the easiest way to make an iterator in Python. So this is how I defined these terms. The Python documentation disagrees with me. What I call a generator function, it calls a generator. What I call a generator object or just a generator, it calls a generator iterator. Which is not a term that I've heard anyone use outside of the Python documentation. I'm sure some people do though. Fortunately a generator expression, it also calls a generator expression, though I do sometimes wish they were called generator comprehensions

Speaker 1: Because it would be easier for me to teach them. So you might think at this point, Trey thinks one thing. The Python documentation thinks another thing. So obviously Trey is wrong. And I would agree with you, except that I'm not the only one who uses these terms this way. Many other Python educators and many other folks in the Python community use the term generator the same way I do. They use it to describe a generator object. In fact, the Python documentation says the term generator may be used to refer to a generator iterator in some contexts. So these terms are confusing. Different people use them differently in different contexts and that's just the way life is in generator land.

Speaker 1: I don't have a solution to this problem. I don't think there really is one. It really depends on the context what these terms mean. mean. So to recap, iterators lazily compute their next item as you loop over them. And the easiest way to make your own iterator is to make a generator. Though you often don't need to actually make your own iterator because the Python, Python itself comes bundled with a lot of lazy looping tools built in, and third-party libraries have a lot of tools that you can use instead. The purpose of iterators is to make your code more memory efficient, even if you're looping over a really big iterable. Also, wrapping iterators and iterators can really help you break your loops, uh your big loops into small chunks.

Speaker 1: and give descriptive names for each of these steps in your loop, which can really help improve the readability of your code, which is my favorite reason of using an iterator in Python. If you were interested in diving into this topic more deeply, I gave a three-hour tutorial on this at PyCon if you would like to sit through three hours of this. It's also more hands-on, so it might actually be more useful to learn from. And that tutorial as well as a whole bunch of other resources are linked at this URL. Thank you.

Speaker 2: Thank you very much, Trey. Um you said that words are hard and one of the things I've always found confusing about this is the word yield. And I've always wondered why that word is chosen as as the um uh as the as the name for for what's happening. And in fact you used it in two different ways. You said it yields control back to the loop and you also said it yields some values. Can you explain what yield is doing there and how we should understand it to help understand what these things do?

Speaker 1: Uh good question. I don't think I can. However, I can say that kind of like how no one really knows what yield signs do in the US, no one really knows what the yield statement does. So I don't think that helped actually at all. But uh I I I see it as it puts itself on pause and gives you an item back at the same time. But I don't really know why the word yield was chosen, although I suspect there was probably an argument on a Python core developer's mailing list years ago that might explain it.

Speaker 3: Y in in your function in your uh example for the generator function, uh you showed that there is a variable and then the yield statement and uh next time you go through a for loop you'll get a new value. So I'm just curious, uh I'm assuming that's a single-threaded uh execution. Uh where does the state of the variable stored for the generator function? I mean Uh is there a separate stack that gets created or what happened?

Speaker 1: Yeah, uh this is another complex question that's putting me on the spot. Uh I don't actually Uh I think that this may not be the the code that I wrote there may not actually be thread safe in the sense of uh if you're executing in two different threads, it uh well I don't know actually. Let's talk about that in the hallway. Someone who probably has a better answer for that might. Because generators are kind of their own interesting thing. Threads are their own interesting thing. And I haven't really mixed the two too much in my own code. I mean in general with web development, it's not that common to use a generator in the sense that we often don't have giant data structures. It's much less common to use a thread, at least in the sense of the request-response loop. So uh maybe we can talk about that later. I'm hoping someone can maybe pop out in the hall and answer that for me.

Speaker 4: Okay, if you have more questions, uh we'll talk to Trey in the hallway. We'll see you in for the next talk after the break.

Speaker 1: Thank you.